1 Going Deep in Medical Image Analysis: Concepts, Methods, Challenges and Future Directions

Fouzia Altaf, Syed M. S. Islam, Naveed Akhtar, Naeem K. Janjua

Abstract—Medical Image Analysis is currently experiencing a paradigm shift due to Deep Learning. This technology has recently attracted so much interest of the Medical Imaging community that it led to a specialized conference in ‘Medical Imaging with Deep Learning’ in the year 2018. This article surveys the recent developments in this direction, and provides a critical review of the related major aspects. We organize the reviewed literature according to the underlying Pattern Recognition tasks, and further sub-categorize it following a taxonomy based on human anatomy. This article does not assume prior knowledge of Deep Learning and makes a significant contribution in explaining the core Deep Learning concepts to the non-experts in the Medical community. Unique to this study is the Computer Vision/Machine Learning perspective taken on the advances of Deep Learning in Medical Imaging. This enables us to single out ‘lack of appropriately annotated large-scale datasets’ as the core challenge (among other challenges) in this research direction. We draw on the insights from the sister research fields of Computer Vision, Pattern Recognition and Machine Learning etc.; where the techniques of dealing with such challenges have already matured, to provide promising directions for the Medical Imaging community to fully harness Deep Learning in the future.

Index Terms—Deep Learning, Medical Imaging, Artificial Neural Networks, Survey, Tutorial, Data sets. !

1 INTRODUCTION of previously unseen data (i.e. testing data). This strong generalization ability of DL currently makes it stand out Deep Learning (DL) [1] is a major contributor of the contem- of the other Machine Learning techniques. Learning of the porary rise of Artificial Intelligence in nearly all walks of life. parameters of a deep model is carried out with the help of This is a direct consequence of the recent breakthroughs re- back-propagation strategy [10] that enables some form of the sulting from its application across a wide variety of scientific popular Gradient Descent technique [11], [12] to iteratively fields; including Computer Vision [2], Natural Language arrive at the desired parameter values. Updating the model Processing [3], Particle Physics [4], DNA analysis [5], brain parameters using the complete training data once is known circuits studies [6], and chemical structure analysis [7] etc. as a single epoch of network/model training. Contemporary Very recently, it has also attracted a notable interest of DL models are normally trained for hundreds of epochs researchers in Medical Imaging, holding great promises for before they can be deployed. the future of this field. Although the origins of Deep Learning can be traced The DL framework allows machines to learn very com- back to 1940s [13], the sudden recent rise in its utilization for plex mathematical models for data representation, that can solving complex problems of the modern era results from subsequently be used to perform accurate data analysis. three major phenomena. (1) Availability of large amount of These models hierarchically compute non-linear and/or training data: With ubiquitous digitization of information

arXiv:1902.05655v1 [cs.CV] 15 Feb 2019 linear functions of the input data that is weighted by the in recent times, very large amount of data is available to model parameters. Treating these functions as data process- train complex computational models. Deep Learning has ing ‘layers’, the hierarchical use of a large number of such an intrinsic ability to model complex functions by simply layers also inspires the name ‘Deep’ Learning. The common stacking multiple layers of its basic computational blocks. goal of DL methods is to iteratively learn the parameters of Hence, it is a convenient choice for dealing with hard the computational model using a training data set such that problems. Interestingly, this ability of deep models is known the model gradually gets better in performing a desired task, for over few decades now [14]. However, the bottleneck of e.g. classification; over that data under a specified metric. relatively smaller training data sets had restricted the utility The computational model itself generally takes the form of Deep Learning until recently. (2) Availability of powerful of an Artificial Neural Network (ANN) [8] that consists computational resources: Learning complex functions over of multiple layers of neurons/perceptrons [9] - basic com- large amount of data results in immense computational re- putational blocks, whereas its parameters (a.k.a. network quirements. Related research communities are able to fulfill weights) specify the strength of the connections between the such requirements only recently. (3) Availability of public neurons of different layers. We depict a deep neural network libraries implementing DL algorithms: There is a growing in Fig.1 for illustration. recent trend in different research communities to publish Once trained for a particular task, the DL models are also the source codes on public platforms. Easy public access to able to perform the same task accurately using a variety DL algorithm implementations has exploded the use of this 2

Fig. 1: Illustration of a deep neural network: The network consists of multiple layers of neurons/perceptrons that are connected in an inter-layer fashion. The neurons compute non-linear/linear transformations of their inputs. Each feature of the input signal is weighted by the network parameters and processed by the neuron layers hierarchically. A network connection performs the weighting operation. The strengths of the connections (i.e. network parameter values) are learned using training data. The network layers not seen by the input and output signals are often termed ‘hidden’ layers. technique in many application domains. few review articles [24], [25], [26] for closely related re- The field of Medical Imaging has been exploiting Ma- search directions also exist at the time of this publication. chine Learning since 1960s [15], [16]. However, the first Whereas those articles collectively provide a comprehensive notable contributions that relate to modern Deep Learning overview of the methods exploiting deep neural networks techniques appeared in the Medical Imaging literature in for medical tasks until the year 2017, none of them sees 1990s [17], [18], [19], [20], [21]. The relatedness of these the existing Medical Imaging literature through the lens methods to contemporary DL comes in the form of using of Computer Vision and Machine Learning. Consequently, ANNs to accomplish Medical Imaging tasks. Nevertheless, they also fall short in elaborating on the root causes of restricted by the amount of training data and computational the challenges faced by Deep Learning in Medical Imaging. resources, these works trained networks that were only two Moreover, limited by their narrower perspective, they also to three layers deep. This is no longer considered ‘deep’ in do not provide insights into leveraging the findings in other the modern era. The number of layers in the contemporary fields for addressing these challenges. DL models generally ranges from a dozen to over one hundred [22]. In the context of image analysis, such models In this paper, we provide a comprehensive review of the have mostly originated in Computer Vision literature [2]. recent DL techniques in Medical Imaging, focusing mainly The field of Computer Vision closely relates to Medical on the very recent methods published in 2018. We cate- Imaging in analyzing digital images. Medical Imaging has a gorize these techniques under different pattern recognition long tradition of profiting from the findings in Computer tasks and further sub-categorize them following a taxonomy Vision. In 2012 [23], DL provided a major breakthrough based on human anatomy. Analyzing the reviewed litera- in Computer Vision by performing a very hard image ture, we establish ‘lack of appropriately annotated large- classification task with remarkable accuracy. Since then, scale datasets’ for Medical Imaging tasks as the fundamental the Computer Vision community has gradually shifted its challenge (among other challenges) in fully exploiting Deep main focus to DL. Consequently, Medical Imaging literature Learning for those tasks. We then draw on the literature of also started witnessing methods exploiting deep neural Computer Vision, Pattern Recognition and Machine Learn- networks in around 2013, and now such methods are ap- ing in general; to provide guidelines to deal with this and pearing at an ever increasing rate. Sahiner et al. [24] noted other challenges in Medical Image Analysis using Deep that the peer-reviewed publications that employed DL for Learning. This review also touches upon the available public radiological images tripled from 2016 (∼100) to 2017 (∼300), datasets to train DL models for the medical imaging tasks. whereas the first quarter of 2018 alone saw over 100 such Considering the lack of in-depth comprehension of Deep publications. Similarly, the main stream Medical Imaging Learning framework by the broader Medical community, conference, i.e. International Conference on ‘Medical Image this article also provides the understanding of the core Computing and Computer Assisted Intervention’ (MICCAI) technical concepts related to DL at an appropriate level. This published over 120 papers in its main proceedings in 2018 is an intentional contribution of this work. that employed Deep Learning for Medical Image Analysis tasks. The remaining article is organized as follows. In Sec- The large inflow of contributions exploiting DL in Med- tion2, we present the core Deep Learning concepts for the ical Imaging has also given rise to a specialized venue in Medical community in an intuitive manner. The main body the form of International Conference on ‘Medical Imag- of the reviewed literature is presented in Section3. We touch ing with Deep Learning’ (MIDL) in 20181 that published upon the public data repositories for Medical Imaging in 82 contributions in this direction. Among other published Section4. In Section5, we highlight the major challenges papers in 2018, our literature review also includes notable faced by Deep Learning in Medical Image Analysis. Recom- contributions from MICCAI and MIDL 2018. We note that mendations for dealing with these challenges are discussed in Section6 as future directions. The article concludes in 1. https://midl.amsterdam/ Section7. 3

2 BACKGROUND CONCEPTS Along with supervised and unsupervised learning, other machine learning types include semi-supervised learning and In this Section, we first briefly introduce the broader types reinforcement learning. Informally, semi-supervised learning of Machine Learning techniques and then focus on the Deep computes models using the training data that provides Learning framework. Machine Learning methods are bo- labels only for its smaller subsets. On the other hand, radly categorized as supervised or unsupervised based on the reinforcement learning provides ‘a kind of’ supervision for training data used to learn the computational models. Deep the learning problem in terms of rewards or punishments Learning based methods can fall in any of these categories. to the algorithm. Due to their remote relevance to the tasks 2.0.0.1 Supervised Learning: In supervised learn- in Medical Imaging, we do not provide further discussion ing, it is assumed that the training data is available in the on these categories. Interested readers are directed to [27] m form of pairs (x, y), where x ∈ R is a training example, for semi-supervised learning, and to [28] for reinforcement and y is its label. The training examples generally belong learning. to different, say C classes of data. In that case, y is often C th represented as a binary vector living in R , such that its c 2.1 Standard Artificial Neural Networks coefficient is ‘1’ if x belongs to the cth class, whereas all other coefficients are zero. A typical task for supervised learning is An Artificial Neural Network (ANN) is a hierarchical com- to find a computational model M with the help of training position of basic computational elements known as neurons data such that it is also able to correctly predict the labels of (or perceptrons [9]). Multiple neurons exist at a single level of data samples that it had not seen previously during training. the hierarchy, forming a single layer of the network. Using The unseen data samples are termed test/testing samples in many layers in an ANN makes it deep, see Fig.1. A neuron the Machine Learning parlance. To learn a model that can performs the following simple computation: perform successful classification of the test samples, we a = f(w|x + b), (1) can formulate our learning problem as estimation of the m m parameters Θ of our model that minimizes a specific loss where x ∈ R is the input signal, w ∈ R contains the L(y, yˆ), where yˆ is the label vector predicted by the model neuron’s weights, and b ∈ R is a bias term. The symbol f(.) for a given test sample. As can be guessed, the loss is defined denotes an activation function, and the computed a ∈ R is the such that it has a small value only if the learned parametric neuron’s activation signal or simply its activation. Generally, model is able to predict the correct label of the data sample. f(.) is kept non-linear to allow an ANN to induce complex Whereas the model loss has its scope limited to only a single non-linear computational models. The classic choices for data sample, we define a cost for the complete training f(.) are the well-known ‘sigmoid’ and ‘hyperbolic tangent’ data. The cost of a model is simply the Expected value of functions. We depict a single neuron/perceptron in Fig.2. the losses computed for the individual data samples. Deep A neural network must learn the weight and bias terms Learning allows us to learn model parameters Θ that are in Eq. (1). The strategy used to learn these parameters (i.e. able to achieve very low cost over very large data sets. back-propagation [10]) requires f(.) to be a differentiable Whereas classification generally aims at learning com- function of its inputs. In the modern Deep Learning era, putational models that map input signals to discrete output Rectified Linear Unit (ReLU) [23] is widely used for this values, i.e. class labels. It is also possible to learn models that function, especially for non-standard ANNs such as CNNs a = max(0, w|x + b) can map training examples to continuous output values. In (see Section 2.2). ReLU is defined as . that case, y is typically a real number scalar or vector. An For the details beyond the scope of this discussion, ReLU example of such task is to learn a model that can predict allows more efficient and generally more effective learning the probability of a tumor being benign or malignant. In of complex models as compared to the classic sigmoid and Machine Learning, such a task is seen as a regression problem. hyperbolic tangent activation functions. Similar to the classification problems, Deep Learning has It is possible to compactly represent the weights associ- been able to demonstrate excellent performance in learning ated with all the neurons in a single layer of an ANN as a W ∈ p×m p computational models for the regression problem. matrix R , where ‘ ’ is the total number of neurons in that layer. This allows us to compute the activations of all 2.0.0.2 Unsupervised Learning: Whereas super- the neurons in the layer at once as follows: vised learning assumes that the training data also provides labels of the samples; unsupervised learning assume that a = f(W x + b), (2) sample labels are not available. In that case, the typical task p of a computational model is to cluster the data samples into where a ∈ R now stores the activation values of all the different groups based on the similarities of their intrinsic neurons in the layer under consideration. Noting the heirar- characteristics - e.g. clustering pixels of color images based chical nature of ANNs, it is easy to see that the functional on their RGB values. Similar to the supervised learning form of a model induced by an L-layer network can be given tasks, models for unsupervised learning tasks can also take as: advantage of minimizing a loss function. In the context of M(x, Θ) = fL(W LfL−1(W L−1fL−2...(W 1x+ (3) Deep Learning, this loss function is normally designed such b1) + ... + bL−1) + bL), that the model learns an accurate mapping of an input signal to itself. Once the mapping is learned, the model is used where the subscripts denote the layer numbers. We collec- to compute compact representations of data samples that tively denote the parameters W i, bi, ∀i ∈ {1, ..., L} as Θ. cluster well. Deep Learning framework has also been found A neural network model can be composed by using dif- very effective for unsupervised learning. ferent number of layers, having different number of neurons 4

Fig. 2: Illustration of a single neuron/perceptron in a stan- dard ANN. Each feature/coefficient ‘xi ∀i ∈ {1, ..., m}’ Fig. 3: Illustration of convolution operation for 2D- of the input signal ‘x’ gets weighted by a corresponding grids (left) and 3D-volumes (right). Complete steps of mov- weight ‘wi’. A bias term ‘b’ is added to the weighted sum ing window are shown for the 2D case whereas only step- ‘w|x’ and a non-linear/linear function f(.) is applied to 1 is shown for 3D. In 3D, the convolved channels are compute the activation ‘a’. combined by simple addition, still resulting in 2D feature maps.

in each layer, and even having different activation functions for different layers. Combined, these choices determine the 2.2.0.1 Convolutional layers: The aim of convolu- architecture of a neural network. The design variables of the tion layers is to learn weights of the so-called2 convolutional architecture and those of the learning algorithm are termed kernels/filters that can perform convolution operations on im- as hyper-parameters of the network. Whereas the model pa- ages. Traditional image analysis has a long history of using rameters (i.e. Θ) are learned automatically, finding the most such filters to highlight/extract different image features, e.g. suitable values of the hyper-parameter is usually a manual Sobel filter for detecting edges in images [30]. However, iterative process. Standard ANNs are also commonly known before CNNs, these filters needed to be designed by man- as Multi-Layer Perceptrons (MLPs), as their layers are gen- ually setting the weights of the kernel in a careful manner. erally composed of standard neurons/perceptrons. One no- The breakthrough that CNNs provided is in the automatic table exception to this layer composition is encountered at learning of these weights under the neural network settings. the very last layer, i.e. softmax layer used in classification. In We illustrate the convolution operation in Fig.3. In contrast to ‘independent’ activation computation by each 2D settings (e.g. grey-scale images), this operation involves neuron in a standard perceptron layer, softmax neurons moving a small window (i.e. kernel) over a 2D grid (i.e. im- compute activations that are normalized across all the ac- age). In each moving step, the corresponding elements of tivation values of that layer. Mathematically, the ith neuron the two grids get multiplied and summed up to compute of a softmax layer computes the activation value as: a scalar value. Concluding the operation results in another 2D-grid, referred to as the feature/activation map in the CNN w|a +b e i L−1 i literature. In 3D settings, the same steps are performed for a = . i p (4) the individual pairs of the corresponding channels of the P w|aL−1+bj e j 3D volumes, and the resulting feature maps are simply j=1 added to compute a 2D map as the final output. Since The benefit of normalizing the activation signal is that color images have multiple channels, convolutions in 3D the output of the softmax layer can be interpreted as a settings are more relevant for the modern CNNs. However, probability vector that encodes the confidence of the network for the sake of better understanding, we often discuss the that a given sample belongs to a particular class. This relevant concepts using the 2D grids. These concept are interpretation of softmax layer outputs is a widely used readily transferable to the 3D volumes. concept in the related literature. A convolutional layer of CNN forces the elements of kernels to become the network weights, see Fig.4 for illus- tration. In the figure, we can directly compute the activation a (2D grids) using Eq. (1), where x, w ∈ 9 are vectors 2.2 Convolutional Neural Networks 1 R formed by arranging w1, .., w9 and x1, ..., x9 in the figure. In the context of DL techniques for image analysis, Convolu- The figure does not show the bias term, which is generally tion Neural Networks (CNNs) [23], [29] are of the primary ignored in the convolutional layers. It is easy to see that importance. Similar to the standard ANNs, CNNs consist under this setting, we can make use of the same tools to of multiple layers. However, instead of simple perceptron learn the convolutional kernel weights that we use to learn layers, we encounter three different kinds of layers in these the weights of a standard ANN. The same concept applies networks (a) Convolutional layers, (b) Pooling layers, and (c) Fully connected layers (often termed fc-layers). We de- 2. Strictly speaking, the kernels in CNNs compute cross-correlations. However, they are always referred to as ‘convolutional’ by convention. scribe these layers below, focusing mainly on the Convolu- Our subsequent explanation of the ‘convolution’ operation is in-line tional layers that are the main source of strength for CNNs. with the definition of this operation used in the context of CNNs. 5

Fig. 5: Illustration of unfolded RNN [1]. A state s is shown with a single neuron for simplicity. At the tth time stamp, the network updates its state st based on the input xt and the previous state. It optionally outputs a signal ot. U, V Fig. 4: Working of a convolutional layer. CNNs force kernel and W are network parameters. weights to become network parameters. (Left) In 2D grids, a single kernel moves over the image/input signal. (Right) A volume of multiple kernel moves over the input volume ANNs. The use of multiple Convolutional and Pooling lay- to result in an output volumes. ers in CNNs gradually reduces the size of resulting activa- tion maps. Finally, the activation maps from a deeper layer are re-arranged into a vector which is then fed to the fully to the 3D volumes, with a difference that we must use connected (fc) layers. It is a common knowledge now that ‘multiple’ kernels to get a volume (instead of a 2D grid) the activation vectors of fc-layers often serve as very good at the output. Each feature map resulting from a kernel compact representations of the input signals (e.g. images). then acts as a separate channel of the output volume of Other than the above mentioned three layers, Batch a convolutional layer. It is a common practice in CNN Normalization (BN) [33] is another layer that is now a days literature to simplify the illustrations in 3D settings by only encountered more often in CNNs than in the standard showing the input and output volumes for different layers, ANNs. The main objective of this layer (with learnable as we do in Fig.4. parameters) is to control the mean and variance of the From the perspective presented above, a convolutional activation values of different network layers such that the layer may look very similar to a standard perceptron layer, induction of the overall model becomes more efficient. This discussed in Section 2.1. However, there are two major idea is inspired by a long known fact that induction of differences between the two. (1) Every input feature gets ANN models generally becomes easier if the inputs are connected to its activation signal through the same kernel normalized to have zero mean and unit variance. The BN (i.e. weights). This implies that all input features share the layer essentially applies a similar principle to the activations kernel weights - called parameter sharing. Consequently, the of deep neural networks. kernels try to adjust their weights such that they resonate well to the basic building patterns of the whole input 2.3 Recurrent Neural Networks signal, e.g. edges for an image. (2) Since the same kernel Standard neural networks assume that input signals are connects all input features to output features/activation, independent of each other. However, often this is not the convolutional layers have very few parameters to learn in case. For instance, a word appearing in a sentence generally the form of kernel weights. This sparsity of connections allows depends on sequence of the words preceding it. Recurrent very efficient learning despite the high dimensionality of Neural Networks (RNNs) are designed to model such se- data - a feat not possible with standard densely connected quences. An RNN can be thought to maintain a ‘memory’ perceptron layers. of the sequence with the help of its internal states. In Fig.5, 2.2.0.2 Pooling layers: The main objective of a pool- we show a typical RNN that is unfolded - complete network ing layer is to reduce the width and height of the activation is shown for the sequence. If the RNN has three layers, it maps in CNNs. The basic concept is to compute a single out- can model e.g. sentences that are three words long. In the th put value ‘v’ for a small np × np grid in the activation map, figure, xt is the input at the t time stamp. For instance, xt where ‘v’ is simply the maximum or average value of that can be some quantitative representation of the tth word in a grid in the activation map. Based on the used operation, this sentence. The memory of the network is maintained by the layer is often referred as max-pooling or average-pooling layer. state st that is computed as: Interestingly, there are no learnable parameters associated s = f(Ux + W s ), (5) with a pooling layer. Hence, this layer is sometimes seen as a t t t−1 part of Convolutional layer. For instance, the popular VGG- where f(.) is typically a non-linear activation function, 16 network [31] does not see pooling layer as a separate e.g. ReLU. The output at a given time stamp ot is a function layer, hence the name VGG-16. On the other hand, other of a weighted version the network state at that time. For in- works, e.g. [32] that use VGG-16 often count more than 16 stance, predicting probability of the next word in a sentence layers in this network by treating the pooling layer as a can assume the output form ot = softmax(V st), where regular network layer. ‘softmax’ is the same operation discussed in Section 2.1. 2.2.0.3 Fully connected layers: These layers are the One aspect to notice in the above equations is that we use same as the perceptron layers encountered in the standard the same weight matrices U, V , W at all time stamps. Thus, 6

we are recursively performing the same operations over Regularized autoencoders also impose sparsity on neuron an input sequence at multiple time stamps. This fact also connections [35] and reconstruction of the original signal inspires the name ‘Recursive’ NN. It also has a significant from its noisy vesion [36] to ensure learning of useful latent implication that for an RNN we need a special kind of back- representation instead of identity mapping. Variational au- propagation algorithm, known as back-propagation through toencoders [37] and contractive autoencoders [38] are also time (BTT). As compared to the regular back-propagation, among the other popular types of autoencoders. BTT must propagate error recursively back to the previous time stamps. This becomes problematic for long sequences 2.4.2 Generative Adversarial Networks that involve too many time stamps. A phenomenon known Recent years have seen an extensive use of Generative as vanishing/exploding gradient is the root cause of this prob- Adversarial Networks (GANs) [39] in natural image anal- lem. This has lead RNN researcher to focus on designing ysis. GANs can be considered a variation of autoencoders networks that can handle longer sequences. Long Short- that aim at mimicking the distribution generating the data. Term Memory (LSTM) network [34] is currently a popular GANs are composed of two parts that are neural networks. type of RNN that is found be reasonably effective for deal- The first part, termed generator, has the ability to generate ing with long sequences. a sample whereas the other, called discriminator can classify LSTM networks have the same fundamental architecture the sample as a real or fake. Here, a ‘real’ sample means of an RNN, however their hidden states are computed that it is actually coming from the training data. The two differently. The hidden units are commonly known as cells in networks essentially play a game where the generator tries the context of LSTMs. Informally, a cell takes in the previous to fool the discriminator by generating more and more re- state and the input at a given time stamp and decides on alistic samples. In the process the generator keeps updating what to remember and what to erase from its memory. its parameters to produce better samples. The adversarial The previous state, current input and the memory is then objective of the generator to fool the discriminator also in- combined for the next time stamp. spires the name of GANs. In natural image analysis, GANs have been successfully applied for many tasks, e.g. inducing 2.4 Using Neural Networks for Unsupervised Learning realism in synthetic images [40], domain adaption [41] and data completion [42]. Such successful applications of GANs Whereas we assumed availability of the label for each data to image processing tasks also open new directions for sample while discussing the basic concepts of neural net- medical image analysis tasks. works in the preceding subsections, those concepts can also be applied readily to construct neural networks to model data without labels. Here, we briefly discuss the mainstream 2.5 Best practices in using CNNs for image analysis frameworks that allow to do so. It is noteworthy that this Convolutional Neural Networks (CNNs) form the back- article does not present neural networks as ‘supervised vs bone of the recent breakthroughs in image analysis. To unsupervised’ intentionally. This is because the core con- solve different problems in this area, CNN based models cepts of neural networks are generally better understood in are normally used in three different ways. (1) A network supervised settings. Unsupervised use of neural networks architecture is chosen and trained from scratch using the simply requires to employ the same ideas under different available training dataset in an end-to-end manner. (2) A overall frameworks. CNN model pre-trained on some large-scale dataset is fine- tuned by further training the model for a few epochs using 2.4.1 Autoencoders the data available for the problem at hands. This approach The main idea behind autoencoders is to map an input is more suitable when limited training data is available for signal (e.g. image, feature vector) to itself using a neural net- the problem under consideration. It is often termed transfer work. In this process, we aim to learn a latent representation learning in the literature. (3) Use a model as a feature extrac- of the data that is more powerful for a particular task than tor for the available images. In this case, training/testing the raw data itself. For instance, the learned representation images are passed through the network and the activations could cluster better than the original data. Theoretically, it is of a specific layer (or a combination of layers) are considered possible to use any kind of network layers in autoencoders as image features. Further analysis is performed using those that are used for supervised neural networks. The unique- features. ness of autoencoders comes in the output layer where the Computer Vision literature provides extensive studies to signal is the same as the input signal to the network instead reflect on the best practices of exploiting CNNs in any of the of e.g. a label vector in classification task. aforementioned three manners. We can summarize the crux Mapping a signal to itself can result in trivial mod- of these practices as follows. One should only consider train- els (learning identity mapping). Several techniques have ing a model from scratch if the available training data size is been adopted in the literature to preclude this possibility, very large, e.g. 50K image or more. If this is not the case, use leading to different kinds of autoencoders. For instance, transfer learning. If the training data is even smaller, e.g. few undercomplete autoencoders ensure that the dimensionality hundred images, it may be better to use CNN only as a of the latent representation is much smaller than the data feature extractor. No matter which approach is adopted, it is dimension. In MLP settings, this can be done by using a better that the underlying CNN is inspired by a model that small number (as compared to the input signal’s dimension) has already proven its effectiveness for a similar task. This of neurons in a hidden layer of the network, and use is especially true for the ‘training from scratch’ approach. the activations of that layer as the latent representation. We refer to the most successful recent CNN models in the 7

Computer Vision literature in the paragraphs to follow. For algorithms come pre-implemented in the libraries of the transfer learning, it is better to use a model that is pre- frameworks. trained on data/problem that is as similar as possible to Below we list the popular Deep Learning frameworks the data/problem at hands. In the case of using CNN as in use now a days. We order them based on their current a feature extractor, one should prefer a network with more popularity in Computer Vision and Pattern Recognition representation power. Normally, deeper networks that are community for the problems of image analysis, starting trained on very large datasets have this property. Due to from the most popular. their discriminative abilities, features extracted from such • Tensorflow [49] is originally developed by Google models are especially useful for classification tasks. Brain, it is fast becoming the most popular deep Starting from AlexNet in 2012 [23], many complex learning framework due to its continuous develop- CNN models have been developed in the last seven years. ment. It provides Python and C++ interface. Whereas still useful, AlexNet is no longer considered a state- • PyTorch [50] is a Python based library supported of-the-art network. A network still applied frequently is by Facebook’s AI Research. It is currently receiving VGG-16 [31] that was proposed in 2014 by the Visual Ge- significant attention due to its ability to implement ometry Group (VGG) of Oxford university. A later version dynamic graphs. of VGG-16 is VGG-19 that uses 19 instead of 16 layers of the • Caffe2 [51] builds on Caffe (see below) and provides learnable parameters. Normally, the representation power of C++ and Python interface. both versions are considered similar. Another popular net- • Keras [52] can be seen as a high level programming work is GoogLeNet [43] that is also commonly known as ‘In- interface that can run on top of Tensorflow and ception’ network. This network uses a unique type of layer Theano [53]. Although not as felxible as other frame- called inception layer/block from which it drives its main works, Keras is particularly popular for quickly de- strength. To date, four different versions of Inception [44], veloping and testing networks using common net- [45] have been introduced by the original authors, with each work layers and algorithms. It is often seen as a subsequent version having slightly better representation gateway to deep learning for new users. power (under a certain perspective) than its predecessor. • MatConvNet [54] is the most commonly used public ResNet [22] is another popular network that enables deep deep learning library for Matlab. learning with models having more than hundred layers. It • Caffe [51] was originally developed by UC Berekely, is based on a concept known as ‘residual learning’, which is providing C++ and Python interface. Whereas Caffe2 currently highly favored by Pattern Recognition community is now fast replacing Caffe, this framework is still in because it enables very deep networks. DenseNet [46] also use because public implementations of many popu- exploits the insights of residual learning to achieve the lar networks are available in Caffe. representation power similar to ResNet, but with a more • Theano [53] is a library of Python to implement deep compact network. learning techniques that is developed and supported Whereas the above-mentioned CNNs are mainly trained by MILA, University of Montreal. for image classification tasks, Fully Convolutional Networks • Torch [55] is a library and scripting language based (FCNs) [47] and U-Net [48] are among the most popular on Lua. Due to its first release in 2011, many first networks for the task of image segmentation. Analyzing generation deep learning models were implemented the architectures and hyperparamter settings of these net- using Torch. works can often reveal useful insights for developing new networks. In fact, some of these networks (e.g. Inception- The above is not an exhaustive list of the frameworks v4/ResNet [45]) already rely on the insights from others for Deep Learning. However, it covers those frameworks (e.g. ResNet [22]). The same practice can yield popular that are currently widely used in image analysis. It should networks in the future as well. We draw further on the best be noted, whereas we order the above list in terms of practices of using CNNs for image analysis in Section6. the ‘current’ trend in popularity of the frameworks, public implementations of many networks proposed in 2012 - 2015 are originally found in e.g. Torch, Theano or Caffe. This also 2.6 Deep Learning Programming Frameworks makes those frameworks equally important. However, it is The rise of Deep Learning has been partially enabled by the often possible to find public implementations of legacy net- public access to programming frameworks that implement works in e.g. Tensorflow, albiet not by the original authors. the core techniques in this area in high-level programming languages. Currently, many of these frameworks are be- 3 DEEP LEARNING METHODS IN MEDICAL IMAGE ing continuously maintained by software developers and most of the new findings are incorporated in them rapidly. ANALYSIS Whereas availability of appropriate Graphical Processing In this Section, we review the recent contributions in Med- Unit (GPU) is desired to fully exploit these modern frame- ical Image Analysis that exploit the Deep Learning tech- works, CPU support is also provided with most of them to nology. We mainly focus on the research papers published train and test small models. The frameworks allow their after December 2017, while briefly mentioning the more users to directly test different network architectures and influential contributions from the earlier years. For a com- their hyperparameter settings etc. without the need of ac- prehensive review of the literature earlier than the year 2018, tually implementing the operations performed by the layers we collectively recommend the following articles [24], [25], and the algorithms that train them. The layers and related [26]. Taking more of a Computer Vision/Machine Learning 8

perspective, we first categorize the existing literature under Chiang et al. [65] developed a CAD technique based on ‘Pattern Recognition’ tasks. The literature pertaining to each a 3D CNN for breast cancer detection using Automated task is then further sub-categorized based on the human whole Breast Ultrasound (ABUS) imaging modality. In their anatomical regions. The taxonomy of our literature review approach, they first extracted Volumes of Interest (VOIs) is depicted in Fig.6. through a sliding window technique, then the 3D CNN was applied and tumor candidates were selected based on the 3.1 Detection/Localization probability resulting from the application of 3D CNN to VOIs. In the experiments 171 tumors are used for testing, The main aim of detection is to identify a particular region achieving sensitivities of up to 95%. Dalmics et al. [66] of interest in an image an draw a bounding box around proposed a CNN based CAD system to detect breast cancer it, e.g. brain tumor in MRI scans. Hence, localization is in MRI images. They used 365 MRI scans for training and also another term used for the detection task. In Medical testing, out of which 161 were malignant lesions. They Image Analysis, detection is more commonly referred to as claimed the achieved sensitivity obtained by their technique Computer Aided Detection (CAD). CAD systems are aimed to be better than the existing CAD systems. For the detection at detecting the earliest signs of abnormality in patients. of breast mass in mammography images, Zhang et al. [67] Lung and breast cancer detection can be considered as the developed a Fully Convolutional Network (FCN) based common applications of CAD. end-to-end heatmap regression technique. They demon- strated that mammography data could be used for digi- 3.1.1 Brain tal breast tomosynthesis (DBT) to improve the detection For the anatomic region of brain, Jyoti et al. [56] employed model. They used transfer learning by fine tunning an FCN a CNN for the detection of Alzheimer’s Disease (AD) us- model on mammography images. The approach is tested on ing the MRI images of OASIS data set [57]. The authors tomosynthesis data with 40 subjects, demonstrating better built on two baseline CNN networks, namely Inception- performance as compared to the model trained from scratch v4 [45] and ResNet [22], to categorize four classes of AD. on the same data. These classes include moderate, mild, very mild and non- demented patients. The accuracies reported by the authors 3.1.3 Eye for these classes are 33%, 62%, 75%, and 99%, respectively. It is claimed that the proposed method does not only perform For the anatomical region of eye, Li et al. [68] recently em- well on the used dataset, but it also has the potential to ployed a deep transfer learning approach which fine tunes generalize to ADNI dataset [58]. Chen et al. [59] pro- the VGG-16 model [31] that is pretrained on ImageNet [69] posed an unsupervised learning approach using an Auto- dataset. To detect and classify Age-related Macular Degen- Encoders (AE). The authors investigated lesion detection eration (AMD) and Diabetic Macular Edema (DME) dis- using Variational Auto Encoder (VAE) [37] and Adversarial eases in eye, they used 207,130 retinal Optical Coherence Auto Encoder (AAE) [60]. The analysis is carried out on Tomography (OCT) images. The proposed method achieved BRATS 2015 datasets, demonstrating good results for the 98.6% prediction detection accuracy in retinal images with Aera Under Curve (AUC) metric. 100%. Ambramoff et al. [70] used a CNN based technique to Alaverdyan et al. [61] used a deep neural network for detect Diabetic Retinopathy (DR) in fundus images. They epilepsy lesion detection in multiparametric MRI images. assessed the device IDx-DR X 2.1 in their study using They also stacked convolution layers in an auto-encoders a public dataset [71] and achieve an AUC score of 0.98. fashion and trained their network using the patches of Schlegl et al. [72] employed deep learning for the detection the original images. Their model was trained using the and quantification of Intraretinal Cystoid Fluid (IRC) and data from 75 healthy subjects in an unsupervised manner. Subretinal Fluid (SRF) in retinal images. They employed an For the automated brain tumor detection in MR images auto encoder-decoder formation of CNNs, and used 1,200 Pandaet al. [62] used discriminative clustering method to OCT retinal images for the experiments, achieving AUC of segregate the vital regions of brain such as Cerebro Spinal 0.92 for SRF and AUC of 0.94 for IRC. Fluid (CSF), White Matter (WM) and Gray Matter (GM). In Deep learning is also being increasingly used for di- another study of automatic detection in MR images [63], agnosing retinal diseases [73], [74]. Li et al. [75] trained Laukampet al. used multi-parametric deep learning model a deep learning model based on the Inception architec- for the detection of meningiomas in brain. ture [43] for the identification of Glaucomatous Optic Neu- ropathy (GON) in retinal images. Their model achieved 3.1.2 Breast AUC of 0.986 for distinguishing healthy from GON eyes. In assessing cancer spread, histopathological analysis of Recently, Christopher et al. [76] also used transfer learning Sentinel Lymph Nodes (SLNs) becomes important for with VGG16, Inception v3, and ResNet50 models for the the task of cancer staging. Bejnordi et al. [64] analyzed identification of GON. They used pre-trained models of deep learning techniques for metastases detection in eosin- ImageNet. For their experiments, they used 14,822 Optic stained tissues and hematoxylin tissue sections of lymph Nerve Head (ONH) fundus images of GON or healthy eyes. nodes of the subjects with cancer. The computational results The achieved best performance for identifying moderate- are compared with human pathologist diagnoses. Interest- to-severe GON in the eyes was reported to be AUC value ingly, out of the 32 techniques analysed, the top 5 deep 0.97 with 90% sensitivity and 93% specificity. Khojasteh et learning algorithms arguably out-performed eleven pathol- al. [77] used pre-trained ResNet-50 on DIARETDB1 [78] and ogists. e-Ophtha [79] datasets for the detection of excudates in the 9

Fig. 6: Taxonomy of literature review: The contributions exploiting Deep Learning technology are first categorized according to the underlying Pattern Recognition tasks. Each category is than sub-divided based on the human anatomical region studied in the papers.

retinal images. They reported an accuracy of 98% with 99% Learning (MIL), and in the testing phase, the model pre- sensitivity of detection on the used data. dicts both labels and class specific localization details. Yi et al. [85] presented a scale recurrent network for the detection 3.1.4 Chest of catheter in X-ray images. Their network architecture is For the pulmonary nodule detection in lungs in Com- organised in an auto encoder-decoder manner. In another puted Tomography (CT) images, Zhu et al. [80] proposed study [86], Masood et al. proposed a deep network, termed a deep network called DeepEM. This network uses a 3D DFCNet, for the automatic computer aided lung pulmonary CNN architecture that is augmented with an Expectation- detection. Maximization (EM) technique for the noisily labeled data of Gonzalez et al. [87] proposed a deep network for Electronic Medical Records (EMRs). They used the EM tech- the detection of Chronic Obstructive Pulmonary Disease nique to train their model in an end-to-end manner. Three (COPD) and Acute Respiratory Disease (ARD) prediction datasets were used in their study, including; the LUNA16 in CT images of smokers. They trained a CNN using 7,983 dataset [81] - the largest publicly available dataset for su- COPDGene cases and used logistic regression for COPD pervised nodule detection, NCI NLST dataset3 for weakly detection and ARD prediction. In another study [88], the supervised detection and Tianchi Lung Nodule Detection same group of researchers used deep learning for weakly dataset. supervised lession localization. Recently, Marsiya et al. [89] For the detection of artefacts in Cardiac Magnetic Res- used NLST and LDIC/IDRI [90] datasets for lung nod- onance (CMR) imaging, Oksuz et al. [82] also proposed ule detection in CT images. They proposed a 3D Group- a CNN based technique. Before training the model, they equivariant Convolutional Neural Network (G-CNN) tech- performed image pre-processing by normalization and re- nique for that purpose. The proposed method was exploited gion of interest (ROI) extraction. The authors used a CNN for fast positive reduction in pulmonary lung nodule detec- architecture with 6-convolutional layers (ReLU activations) tion. The authors claim their method performs on-par with followed by 4-pooling layers, 2 fc layers and a softmax standard CNNs while trained using ten times less data. layer to estimate the motion artefact labels. They showed good performance for the classification of motion artefacts 3.1.5 Abdomen in videos. The authors essentially built on the insights of [83] Alensary et al. [91] proposed a deep reinforcement learning in which video classification is done using a spatio-temporal technique for the detection of multiple landmarks with ROIs 3D CNN. in 3D fetal head scans. Ferlaino et al. [92] worked on plan- Zhe et al. [84] proposed a technique for the localization cental histology using deep learning. They classified five and identification of thoraic diseases in public database NIH different classes with an accuracy of 89%. Their model also X-ray4 that comprises 120 frontal view X-ray images with learns deep embedding encoding phenotypic knowledge 14 labels. Their model performs the tasks of localization that classifies five different cell populations and learns inter- and identification simultaneously. They used the popular class variances of phenotype. Ghesu et al. [93] used a large ResNet [22] architecture to build the computational model. data of 1,487 3D CT scans for the detection of anatomic sites, In their model, an input image is passed through the CNN exploiting multi-scale deep reinforcement learning. for feature map extraction, then a max pooling or bi-linear Katzmann et al. [105] proposed a deep learning based interpolation layer is used for resizing the input image by technique for the estimation of Colorectal Cancer (CRC) in a patch slicing layer. Afterwards, fully convolutional layers CT tumor images for early treatment. Their model achieved are used to eventually perform the recognition. For train- high accuracies in growth and survival prediction. Meng et ing, the authors exploit the framework of Multi-Instance al. [106] formulated an automatic shadow detection tech- 3. https://biometry.nci.nih.gov/cdas/datasets/nlst/ nique in 2D ultrasound images using weakly supervised 4. https://www.kaggle.com/nih-chest-xrays/data annotations. Their method highlights the shadow regions 10 TABLE 1: Summary of highly influential papers appeared in 2016 and 2017 (based on Google Scholar citation index in January 2019) that exploit deep learning for Detection/Localization in Medical Image Analysis.

Reference Anatomic Site Image Modality Network type Data Citations Shin et al. [94] (2016) Lung CT CNN - 783 Sirinukunwattana et al. [95] (2016) Abdomen Histopathology CNN 20000+ images 252 Setio et al. [96] (2016) Lung CT CNN 888 images 247 Xu et al. [97] (2016) Breast Histopathology AE 500 images 220 Wang et al. [98] (2016) Breast Histopathology CNN 400 images 182 Kooi et al. [99] (2017) Breast Mammography CNN 45000 images 162 Rajpurkar et al. [100] (2017) Chest X-ray CNN 100000+ images 139 Liu et al. [101] (2017) Breast Histopathology CNN 400 slides 98 Ghafoorian et al. [102] (2017) Brain MRI CNN LUNA16 ISBI 2016 90 Dou et al. [103] (2017) Lung CT 3D CNN 1075 images 40 Zhang et al. [104] (2017) Brain MRI FCN 700 subjects 32

which is particularly useful for the segmentation task. Horie modalities, including Computed Tomography (CT), X-ray, et al. [107] recently applied a CNN technique for easophagal Positron-Emission Tomography (PET), Ultrasound, Mag- cancer detection. They used 8,428 WGD images and attained netic Resonance Imaging (MRI) and Optical Cohrence To- 98% results for the sensitivity. Yasaka et al. [108] used a mography (OCT) etc. Segmentation is the process of parti- deep CNN architecture for the diagnosis of three different tioning an image into different meaningful segments (that phases (noncontrast-agent enhanced, arterial, and delayed) share similar characteristics) through automatic or semi- of masses of liver in dynamic CT images. automatic outlining of the boundaries within the image. In medical imaging, these segments usually commensurate to 3.1.6 Miscellaneous different tissue classes, pathologies, organs or some other Zhang et al. [109] achieved 98.51% accuracy and a local- biological structure [115]. ization error 2.45mm for the detection of inner ear in CT images. They used 3D U-Net [110] to map the whole 3D 3.2.1 Brain image which consists of multiple convolution-pooling layers Related to the anatomical region of brain, Dey et al. [116] that convert the raw input image into the low resolution trained a complementary segmentation network, termed and highly abstracted feature maps. They applied false CompNet, for skull stripping in MRI scans for normal and positive suppression technique in the training process and pathological brain images. The OASIS dataset [117] was used a shape based constraint during training. Rajpurkar et used for the training purpose. In their approach, the features al. [111] recently released a data set MURA which consists of used for segmentation are learned using an encoder-decoder 40,561 images from 14,863 musculoskeletal studies labeled network trained from the images of brain tissues and its by radiologists as either normal or abnormal. The authors complimentary part outside the brain. The approach is used CNN with 169-layers for the detection of normality compared with a plain U-Net and a dense U-Net [118]. The and abnormality in each image study. Li et al. [112] proposed accuracy achieved by the CompNet for the normal images a Neural Conditional Random Field (NCRF) technique for is 98.27%, and for the pathological images is 97.62%. These the metastasis cancer detection in whole slide images. Their results are better than those achieved by [118]. model was trained end-to-end using back-propagation and Zaho et al. [119] proposed a deep learning technique it obtained successful FROC score of 0.8096 in testing us- for brain tumor segmentation by integrating Fully Convo- ing Camelyon16 dataset [113]. Codella et al. [114] recently lutional Networks (FCNs) and Conditional Random Fields organized a challenge at the International Symposium on (CRFs) in a combined framework to achieve segmentation Biomedical Imaging (ISBI), 2017 for skin lession analysis with appearance and spatial consistency. They trained 3 for melanoma detection. The challenge task provided 2,000 segmentation models using 2D image patches and slices. training images, 150 validation images, and 600 images for First, the training is performed for the FCN using image testing. It eventually published the results of 46 submission. patches, then CRF is trained with a Recurrent Neural Net- We refer to [114] for further details on the challenge itself work (CRF-RNN) using image slices. During this phase, and the submissions. We also mention few techniques in the parameters of the FCN are fixed. Afterwards, FCN Table1 related to the task of detection/localization. These and CRF-RNN parameters are jointly fined tuned using methods appeared in the literature in the years 2016-17. the image slice. The authors used the MRI image data Based on Google Scholar’s citation index, these methods are provided by Multimodal Brain Tumor Image Segmentation among the highly influential techniques in the current re- Challenge (BRATS) 2013, BRATS 2015 and BRATS 2016. In lated literature. This article also provides similar summaries their work, Nair et al. [120] used a 3D CNN approach for of the highly influential papers from the years 2016-17 for the segmentation and detection of Multiple Sclerosis (MS) each pattern recognition task considered in the Sections to lesions in MRI sequences. Roy et al. [121] used voxel-wise follow. Baysian FCN for the whole brain segmentation by using Monte Carlo sampling. They demonstrated high accuracies 3.2 Segmentation on four datasets, namely MALC, ADNI-29, CANDI-13, and In Medical Image Analysis, deep learning is being ex- IBSR-18. Robinson et al. [122] also proposed a real-time deep tensively used for image segmentation with different learning approach for the cardiavor MR segmentation. 11

3.2.2 Breast [135] which is the fastest method among the tested ap- In their study, Singh et al. [123] described a conditional proaches. Bai et al. [137] used FCN and RNN for the Generative Adversarial Networks (cGAN) model for breast pixel-wise segmentation of Aortic sequences in MR images. mass segmentation in mammography. Experiments were They trained their model in an end-to-end manner from conducted on Digital Database for Screening Mammogra- sparse annotations by using a weighted loss function. The phy (DDSM) public dataset and private dataset of mammo- proposed method consists of two parts, the first extracts grams from Hospital Universitari Sant Joan de Reus-Spai. features from the FCN using a U-Net architecture [48]. The They additionally used a simpler CNN to classify the seg- second feeds these features to an RNN for segmentation. mented tumor area into different shapes (irregular, lobular, Among the used 500 Arotic MR images (provided by the UK oval and round) and achieved an accuracy of 72% on DDSM Biobank), the study used random 400 images for training, dataset. Zhang et al. [124] exploited deep learning for and the rest were used for testing the models. Another image intensity normalization in breast segmentation task. recent study on semi supervised myocardiac segmentation They used 460 subjects from Dynamic Contrast-enhanced has been conducted by Chartsias et al. [138], which was Magnetic Resonance Imaging (DCEMRI). Each subject con- presented as an oral paper in MICCAI 2018. Their proposed tained one T1 weighted pre-contrast and three T1 weighted network, called Spatial Decomposition Network (SDNet), post-contrast images. Men et al. [125] trained a deep dilated model 2D input images in two representations, namely ResNet for segmentation of Clinical Target Volume (CTV) in spatial representation of myocardiac as a binary mask and breasts. Lee et al. [126] based on fully convolution neural a latent representation of the remaining features in a vector network proposed an automated segmentation technique form. While not being a fully supervised techniques, their for the breast density estimation in mammograms. For the method still achieves remarkable results for the segmenta- evaluation of their approach, they used full-field digital tion task. screening mammograms of 604 subjects. They fine tuned Kervadec et al. [139] proposed a CNN based ENet [140] the pre-trained network for breast density segmentation constrained loss function for segmentation of weakly super- and estimation. The Percent Density (PD) estimation by vised Cardiac images. They achieved 90% accuracy on the 5 their approach showed similarities with BI-RADS density public datasets of 2017 ACDC challenge . Their approach assessment by radiologists and outperformed the then state- is closes the gap between weakly and fully supervised of-the-art computational approaches. segmentation in semantic medical imaging. In another study of cariac CT and MRI images for multi-class image segmen- 3.2.3 Eye tation, Joyce et al. [141] proposed an adversarial approach Retinal blood image segmentation is considered an im- consisting of a shallow UNet like method. They also demon- portant and a challenging task in retinopathology. Zhang strated improved segmentation with an unsupervised cost. et al. [127] used a deep neural network for this pur- LanLonde et al. [142] introduced a CNN based tech- pose by exploiting the U-Net architecture [48] with resid- nique, termed SegCaps, by exploiting the capsule networks ual connection. They shown their results on three public [143] for object segmentation. The authors exploited the datasets STARE [128], CHASEDB1 [129] and DRIVE [130] LUNA16 subset of the LIDC-IDRI database and demon- and achieved an AUC value of 97.99% for the DRIVE strated the effectiveness of their method for analysing CT dataset. De et al. [131] applied deep learning for the retinal lung scans. It is shown that the method achieves better tissue segmentation. They used 14,884 three dimensional segmentation performance as compared to the popular U- OCT images for training their network. Their approach Net. The SegCaps are able to handle large images with size is claimed to be device independent - it maintains seg- 512 × 512. In another study [144] of lung cancer segmen- mentation accuracy while using different device data. In tation using the LUNA16 dataset, Nam et al. proposed a another study of retinal blood vessels, Jebaseeli et al. [132] CNN model using 24 convolution layers, 2 pooling, 2 decon- proposed a method to enhance the quality of retinal vessel volutional layers and one fully connected layer. Similarly, segmentation. They analysed the severity level of diabetic Burlutskiy et al. [145] developed a deep learning framework retinopathy. Their proposed method, Deep Learning Based for lung cancer segmentation. They trained their model SVM (DLBSVM) model uses DRIVE, STARE, REVIEW, HRF, using the scans of 712 patients and tested on the scans of and DRIONS databases for training. Liu et al. [133] proposed 178 patients of fully annotated Tissue Micro-Arrays (TMAs). a semi-supervised learning for retinal layer and fluid region Their model is aimed at finding high potential cancer areas segmentation in retinal OCT B-scans. Adversarial technique in TMA cores. was exploited for the unlabeled data. Their technique re- sembles the U-Net fully convolutional architecture. 3.2.5 Abdomen Roth et al. [146] built a 3D FCN model for automatic se- 3.2.4 Chest mantic segmentation of 3D images. The model is trained on Duan et al. [134] proposed a Deep Nested Level Set (DNLS) clinical Computed Tomography (CT) data, and it is shown technique for the muti-region segmentation of cardiac MR to perform automated multi-organ segmentation of abdom- images in patients with Pulmonary Hypertension (PH). inal CT with 90% average Dice score across all targeted They compared their approach with a CNN method [135] organs. A CNN method, termed Kid-Net, is proposed for and the conditional random field (CRF) CRF-CNN ap- kidney vessels; artery, vein and collecting system (ureter) proach [136]. DNLS is shown to outperform those tech- segmentation by Taha et al. [147]. Their model is trained in niques for all anatomical structures, specifically for my- ocardium. However, it requires more computations than 5. https://www.creatis.insa-lyon.fr/Challenge/acdc/ 12 an end-to-end fashion using 3D CT-volume patches. One technique for 3D image instance segmentation. Their model promising claim made by the authors is that their method is trainable with weak annotations that needs 3D bounding reduces kidney vessels segmentation time from hours to boxes for all instances and full voxel annotations for only a minutes. Their approach uses feature down-sampling and small fractions of instances. Liu et al. [164] employed a novel up-sampling to achieve higher classification and localiza- deep reinforcement learning approach for the segmentation tion accuracies. Their network training methodology also and classification of surgical gesture. Their approach per- handles unbalanced data, and focuses on reducing false forms well on JIGSAW dataset in terms of edit score as com- positives. It is also claimed that the proposed method en- pared to previous similar works. Arif et al. [165] presented a ables high-resolution segmentation with a limited memory deep FCN model called SPNet, as shape predictor for object budget. The authors exploit the findings in [148] for that segmentation. The X-ray images used in their study are purpose. of cervical vertebra. The dataset used in their experiments Oktay et al. [149] recently presented an ‘attention gate’ included 124 training and 172 test images .Their SPNet was model to automatically find the target anatomy of differ- trained for 30 epochs with a batch size of 50 images. ent shapes and sizes. They essentially extended the U-Net Sarker et al. [174] analyzed skin lesion segmentation with model to an attention U-Net model for pancreas segmenta- deep learning. They used autoencoder and decoder net- tion. Their model can be utilized for organ localization and works for feature extraction. The loss function is minimized detection tasks. They used 120 images of CT for training in their work by combining negative Log Likelihood and their model, and 30 images for testing. Overall, the algo- end-point-error for the segmentation of melanoma regions rithm achieves good performance with 2 to 3% increase in with sharp edges. They evaluated their method SLSDeep on dice score as compared to the existing methods. A related ISBI datasets [114], [175] for skin lesion detection, achieving research on pancreas segmentation had been conducted encouraging segmentation results. In another related study previously using dense connection by Gibson et al. [150] of skin cancer, Mirikharaji et it. [176] also developed a deep and sparse convolutions by Heinrich et al. [151], [152], FCN for skin lesion segmentation. They presented good [153]. For multi-organ segmentation (i.e lung, heart, liver, results on ISBI 2017 dataset of dermoscopy images. They bone) in unlabeled X-ray images, Zhang et al. [154] proposed used two fully convolutional networks based on U-Net [48] a Task Driven Generative Adversarial Network (TD-GAN) and ResNet-DUC [22] in their technique. Yuan et al. [177] automated technique. This is an unsupervised end-to-end proposed a deep fully convolutional-deconvolutional neural method for medical image segmentation. They fine tuned network (CDNN) for the automatic skin lesion segmenta- a dense image-to-image network (DI2I) [46], [155] on syn- tion, and acquired Jaccard index of 0.784 on the validation thetic Digitally Reconstructed Radiographs (DRRs) and X- set of ISBI. Ambellan et al. [178] proposed a CNN based ray images. In another study of multi organ segmentation, on 3D Statistical Shape Models (SSMs) for the segmentation Tong et al. [156] proposed an FCN with a shape represen- of knee and cartilage in MRI images. In Table2, we also tation model. Their experiments were carried out on H&N summarize few popular methods in medical image segmen- datasets of volumetric CT scans. tation that appeared prior to the year 2018. Yang et al. [157] used a conditional Generative Adver- sarial Network (cGAN) to segment the human liver in 3D 3.3 Registration CT images. Lessmann et al. [158] proposed an FCN based technique for the automatic vetebra segmentation in CT Image registration is a common task in medical image images. The underlying architecture of their network is analysis that allows spatial alignment of images to a com- inspired by U-Net. Their model is able to process a patch mon anatomical space [179]. It aims at aligning a source size of 128 × 128 × 128 voxels. It achieves 95.8% accuracy image with a target image through transformations. Image for classification and 92.1% for segmentation in the spinal registration is one of the main stream tasks in medical image images used by the authors. Jin et al. [159] proposed a 3D analysis that has received ample attention even before the CGAN to learn lung nodules conditioned on a Volume deep learning era [180], [181], [182], [183], [184], [185], [186]. Of Interest (VOI) with an erased central region in 3D CT Advent of deep learning has also caused neural networks images. They trained their model on 1,000 nodules taken to penetrate in medical image registration [187], [188], [189], from LIDC dataset. The proposed CGAN was further used [190]. to generate a dataset for Progressive Holistically Nested Network (P-HNN) model [160] which demonstrates im- 3.3.1 Brain proved segmentation performance. Van et al. [191] proposed a stacked bidirectional convolu- tional LSTM (C-LSTM) network for the reconstruction of 3.2.6 Miscellaneous 3D images from the 4D spatio-temporal data. Previously, For memory and computational efficiency, Xu et al. [161] [192], [193] used CNN techniques for the reconstruction of applied a quantization mechanism to FCNs for the seg- 3D CT and MRI images using four 3D convolutional layers. mentation tasks in Medical Image Analysis. They also Lonning et al. [194] presented a deep learning method using used quantization to mitigate the over fitting issue for Recurrent Inference Machines (RIM) for the reconstruction better performance. The effectiveness of the developed of MRI. Deep learning based deformable image registration method is demonstrated for 2015 MICCAI Gland Challenge has also been recently performed by Sheikh et al. [195]. dataset [162]. As compared to [163] their method improves They used deep FCN to generate spatial transformations the results by up to 1% with 6.4× reduction in the memory through under feed forward networks. In their experiments, usage. Recently, Zhao et al. [119] proposed a deep learning they used cardiac MRI images from ACDC 2017 dataset 13 TABLE 2: Summary of influential papers appeared in 2016 and 2017 (based on Google Scholar citation index in January 2019) that exploit deep learning for the Segmentation tasks in Medical Image Analysis.

Reference Anatomic Site Image Modality Network type Data Citations Milletari et al. [148] (2016) Brain MRI FCN Private data 550 Ciceck et al. [110] (2016) Kidney CT 3D U-Net Private data 460 Pereira et al. [166] (2016) Brain MRI CNN BRATS2013, 2015 411 Moeskops et al. [167] (2016) Brain MRI CNN 5 datasets 242 Liskowski et al. [168] (2016) Eye Opthalmology DNN 400 images, DRIVE, STARE, CHASE 177 Ibragimov et al. [169] (2016) Breast CT AE 1400 images 174 Havaei et al. [170] (2017) Brain MRI CNN 2013 BRATS 596 Kamnistas et al. [171](2017) Brain MRI 11 layers 3D-CNN BRATS 2015 and ISLES 2015 517 Fang et al. [172] (2017) Eye OCT CNN 60 volumes 86 Ibragimov et al. [173](2017) Liver CT CNN 50 images 56 and showed promising results in comparison to a moving method. In their method they first extracted feature points mesh registration technique. Hou et al. [196] also proposed by speed up robust feature (SURF) detector and partial a learning based image registration technique using CNNs. intensity invariant feature descriptor (PIIFD) from model They used 2D image slice transformation to construct 3D and target retinal image. Then they used MFSP for feature images using a canonical co-ordinate system. First, they map detection. simulated their approach on fetal MRI images and then used Pan et al. [203] developed a deep learning technique real fetal brain MRI images for the experiments. Their work to remove eye position difference for longitudinal 3D is also claimed to be promising for computational efficiency. retinal OCT images. In their method, they first perform In another study of image based registration [197], the pre-processing for projection image then, to detect vessel same group of authors evaluated their technique on CT shadows, they apply enhancement filters. The SURF al- and MRI datasets for different loss functions using SE(3) gorithm [204] is then used to extract the feature points, as a benchmark. They trained CNN directly on SE(3) and whereas RANSAC [205] is applied for cleaning the outliers. proposed a Riemannian manifold based formulation for Mahapatra et al. [206] also proposed an end-to-end deep pose estimation problem. The registration accuracy with learning technique for image registration. They used GANs their approach increased from 2D to 3D image based regis- which registered images in a single pass with deformation tration as compared to previous methods. The authors fur- field. They used ADAM optimizer [11] for minimizing the ther showed in [198] that CNN can reliably reconstruct 3D network cost and trained the model using the Mean Square images using 2D image slices. Recently, Balakrishnan et al. Error (MSE) loss. [199] worked on 3D pairwise MR brain image registration. They proposed an unsupervised learning technique named 3.3.3 Chest VoxelMorph CNN. They used a pair of two 3D images Relating to the anatomical region of chest, Eppenhof et as input, with dimensions 160 × 192 × 224; and learned al. [207] proposed a 3D FCN based technique for the shared parameters in their network for convolution layers. registration of CT lung inspiration-expiration image pairs. They demonstrated their method on 8 publicly available They validated the performance of their method using datasets of brain MRI images. On ABIDE dataset their two datasets, namely DIRLAB [208] and CREATIS [209]. model achieved 1.5% improvement in the dice score. It In general, there is a growing perception in the Medical is claimed that their method is also computationally more Imaging community that Deep learning is a promising tool efficient than the exiting techniques for this problem. for 2D and 3D image registration for the chest regions. De et al. [210] also trained a CNN model for the affine 3.3.2 Eye and deformable image registration. Their technique allows Costa et al. [200] used adversarial autoencoders for the syn- registration of the pairs of unseen images in a single pass. thesis of retinal colored images. They trained a generative They applied their technique to cardiac cine MRI and chest model to generate synthetic images and another model CT images for registration. Zheng et al. [211] trained a to classify its output into a real or synthetic. The model CNN model for 2D/3D image registration problem under results in an end-to-end retinal image synthesis system a Pairwise Domain Adaptation (PDA) technique that uses and generates as many images as required by its users. synthetic data. It is claimed that their method can learn It is demonstrated that the image space learned by the effective representations for image registration with only model has an arguably well defined semantic structure. a limited number of training images. They demonstrated The synthesized images were shown to be visually and generalization and flexibility of their method for clinical quantitatively different from the images used for training applications. Their PDA method can be specially suitable their model. The shown images reflect good visual quality. where small training data is available. Mahapatra et al. [201] proposed an end-to-end deep learning method using generative adversarial networks for multi- 3.3.4 Abdomen modal image registration. They used retinal and cardiac Lv et al. [212] proposed a CNN based technique for the images for registration. Tang et al. [202] demonstrated a 3D MRI abdomen image registration. They trained their robust image registration approach based on mixture feature model for the spatial transformation analysis of different and structure preservation (MFSP) non rigid point matching images. To demonstrate the effectiveness of their technique, 14

they compared their method with three other approaches (AD) patients. They reported up to 99% accuracy on ADNI and claimed a reduction in the reconstruction time from dataset. They exploited Transfer Learning to handle the data 1h to 1 minute. In another related study, Lv et al. [213] scarcity issue, and used a model that is pre-trained with the proposed a deep learning framework based on the popular help of CAD Dementia dataset. Their network architecture U-net architecture. To evaluate the performance of their is based on 3D convolutional kernels that models generic technique they used 8 ROI’s from cortex and medulla of brain features from sMRI data. The overall classification segmented kidney. It is demonstrated by the authors that process in their technique first spatially normalizes the brain during free breathing measurements, their normalized root- sMRI data, then it learns the 3D CNN model using the mean-square error (NRMSE) values for cortex and medulla normalized data. The model is eventually fine-tuned on were significantly lower after registration. the target domain, where the fine-tuning is performed in a supervised manner. 3.3.5 Miscellaneous Recently, Yan et al. [230] proposed a deep chronectome Yan et al. [214] presented an Adversarial Image Registration learning framework for the classification of MCI in brain (AIR) method for multi-modal image MR-TRUS registra- using Full Bidirectional Long Short-Term Memory (Full- tion [215]. They trained two deep networks concurrently, BiLSTM) networks. Their method can be divided into two one for the generator component of the adversarial frame- parts, firstly a Full-LSTM is used to gather time varying work, and the other for the discriminator component. In information in brain for which MCI can be diagnosed. their work, the authors learned not only an image regis- Secondly, to mine the contextual information hidden in tration network but also a so-called metric network which dFC, they applied BiLSTM to access long range context computes the quality of image registration. The data used in both directions. They reported the performance of their in their experimentation consists of 763 sets of 3D TRUS model on public dataset ADNI-2,achieving 73.6% accuracy. volume and 2D MR volume with 512x512x26 voxels. The Hensfeld et al. [231] also proposed a deep learning algorithm developed AIR network is also evaluated on clinical datasets for the Autism Spectrum disorder (ASD) classification in acquired through image-fusion guided prostate biopsy pro- rs-fMRI images on multi-site database ABIDE. They used cedures. For the visualization of 3D medical image data denoising autoencoders for unsupervised pretraining. The Zhao et al. [216] recently proposed a deep learning based classification accuracy achieved by their algorithm on the technique, named Respond-weighted Class Activation Map- said dataset is 70%. ping (Respond-CAM). As compared to Grade-CAM [217] In the context of classification related to the anatomical they claim better performance. Elss et al. [218] also employed region of brain, Soussia et al. [232] provided a review of 28 Convolutional networks for single phase image motion in papers from 2010 to 2016 published in MICCAI. They re- cardiac CT 2D/3D images. They trained regression network viewed neuroimaging-based technical methods developed to successfully learn 2D motion estimation vectors. We also for the Alzheimer Disease (AD) and Mild-Cognitive Im- summarize few worth noting contributions from the years pairment (MCI) classification tasks. The majority of papers 2016 and 2017 in Table3. used MRI for dementia classification and few worked to predict MCI conversion to AD at later observations. We refer to [232] for the detailed discussions on the contributions 3.4 Classification reviewed by this article. Gutierrez et al. [233] proposed a Classification of images is a long standing problem in Med- deep neural network, termed Multi-structure point network ical Image Analysis and other related fields, e.g. Computer (MSPNet), for the shape analysis on multiple structures. Vision. In the context of medical imaging, classification This network is inspired by PointNet [234] that can directly becomes a fundamental task for Computer Aided Diagnosis process point clouds. MSPNet achieves good classification (CAD). Hence, it is no surprise that many researchers have accuracy for AD and MCI for the ADNI database. recently tried to exploit the advances of deep learning for this task in medical imaging. 3.4.2 Breast Awan et al. [235] proposed to use more context information 3.4.1 Brain for breast image classification. They used features of a CNN Relating to the anatomical region of Brain, Li et al. [228] used that is pre-trained on ICIAR 2018 dataset for histological deep learning to detect Autism Spectrum disorder (ASD) images [236], and classified breast cancer as benign, carci- in functional Magnetic Resonance Imaging (fMRI). They noma insitu (CIS) or breast invasive carcinoma (BIC). Their developed a 2-stage neural network method. For the first technique performs patch based and context aware image stage, they trained a CNN (2CC3D) with 6 convolutional classification. They used ResNet50 architecture and over- layers, 4 max-pooling layers and 2 fully connected layers. lapping patches of size 512 × 512. The extracted features are Their network uses a sigmoid output layer. For the second classified using a Support Vector Machine in their approach. stage, in order to detect biomarkers for ASD, they took Due to the unavailability of large-scale data, they used advantage of the anatomical structure of brain fMRI. They random rotation and flipping data augmentation techniques developed a frequency normalized sampling method for during the training process. It is claimed that their trained that purpose. Their method is evaluated using multiple model can also be applied to other tasks where contextual databases, showing robust results for neurological function information and high resolution are required for optimal of biomarkers. prediction. In their work, Hosseini-Asl et al. [229] employed an auto- When only weak annotations are available for images, encoder architecture for diagnosing Alzheimer’s Disease such as in heterogeneous images, it is often useful to turn 15 TABLE 3: Summary of influential papers appeared in 2016 and 2017 (based on Google Scholar citation index in January 2019) that exploit deep learning for the Registration task in Medical Image Analysis.

Reference Anatomic Site Imaging Modality Network type Data Citations Miao et al. [219] (2016) Chest X-ray CNN regression Synthetic 101 Wu et al. [220] (2016) Brain MRI CNN LONI, ADNI databases 58 Simonvosky et al. [221] (2016) - Multiple CNN IXI 57 Yang et al. [222] (2016) Brain MRI Encoder-decoder OASIS dataet 40 Barahmi et al. [193] (2016) Brain MRI CNN 15 subjects 36 Zhang et al. [223] (2016) Head, Abdomen, Chest CT CNN Private 30 Nie et al. [224] (2017) Multi task MRI, CT FCN Private 110 Kang et al. [225] (2017) Abdomen CT CNN CT low-dose Grand Challenge 98 Yang et al. [190] (2017) Brain MRI Encoder-decoder 373 OASIS and 375 IBIS images 64 De et al. [226] (2017) Brain MRI CNN Sunnybrook Cardiac Data [227] 50

to multiple instance learning (MIL). Courture et al. [237] 3.4.4 Chest described a CNN using quantile function for the classi- fication of 5 types of breast tumor histology. They fine- tuned AlexNet [23]. The data used in their study consists Dey et al. [249] studied 3D CNNs for the diagnostic classi- of 1,713 images from the Carolina Breast Cancer Study, fication of lung cancer between benign and malignant in Phase 3 [238]. They improved the classification accuracy CT images. Four networks were analyzed for their clas- on this dataset from 68.6 to 85.6 for estrogen receptor (ER) sification task, namely a basic 3D CNN; a multi-output task in breast images. Recently, MIL has also been used CNN; a 3D DenseNet, and an augmented 3D DenseNet with for breast cancer classification in [239] and [240] that per- multi-outputs. They employed the public dataset LIDC- form patch based classification of histopathology images. IDRI with 1, 010 CT images and a private dataset of 47 CT Antropova et al. [241] used 690 cases with 5 fold cross- images with both malignant and benign in this study. The validation of MRI maximum intensity projection for breast best results are achieved by the 3D multi-output DenseNet lession classification. They used a pre-trained VGGNet [31] (MoDenseNet) on both datasets, having accuracy 90.40% as for feature extraction, followed by an SVM classifier. Ribli compared to previously reported accuracy of 89.90% [250]. et al. [242] applied a CNN based on VGG16 for lession Gao et al. [251] proposed a deep CNN for the classification classification in mammograms. They trained their model of Interstitial Lung Disease (IDL) patterns on CT images. using DDSM dataset and tested it on INbreast [243]. They Previously, batch based algorithms were being used for this achieved the second best score for Mammography DREAM purpose [252], [253]. In contrast, Gao et al. performed holistic Challenge, with AUC of 0.95. Zheng et al. [244] proposed a classification using the complete image as network input. CAD technique for breast cancer classification using CNN Their experiments used a public data [254] on which the based on pre-trained VGG-19 model. They evaluated their classification accuracy improved to 87.9% from the previous technique’s performance on digital mammograms of pairs results of 86.1% [253]. For the holistic image classification, of 69 cancerous and 27 healthy subjects. They achieved the the overall accuracy of 68.8% was achieved. In another values of 0.928 for sensitivity and 0.991 for specificity of work, Hoo et al. [94] et al. also analyzed three CNN ar- classification. chitectures, namely CifarNet, AlexNet, and GoogLeNet, for interstitial lung disease classification. Biffi et al. [255] proposed a 3D convolutional genera- 3.4.3 Eye tive model for the classification of cardiac diseases. They achieved impressive performance for the classification of Pertaining to the region of eye, Gergeya et al. [71] took a data healthy and hypertrophic cardiomyopathy MR images. For driven approach using deep learning to classify Diabetic ACDC MICCAI 2017 dataset they were able to achieve retinopathy (DR) in color fundus images. The authors used 90% accuracy for classification of healthy subjects. Chen public databases MESSIDOR 2 and E-ophtha to train and et al. [256] proposed a CNN based technique RadBot-CXR test their models and achieved 0.94 and 0.95 AUC score to categorize focal lung opacities, diffuse lung opacity, car- respectively on the test partitions of these datasets. A con- diomegaly, and abnormal hilar prominence in chest X-ray volutional network is also employed by Pratt et al. [245] images. They claim that their algorithm showed radiologists for diagnosing and classifying the severity of DR in color level performance for this task. Wang et al. [257] used fundus images. Their model is trained using the Kaggle deep learning in analyzing histopathology images for the datasets, and it achieved 75% DR severicity accuracy. Simi- whole slide lung cancer classification. Coudray et al. [258] larly, Ayhan et al. [246] also exploited the deep CNN archi- used Inception3 CNN model to analyze whole slide images tecture of ResNet50 [247] for the fundus image classification. to classify lung cancer between adenocarcinoma (LUAD), Mateen et al. [248] proposed a DR classification system based squamous cell carcinoma (LUSC) or normal tissues. More- on VGG-19. They also performed evaluation using Kaggle over, they also trained their model for the prediction of dataset of 35,126 fundus images. It is claimed that their ten most common mutated genes in LUAD and achieved model outperforms the more conventional techniques, e.g. good accuracies. Masood et al. [86] proposed a deep learning SIFT as well as earlier deep networks, e.g. AlexNet in terms approach DFCNet based on FCN, which is used to classify of accuracy for the same task. the four stages of detected pulmonary lung cancer nodule. 16

3.4.5 Abdomen typical examples of the commonly used datasets in med- Relating to abdomen, Tomczak et al. [259] employed deep ical imaging by deep learning approaches in Table5. The Multiple Instance Learning (MIL) framework [260] for the Table is not intended to provide an exhaustive list. We classification of esophageal cancer in histopathology im- recommend the readers internet search for that purpose. ages. In another contribution, Frid et al. [261] used GANs A brief search can result in a long compilation of medical for the synthetic medical image data generation. They imaging datasets. However, we summarize few examples made classification of CT liver images as their test bed of contemporary datasets in Table5 to make an important and performed classification of 182 lesions. The authors point regarding deep learning research in the context of demostrated that by using augmented data with the GAN Medical Image Analysis. With the exception of few datasets, framework, upto 7% improvement is possible in classifi- the public datasets currently available for medical imaging cation accuracy. For automatic classification of ultrasound tasks are small in terms of the number of samples and abdominal images, Xu et al. [262] proposed a multi-task patients. As compared to the datasets for general Computer learning framework based on CNN. For the experiments Vision problems, where datasets typically range from few they used 187,219 ultrasound images and claimed better hundred thousand to millions of annotated images, the classification accuracy than human clinical experts. dataset sizes for Medical imaging tasks are too small. On the other hand, we can see the emerging trend in Medical 3.4.6 Miscellaneous Imaging community of adopting the practices of broader Esteva et al. [263] presented a CNN model to classify skin Pattern Recognition community, and aiming at learning cancer lesions. Their model is trained in an end-to-end man- deep models in end-to-end fashion. However, the broader ner directly from the images, taking pixels and disease labels community has generally adopted such practices based on as inputs. They used datasets of 129,450 clinical images the availability of large-scale annotated datasets, which is an consisting of 2,032 different diseases to train CNN. They important requirement for inducing reliable deep models. classified two most common cancer diseases; keratinocyte Hence, it remains to be seen that how effectively end-to- carcinomous versus benign seborrheic keratosis, and the end trained models can really perform the medical image deadliest skin cancer; malignant melanomas versus benign analysis tasks without over-fitting to the training datasets. nevi. For skin cancer sun exposure classification, Combalia et al. [264] also applied a Monte Carlo method to only high- 5 CHALLENGESIN GOING DEEP light the most elistic regions. Antony et al. [265] employed a deep CNN model for automatic quantification of severity of In this Section, we discuss the major challenges faced in fully knee osteoarthritis (OA). They used Kellgren and Lawrence exploiting the powers of Deep Learning in Medical Image (KL) grades to assess the severity of knee. In their work, Analysis. Instead of describing the issues encountered in using deep CNN pre-trained on ImageNet and fine-tuned specific tasks, we focus more on the fundamental challenges on knee OA images resulted in good classification perfor- and explain the root causes of these problems for the Medi- mance. Paserin et al. [266] recently worked on diagnosis cal Imaging community that can also help in understanding and classification of developmental dysplasia of hip (DDH). the task-specific challenges. Dealing with these challenges is They proposed a CNN-RNN technique for 3D ultrasound the topic of discussion in Section6. volumes for DDH. Their model consists of convolutional 5.0.0.1 Lack of appropriately annotated data: It can layers for feature learning followed by recurrent layers for be argued that the single aspect of Deep Learning that sets spatial relationship of their responses. Inspired by VGG it apart from the rest of Machine Learning techniques is its network [31], they used CNN with 5 convolutional layers ability to model extremely complex mathematical functions. for feature extraction with ReLU activations, and 2 × 2 max- Generally, we introduce more layers to learn more complex pooling with a stride of 2. Finally, they used LSTM network models - i.e. go deep. However, a deeper network must also that has 256 units. They achieved 82% accuracy with AUC learn more model parameters. A model with a large number 0.83 for 20 test volumes. Few notable contributions from the of parameters can only generalize well if we correspond- years 2016 and 2017 are also summarized in Table4. ingly use a very large amount of data to infer the parameter values. This phenomenon is fundamental to any Machine Learning technique. A complex model inferred using a 4 DATASETS limited amount of data normally over-fits to the used data Good quality data has always remained the primary re- and performs poorly on any other data. Such modeling is quirement for learning reliable computational models. This highly undesirable because it gives a false impression of is also true for deep models that also have the additional learning the actual data distribution whereas the model is requirement of consuming large amount of training data. only learning the peculiarities of the used training data. Recently, many public datasets for medical imaging tasks Learning deep models is inherently unsuitable for the have started to emerge. There is also a growing trend in domains where only limited amount of training data is the research community to compile lists of these datasets. available. Unfortunately, Medical Imaging is one such do- For instance we can find few useful compilation of public main. For most of the problems in Medical Image Analysis, dataset lists at Github repositories and other webpages. there is only a limited amount of data that is annotated in a Few medical image analysis products are also helping in manner that is suitable to learn powerful deep models. We providing public datasets. Whereas detailed discussion on encounter the problem of ‘lack of appropriately annotated the currently available public datasets for medical imag- data’ so frequently in the current Deep Learning related ing tasks is outside the scope of this article, we provide Medical Imaging literature that it is not difficult to single out 17 TABLE 4: Summary of notable contributions appearing in 2016 and 2017 that exploit deep learning for the Classification task in Medical Image Analysis. The citation index is based on Google Scholar (January 2019).

Reference Anatomic Site Image Modality Network type Data Citations Anthimopoulos et al. [267] (2016) Lung CT CNN Uni. Hospital of Geneva [254]& Inselspital 253 Kallenberg et al. [268] (2016) Breast Mammography CNN 3 different databases 111 Huynh et al. [269] (2016) Breast Mammography CNN Private data (219 lesions) 76 Yan et al. [270] (2016) 12 regions CT CNN Synthetic & Private 73 Zhang et al. [223] (2016) Breast Elastography MLP Private data (227 images) 50 Esteva et al. [263] (2017) Skin Histopatholohy CNN Private large-scale 1386 Sun et al. [271] (2017) Breast Mammography CNN Private data 40 Christodoulidis et al. [272] (2017) Lung CT CNN Texture data as Transfer learning source 36 Lekadir et al. [273] (2017) Heart US CNN Private data 26 Nibali et al. [250] (2017) Lung CT CNN LIDC/IDRI dataset 22

TABLE 5: Examples of popular databases used by Medical Image Analysis techniques that exploit Deep Learning.

Sr. Database Anatomic site Image modality Main task Patients/Images 1 ILD[254] Lung CT Classification 120 patients 2 LIDC-IDRI[90] Lung CT Detection 1,018 patients 3 ADNI[58] Brain MRI Classification ¿800 patients 4 MURA[111] Musculoskeletal X-ray Detection 40,561 images 5 TCIA Multiple Multiple Multiple ¿35,000 patients 6 BRATS[274] Brain MRI Segmentation - 7 DDSM[275] Breast Mamography Segmentation 2,620 patients 8 MESSIDOR-2[276], [277] Eye OCT Classification 1,748 images 9 ChestX-ray14[278] Chest X-ray Multiple ¿100,000 images 10 ACDC 2017 Brain MRI Classification 150 patients 11 2015 MICCAI Gland Challenge Glands Histopathology Segmentation 165 images 12 OAI Knee X-ray, MRI Multiple 4,796 patients 13 DRIVE[128], [279] Eye SLO Segmentation 400 patients 14 STARE[130] Eye SLO Segmentation 400 images 15 CHASEDB1[129] Eye SLO Segmentation 28 images 16 OASIS-3[57], [280], [281], [282], [283], [284], [117] Brain MRI, PET Segmentation 1,098 patients 17 MIAS[285] Breast Mammography Classification 322 patients 18 ISLES 2018 Brain MRI Segmentation 103 patients 19 HVSMR 2018[286] Heart CMR Segmentation 4 patients 20 CAMELYON17[113] Breast WSI Segmentation 899 images 21 ISIC 2018 Skin JPEG Detection 2,600 images 22 OpenNeuro Brain Multiple Classification 4,718 patients 23 ABIDE Brain MRI Disease Diagnosis 1,114 patients 24 INbreast[243] Breast Mammography Detection/Classification 410 images this problem as ‘the fundamental challenge’ that Medical scale datasets. Moreover, the required annotations for deep Imaging community is currently facing in fully exploiting models often do not perfectly align with the general medical the advances in Deep Learning. routines. This becomes an additional problem even for the The Computer Vision community has been able to take experts to provide noise-free annotations. full advantage of Deep Learning because data annotation Due to the primary importance of large-scale training is relatively straightforward in that domain. Simple crowd datasets in Deep Learning there is an obvious need to sourcing can yield millions of accurately annotated images. develop such public datasets for Medical Imaging tasks. This is not possible for Medical Images that require high However, considering the practical challenges in accom- level of specific expertise for annotation. Moreover, the plishing this goal it is also imperative to simultaneously stakes are also very high due to the nature of medical develop techniques of exploiting Deep Learning with less application, requiring extra care in annotation. Although we amount of data. We provide discussion on future directions can also find large number of images in medical domain via along both of these dimension in Section6. systems like PACS and OIS, however using them to train 5.0.0.2 Imbalanced data: One problem that occurs deep models is still not easy because they lack appropriate much more commonly in Medical Imaging tasks as com- level of annotations that is generally required for training pared to general Computer Vision tasks is the imbalance useful deep models. of samples in datasets. For instance, a dataset to train a With only a few exceptions, e.g. [278] the public datasets model for detecting breast cancer in mammograms may available in the Medical Imaging domain are not large- contain only a limited number of positive samples but a very scale - a requirement for training effective deep models. large number of negative samples. Training deep networks In addition to the issues of hiding patient’s privacy, one with imbalanced data can induce models that are biased. major problem in forming large-scale public datasets is that Considering the low frequency of occurrences of positive the concrete labels required for computational modeling samples for many Medical Imaging tasks, balancing out the can often not be easily inferred from medical reports. This original data can become as hard as developing large-scale is problematic if inter-observers are used to create large- dataset. Hence, extra care must be taken in inducing deep 18 models for Medical Imaging tasks. Zamir et al. [290] recently showed that performance 5.0.0.3 Lack of confidence interval: Whereas the of transfer learning can be improved by carefully select- Deep Learning literature often refers to the output of a ing the source and target domains/tasks. By organizing model as ‘prediction confidence’; the output signal of a neu- different tasks that let the deep models transfer well be- ron can only be interpreted as a single probability value. The tween themselves, they developed a so-called ‘taskonomy’ lack of provision of confidence interval around a predicted to guide the use of transfer learning for natural images. value is generally not desirable in Medical Imaging tasks. This concept has received significant appreciation in the Litjens et al. [25] has noted that an increasing number of Computer Vision community, resulting in the ‘best paper’ deep learning methods in Medical Imaging are striving to award for the authors at the prestigious IEEE Conference on learn deep models in an end-to-end manner. Whereas end- Computer Vision and Pattern Recognition (CVPR), 2018. A to-end learning is the epitome of Deep Learning, it is not similar concept is worth exploring for the data deprived certain if this is the right way to exploit this technology Medical Imaging tasks. Disentangling medical tasks for in Medical Imaging. To an extent, this conundrum is also transfer learning may prove very beneficial. Another related hindering the widespread use of Deep Learning in Medical research direction that can help in dealing with smaller Imaging. data size is to quantify the suitability of transfer learning between medical imaging and natural imaging tasks. A definitive understanding of the knowledge transfer abilities 6 FUTUREDIRECTIONS of the existing natural image models to the medical tasks With the recent increasing trend of exploiting Deep Learning can have a huge impact in Medical Image Analysis using in Medical Imaging tasks, we are likely to see a large influx Deep Learning. of papers in this area in the near future. Here, we provide guidelines and directions to help those works in dealing 6.1.2 Wrapping Deep Features for Medical Imaging Tasks with the inherent challenges faced by Deep Learning in The existing literature shows an increasing trend of training Medical Image Analysis. We draw our insights from the deep models for Medical tasks in an ‘end-to-end’ manner. reviewed literature and the literature in the sister fields of For Deep Learning, end-to-end modeling is generally more Computer Vision, Pattern Recognition and Machine Learn- promising for the domains where large-scale annotated ing. Due to the earlier use of Deep Learning in those fields, data is available. Exploiting the existing deep models as the techniques of dealing with the related challenges have feature extractors and then performing further learning on considerably matured in those areas. Hence, Medical Image those features is a much more promising direction in the Analysis can readily benefit from those findings in setting absence of large-scale training datasets. There is a consid- fruitful future directions. erable evidence in the Pattern Recognition literature that Our discussion in this Section is primarily aimed at activation signals of deeper layers in neural networks often providing guiding principles for the Medical Imaging com- form highly expressive image features. For natural images, munity. Therefore, we limit it to the fundamental issues Akhtar et al. [291] demonstrated that features extracted from in Deep Learning. Based on the challenges mentioned in deep models can be used to learn further effective higher the preceding Section, and the insights from the parallel level features using the techniques that require less training scientific fields, we present our discussion along three di- samples. They used Dictionary Learning framework [292] rections, addressing the following major questions. (1) How to further wrap the deep features before using them with a can Medical Image Analysis still benefit from Deep Learning classifier. in the absence of large-scale annotated datasets? (2) What We note that Medical Image Analysis literature has can be done for developing large-scale Medical Imaging already seen reasonably successful attempts of using the datasets. (3) What should be the broader outlook of this existing natural image deep models as feature extractors research direction to catapult it in taking full advantage of e.g. [293], [294], [295]. However, such attempts generally the advances in Deep Learning? directly feed the features extracted from the pre-trained models to a classifier. The direction we point towards entail 6.1 Dealing with smaller data size post-processing of deep features to better suit the require- ments of the underlying Medical Image Analysis task. 6.1.1 Disentangling Medical Task Transfer Learning Considering the obvious lack of large-scale annotated 6.1.3 Training Partially Frozen Deep Networks datasets, Medical Imaging community has already started As a general principle in Machine Learning, more amount exploiting ‘transfer learning’ [287], [288], [289]. In transfer of training data is required to train more complex com- learning, one can learn a complex model using data from putational models. In Deep Learning, the network depth a source domain where large-scale annotated images are normally governs the model complexity, whereas deeper available (e.g. natural images). Then, the model is further networks also have more parameters that require large-scale fine-tuned with data of the target domain where only a small datasets for training. It is known that the layers of CNNs number of annotated images are available (e.g. medical - the most relevant neural networks for image analysis - images). It is clear from the literature that transfer learn- systematically break down images into their features from ing is proving advantageous for Medical Image Analysis. lower level of abstraction to higher level of abstraction [296]. Nevertheless, one promising recent development in transfer It is also known that the initial layers of CNNs learn very learning [290] in Computer Vision literature remains com- similar filters for a variety of natural images. These observa- pletely unexplored for Medical Image Analysis. tions point towards the possibility of reducing the number 19 of learn-able parameters in a network by freezing few of its appear the same as the original images to humans, layers to the parameter values that are likely to be similar for however, a deep model is not able to recognize a variety of images. Those parameter values can be directly them. Whereas developing such images is a different borrowed from other networks trained on similar tasks. The research direction, one finding from that direction remainder of the network - that now has less parameters is that including those images in training data can but has the same complexity - can then be trained for the improve the performance of deep models [300]. Since target task as normal. Training partially frozen networks for adversarial examples are generated from the original Medical Imaging task can mitigate the issues caused by the images, they provide a useful data augmentation lack of large-scale annotated datasets. method that can be harnessed for Medical Imaging tasks. 6.1.4 Using GANs for Synthetic Data Generation • Rotation and random noise addition: In the context of Generative Adversarial Networks (GANs) [39] are currently 3D data, rotating the 3D scans and adding small receiving tremendous attention of Computer Vision commu- amount of random noise (emulating jitters) are also nity for their ability to mimic the distributions from which considered useful data augmentation strategies [234]. images are sampled. Among other uses of GANs, one can use the GAN framework to generate realistic synthetic im- 6.2 Enhancing dataset sizes ages for any domain. These images can then be used to train Whereas the techniques discussed in Section 6.1 can alle- deeper models for that domain that generally outperform viate the issues caused by smaller training datasets, the the models trained with only (limited) original data. This root cause of those problems can only be eliminated by property of GANs is of particular interest for Medical Image acquiring Deep Learning compatible large-scale annotated Analysis. Therefore, we can expect to see a large number of datasets for Medical Image Analysis tasks. Considering that future contributions in Medical Imaging that will exploit Deep Learning has started outperforming human experts in GANs. In fact, our literature review already found few Medical Image Analysis tasks [301], there is a strong need initial applications of GANs in medical image analysis [297], to implement protocols that make medical reports readily [298], [200], [299]. However, extra care should be taken convertible to the formats useful for training computational while exploiting the GAN framework. It should be noted models, especially Deep Learning models. In this context, that GANs do not actually learn the original distribution techniques from the fields of Document Analysis [302] and of images, rather they only mimic it. Hence, the synthetic Natural Language Processing (NLP) [303] can be used to images generated from GANs can still be very different from alleviate the extra burden falling on the medical experts due the original images. Therefore, instead of training the final to the implementations of such protocols. model with the data that includes GAN-generated data, it Besides generating the new data at large-scales that is often better to finally fine-tune such model with only the is useful in learning computational models, it is also im- original images. portant to take advantage of the existing medical records for exploiting the current advances in Deep Learning. To 6.1.5 Miscellaneous Data Augmentation Techniques handle the large volume of un-organized data (in terms In general, Computer Vision and Pattern Recognition lit- of compatibility with Machine Learning), data mining with erature has also developed few elementary data augmen- humans-in-the-loop [304] and active learning [27] can prove tation techniques that have shown improvement in the beneficial. Advances in Document Analysis and NLP can performance of deep models. Whereas these techniques are also be exploited for this task. generally not as effective as sophisticated methods, such as using GANs to increase data samples; they are still worth 6.3 Broader outlook taking advantage of. We list the most successful techniques We can make one important observation regarding Deep below. Again, we note that some of these methods have Learning research by exploring the literature of different already proven their effectiveness in the context of Medical research fields. That is, the advancement in Deep Learn- Image Analysis: ing research has often experienced a quantum leap under • Image flipping: A simple sideways flip of images the breakthroughs provided by different sister fields. For doubles the number of training samples, that often example, the ‘residual learning’ concept [22] that enabled results in a better model. For medical images, top- very deep networks was first introduced in the literature of down flip is also a possibility due to the nature of Computer Vision. This idea (along with other breakthroughs images. in core Machine Learning research) eventually enabled the • Image cropping: Cropping different areas of a larger tabula rasa algorithm of AlphaGo Zero [305]. Following up image into smaller images and treating each one on this observation, we can argue that significant advances of the cropped versions as an original image also can be made in Deep Learning research in the context of benefits deep models. Five crops of equal dimensions Medical Image Analysis if researchers from the sister fields from an image is a popular strategy in Computer of Computer Vision and Machine Learning are able to better Vision literature. The crops are made using the four understand the Medical Image Analysis tasks. corners and the central region of the image. Indeed, Medical Imaging community already involves • Adversarial training: Very recently, it is discovered that experts from other related fields. However, this involve- we can ‘fool’ deep models using adversarial images [2]. ment is at a smaller scale. For the involvement of broader These images are carefully computed such that they Machine Learning and Computer Vision communities, a 20 major hindrance is the Medical Imaging literature jargon. provide an intuitive understanding of the core concepts in Medical literature is not easily understood by the experts of Deep Learning for the Medical community, we also high- other fields. One effective method to mitigate this problem light the root cause of the challenges faced in this direction can be regular organization of Medical Imaging Workshops and recommend fruitful future directions by drawing on the and Tutorials in the reputed Computer Vision and Machine insights from multiple scientific fields. Learning Conferences, e.g. IEEE CVPR, ICCV, NeurIPS and From the reviewed literature, we can single out the ‘lack ICML. These events should particularly focus on playing the of large-scale annotated datasets’ as the major problem for role of translating the Medical Imaging problems to other Deep Learning in Medical Image Analysis. We have dis- communities in terms of their topics of interest. cussed and recommended multiple strategies for the Medi- Another effective strategy to take advantage of Deep cal Imaging community that are adopted to address similar Learning advances is to outsource the Medical Imaging problems in the sister scientific fields. We can conclude that problems by organizing online challenges, e.g. Kaggle com- Medical Imaging can benefit significantly more from Deep petitions. The authors are already aware of few Kaggle Learning by encouraging collaborative research with Com- competitions related to Medical Imaging, e.g. Histopatho- puter Vision and Machine Learning research communities. logic cancer detection. However, we can easily notice that Medical Imaging competitions are normally attracting fewer teams as compared to other competitions - currently 361 REFERENCES for Histopathologic cancer detection. Generally, the number [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. of teams are orders of magnitude lower for the Medical 521, no. 7553, p. 436, 2015. Imaging competitions than those for the typical imaging [2] N. Akhtar and A. Mian, “Threat of adversarial attacks on deep competitions. In authors’ opinion, strict Medical parlance learning in computer vision: A survey,” IEEE Access, vol. 6, pp. adopted in organizing such competitions is the source of this 14 410–14 430, 2018. [3] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence problem. Explanation of Medical Imaging tasks using the learning with neural networks,” in Advances in neural information terms more common among Computer Vision and Machine processing systems, 2014, pp. 3104–3112. Learning communities can greatly help in improving the [4] T. Ciodaro, D. Deva, J. De Seixas, and D. Damazio, “Online particle detection with neural networks based on topological popularity of Medical Imaging in those communities. calorimetry information,” in Journal of physics: conference series, In short, one of the key strategies to fully exploit Deep vol. 368, no. 1. IOP Publishing, 2012, p. 012030. Learning advances in Medical Imaging is to get the experts [5] H. Y. Xiong, B. Alipanahi, L. J. Lee, H. Bretschneider, D. Merico, from other fields, especially Computer Vision and Machine R. K. Yuen, Y. Hua, S. Gueroussov, H. S. Najafabadi, T. R. Hughes et al., “The human splicing code reveals new insights into the Learning; to involve in solving Medical Imaging tasks. To genetic determinants of disease,” Science, vol. 347, no. 6218, p. that end, the Medical Imaging community must put an 1254806, 2015. extra effort in making its literature, online competitions [6] M. Helmstaedter, K. L. Briggman, S. C. Turaga, V. Jain, H. S. and the overall outlook of the filed more understandable Seung, and W. Denk, “Connectomic reconstruction of the inner plexiform layer in the mouse retina,” Nature, vol. 500, no. 7461, to the experts from the other fields. Deep Learning is being p. 168, 2013. dubbed as ‘modern electricity’ by the experts. In the future, [7] J. Ma, R. P. Sheridan, A. Liaw, G. E. Dahl, and V. Svetnik, its ubiquitous nature will benefit those fields the most that “Deep neural nets as a method for quantitative structure–activity relationships,” Journal of chemical information and modeling, vol. 55, are better understood by the wider communities. no. 2, pp. 263–274, 2015. [8] R. J. Schalkoff, Artificial neural networks, vol. 1. [9] F. Rosenblatt, “Principles of neurodynamics. perceptrons and the 7 CONCLUSION theory of brain mechanisms,” CORNELL AERONAUTICAL LAB INC BUFFALO NY, Tech. Rep., 1961. This article presented a review of the recent literature in [10] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Deep Learning for Medical Imaging. It contributed along representations by back-propagating errors,” nature, vol. 323, no. three major directions. First, we presented an instructive 6088, p. 533, 1986. introduction to the core concepts of Deep Learning. Keeping [11] D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” arXiv preprint arXiv:1412.6980, 2014. in view the general lack of understanding of Deep Learning [12] N. Qian, “On the momentum term in gradient descent learning framework among Medical Imaging researchers, we kept algorithms,” Neural networks, vol. 12, no. 1, pp. 145–151, 1999. our discussion intuitive. This part of the paper can be un- [13] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep derstood as a tutorial of Deep Learning concepts commonly learning. MIT press Cambridge, 2016, vol. 1. [14] R. Beale and T. Jackson, Neural Computing-an introduction. CRC used in Medical Imaging. The second part of the paper Press, 1990. presented a comprehensive overview of the approaches in [15] H. Becker, W. Nettleton, P. Meyers, J. Sweeney, and C. Nice, Medical Imaging that employ Deep Learning. Due the avail- “Digital computer determination of a medical diagnostic index directly from chest x-ray images,” IEEE Transactions on Biomedical ability of other review articles until the year 2017, we mainly Engineering, no. 3, pp. 67–72, 1964. focused on the literature published in the year 2018. The [16] G. S. Lodwick, T. E. Keats, and J. P. Dorst, “The coding of roent- third major part of the article discussed the major challenges gen images for computer analysis as applied to lung cancer,” faced by Deep Learning in Medical Image Analysis, and Radiology, vol. 81, no. 2, pp. 185–200, 1963. [17] Y. Wu, M. L. Giger, K. Doi, C. J. Vyborny, R. A. Schmidt, and C. E. discussed the future directions to address those challenges. Metz, “Artificial neural networks in mammography: application Beside focusing on the very recent literature, this article to decision making in the diagnosis of breast cancer.” Radiology, is also different from the existing related literature surveys vol. 187, no. 1, pp. 81–87, 1993. in that it provides a Computer Vision/Machine Learning [18] S.-C. Lo, S.-L. Lou, J.-S. Lin, M. T. Freedman, M. V. Chien, and S. K. Mun, “Artificial convolution neural network techniques perspective to the use of Deep Learning in Medical Image and applications for lung nodule detection,” IEEE Transactions Analysis. Using that perspective, we are not only able to on Medical Imaging, vol. 14, no. 4, pp. 711–718, 1995. 21

[19] B. Sahiner, H.-P. Chan, N. Petrick, D. Wei, M. A. Helvie, D. D. [42] R. A. Yeh, C. Chen, T.-Y. Lim, A. G. Schwing, M. Hasegawa- Adler, and M. M. Goodsitt, “Classification of mass and normal Johnson, and M. N. Do, “Semantic image inpainting with deep breast tissue: a convolution neural network classifier with spatial generative models.” in CVPR, vol. 2, no. 3, 2017, p. 4. domain and texture images,” IEEE transactions on Medical Imag- [43] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, ing, vol. 15, no. 5, pp. 598–610, 1996. D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with [20] H.-P. Chan, S.-C. B. Lo, B. Sahiner, K. L. Lam, and M. A. convolutions,” in Proceedings of the IEEE conference on computer Helvie, “Computer-aided detection of mammographic microcal- vision and pattern recognition, 2015, pp. 1–9. cifications: Pattern recognition with an artificial neural network,” [44] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, Medical Physics, vol. 22, no. 10, pp. 1555–1567, 1995. “Rethinking the inception architecture for computer vision,” in [21] W. Zhang, K. Doi, M. L. Giger, R. M. Nishikawa, and R. A. Proceedings of the IEEE conference on computer vision and pattern Schmidt, “An improved shift-invariant artificial neural network recognition, 2016, pp. 2818–2826. for computerized detection of clustered microcalcifications in [45] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception- digital mammograms,” Medical Physics, vol. 23, no. 4, pp. 595– v4, inception-resnet and the impact of residual connections on 601, 1996. learning.” in AAAI, vol. 4, 2017, p. 12. [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning [46] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, for image recognition,” in Proceedings of the IEEE conference on “Densely connected convolutional networks.” in CVPR, vol. 1, computer vision and pattern recognition, 2016, pp. 770–778. no. 2, 2017, p. 3. [23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifi- [47] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional net- cation with deep convolutional neural networks,” in Advances in works for semantic segmentation,” in Proceedings of the IEEE neural information processing systems, 2012, pp. 1097–1105. conference on computer vision and pattern recognition, 2015, pp. 3431–3440. [24] B. Sahiner, A. Pezeshk, L. M. Hadjiiski, X. Wang, K. Drukker, K. H. Cha, R. M. Summers, and M. L. Giger, “Deep learning in [48] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional medical imaging and radiation therapy,” Medical physics, 2018. networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted inter- [25] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, vention. Springer, 2015, pp. 234–241. M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. [49] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, Sanchez,´ “A survey on deep learning in medical image analysis,” M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: a Medical image analysis, vol. 42, pp. 60–88, 2017. system for large-scale machine learning.” in OSDI, vol. 16, 2016, [26] J. Ker, L. Wang, J. Rao, and T. Lim, “Deep learning applications in pp. 265–283. medical image analysis,” IEEE Access, vol. 6, pp. 9375–9389, 2018. [50] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, [27] X. Zhu, “Semi-supervised learning literature survey,” Computer Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic Science, University of Wisconsin-Madison, vol. 2, no. 3, p. 4, 2006. differentiation in pytorch,” 2017. [28] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement [51] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, learning: A survey,” Journal of artificial intelligence research, vol. 4, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture pp. 237–285, 1996. for fast feature embedding,” in Proceedings of the 22nd ACM [29] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based international conference on Multimedia. ACM, 2014, pp. 675–678. learning applied to document recognition,” Proceedings of the [52] F. Chollet et al., “Keras,” https://keras.io, 2015. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. [53] Theano Development Team, “Theano: A Python framework [30] I. Sobel and G. Feldman, “A 3x3 isotropic gradient operator for for fast computation of mathematical expressions,” arXiv image processing,” a talk at the Stanford Artificial Project in, pp. e-prints, vol. abs/1605.02688, May 2016. [Online]. Available: 271–272, 1968. http://arxiv.org/abs/1605.02688 [31] K. Simonyan and A. Zisserman, “Very deep convolutional [54] A. Vedaldi and K. Lenc, “Matconvnet: Convolutional neural networks for large-scale image recognition,” arXiv preprint networks for matlab,” in Proceedings of the 23rd ACM international arXiv:1409.1556, 2014. conference on Multimedia. ACM, 2015, pp. 689–692. [32] S. Sun, N. Akhtar, H. Song, A. Mian, and M. Shah, “Deep [55] R. Collobert, S. Bengio, and J. Mariethoz,´ “Torch: a modular affinity network for multiple object tracking,” arXiv preprint machine learning software library,” Idiap, Tech. Rep., 2002. arXiv:1810.11780, 2018. [56] J. Islam and Y. Zhang, “Early diagnosis of alzheimers disease: [33] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep A neuroimaging study with deep learning architectures,” in network training by reducing internal covariate shift,” arXiv Proceedings of the IEEE Conference on Computer Vision and Pattern preprint arXiv:1502.03167, 2015. Recognition Workshops, 2018, pp. 1881–1883. [34] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” [57] D. S. Marcus, A. F. Fotenos, J. G. Csernansky, J. C. Morris, and Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. R. L. Buckner, “Open access series of imaging studies: longi- tudinal mri data in nondemented and demented older adults,” [35] C. Poultney, S. Chopra, Y. L. Cun et al., “Efficient learning of Journal of cognitive neuroscience, vol. 22, no. 12, pp. 2677–2684, sparse representations with an energy-based model,” in Advances 2010. in neural information processing systems, 2007, pp. 1137–1144. [58] R. C. Petersen, P. Aisen, L. A. Beckett, M. Donohue, A. Gamst, [36] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, D. J. Harvey, C. Jack, W. Jagust, L. Shaw, A. Toga et al., “Stacked denoising autoencoders: Learning useful representa- “Alzheimer’s disease neuroimaging initiative (adni): clinical Journal tions in a deep network with a local denoising criterion,” characterization,” Neurology, vol. 74, no. 3, pp. 201–209, 2010. of machine learning research, vol. 11, no. Dec, pp. 3371–3408, 2010. [59] X. Chen and E. Konukoglu, “Unsupervised detection of lesions [37] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in brain mri using constrained adversarial auto-encoders,” arXiv arXiv preprint arXiv:1312.6114, 2013. preprint arXiv:1806.04972, 2018. [38] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contrac- [60] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey, “Ad- tive auto-encoders: Explicit invariance during feature extraction,” versarial autoencoders,” arXiv preprint arXiv:1511.05644, 2015. in Proceedings of the 28th International Conference on International [61] Z. Alaverdyan, J. Jung, R. Bouet, and C. Lartizien, “Regularized Conference on Machine Learning. Omnipress, 2011, pp. 833–840. siamese neural network for unsupervised outlier detection on [39] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- brain multiparametric magnetic resonance imaging: application Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adver- to epilepsy lesion screening,” 2018. sarial nets,” in Advances in neural information processing systems, [62] A. Panda, T. K. Mishra, and V. G. Phaniharam, “Automated 2014, pp. 2672–2680. brain tumor detection using discriminative clustering based mri [40] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and segmentation,” in Smart Innovations in Communication and Compu- R. Webb, “Learning from simulated and unsupervised images tational Sciences. Springer, 2019, pp. 117–126. through adversarial training.” in CVPR, vol. 2, no. 4, 2017, p. 5. [63] K. R. Laukamp, F. Thiele, G. Shakirin, D. Zopfs, A. Faymonville, [41] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krish- M. Timmer, D. Maintz, M. Perkuhn, and J. Borggrefe, “Fully nan, “Unsupervised pixel-level domain adaptation with genera- automated detection and segmentation of meningiomas using tive adversarial networks,” in The IEEE Conference on Computer deep learning on routine multiparametric mri,” European radi- Vision and Pattern Recognition (CVPR), vol. 1, no. 2, 2017, p. 7. ology, vol. 29, no. 1, pp. 124–132, 2019. 22

[64] B. E. Bejnordi, M. Veta, P. J. Van Diest, B. Van Ginneken, [82] I. Oksuz, B. Ruijsink, E. Puyol-Anton,´ A. Bustin, G. Cruz, C. Pri- N. Karssemeijer, G. Litjens, J. A. Van Der Laak, M. Hermsen, eto, D. Rueckert, J. A. Schnabel, and A. P. King, “Deep learning Q. F. Manson, M. Balkenhol et al., “Diagnostic assessment of deep using k-space based data augmentation for automated cardiac mr learning algorithms for detection of lymph node metastases in motion artefact detection,” in International Conference on Medical women with breast cancer,” Jama, vol. 318, no. 22, pp. 2199–2210, Image Computing and Computer-Assisted Intervention. Springer, 2017. 2018, pp. 250–258. [65] T.-C. Chiang, Y.-S. Huang, R.-T. Chen, C.-S. Huang, and R.-F. [83] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, Chang, “Tumor detection in automated breast ultrasound using “Learning spatiotemporal features with 3d convolutional net- 3-d cnn and prioritized candidate aggregation,” IEEE Transactions works,” in Proceedings of the IEEE international conference on com- on Medical Imaging, vol. 38, no. 1, pp. 240–249, 2019. puter vision, 2015, pp. 4489–4497. [66] M. U. Dalmıs¸, S. Vreemann, T. Kooi, R. M. Mann, N. Karssemeijer, [84] Z. Li, C. Wang, M. Han, Y. Xue, W. Wei, L.-J. Li, and F.-F. Li, and A. Gubern-Merida,´ “Fully automated detection of breast “Thoracic disease identification and localization with limited cancer in screening mri using convolutional neural networks,” supervision,” arXiv preprint arXiv:1711.06373, 2017. Journal of Medical Imaging, vol. 5, no. 1, p. 014502, 2018. [85] X. Yi, S. Adams, P. Babyn, and A. Elnajmi, “Automatic catheter [67] J. Zhang, E. H. Cain, A. Saha, Z. Zhu, and M. A. Mazurowski, detection in pediatric x-ray images using a scale-recurrent net- “Breast mass detection in mammography and tomosynthesis work and synthetic data,” arXiv preprint arXiv:1806.00921, 2018. via fully convolutional network-based heatmap regression,” in [86] A. Masood, B. Sheng, P. Li, X. Hou, X. Wei, J. Qin, and D. Feng, Medical Imaging 2018: Computer-Aided Diagnosis, vol. 10575. In- “Computer-assisted decision support system in pulmonary can- ternational Society for Optics and Photonics, 2018, p. 1057525. cer detection and stage classification on ct images,” Journal of [68] F. Li, H. Chen, Z. Liu, X. Zhang, and Z. Wu, “Fully automated biomedical informatics, vol. 79, pp. 117–128, 2018. detection of retinal disorders by image-based deep learning,” [87] G. Gonzalez,´ S. Y. Ash, G. Vegas-Sanchez-Ferrero,´ Graefe’s Archive for Clinical and Experimental Ophthalmology, pp. J. Onieva Onieva, F. N. Rahaghi, J. C. Ross, A. D´ıaz, R. San 1–11, 2019. Jose´ Estepar,´ and G. R. Washko, “Disease staging and prognosis [69] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Im- in smokers using deep learning in chest computed tomography,” agenet: A large-scale hierarchical image database,” in Computer American journal of respiratory and critical care medicine, vol. 197, Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference no. 2, pp. 193–203, 2018. on. Ieee, 2009, pp. 248–255. [88] C. Gonzalez-Gonzalo,´ B. Liefers, B. van Ginneken, and C. I. [70] M. D. Abramoff,` Y. Lou, A. Erginay, W. Clarida, R. Amelon, Sanchez,´ “Improving weakly-supervised lesion localization with J. C. Folk, and M. Niemeijer, “Improved automated detection iterative saliency map refinement,” 2018. of diabetic retinopathy on a publicly available dataset through [89] M. Winkels and T. S. Cohen, “3d g-cnns for pulmonary nodule integration of deep learning,” Investigative ophthalmology & visual detection,” arXiv preprint arXiv:1804.04656, 2018. science, vol. 57, no. 13, pp. 5200–5206, 2016. [90] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. [71] R. Gargeya and T. Leng, “Automated identification of diabetic Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. retinopathy using deep learning,” Ophthalmology, vol. 124, no. 7, Hoffman et al., “The lung image database consortium (lidc) and pp. 962–969, 2017. image database resource initiative (idri): a completed reference [72] T. Schlegl, S. M. Waldstein, H. Bogunovic, F. Endstraßer, database of lung nodules on ct scans,” Medical physics, vol. 38, A. Sadeghipour, A.-M. Philip, D. Podkowinski, B. S. Gerendas, no. 2, pp. 915–931, 2011. G. Langs, and U. Schmidt-Erfurth, “Fully automated detection and quantification of macular fluid in oct using deep learning,” [91] A. Alansary, O. Oktay, Y. Li, L. Le Folgoc, B. Hou, G. Vaillant, Ophthalmology, vol. 125, no. 4, pp. 549–558, 2018. B. Glocker, B. Kainz, and D. Rueckert, “Evaluating reinforcement learning agents for anatomical landmark detection,” 2018. [73] D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan et al., “Identi- [92] M. Ferlaino, C. A. Glastonbury, C. Motta-Mejia, M. Vatish, fying medical diagnoses and treatable diseases by image-based I. Granne, S. Kennedy, C. M. Lindgren, and C. Nellaker,˚ “Towards deep learning,” Cell, vol. 172, no. 5, pp. 1122–1131, 2018. deep cellular phenotyping in placental histology,” arXiv preprint [74] U. Food, D. Administration et al., “Fda permits marketing of ar- arXiv:1804.03270, 2018. tificial intelligence-based device to detect certain diabetes-related [93] F.-C. Ghesu, B. Georgescu, Y. Zheng, S. Grbic, A. Maier, J. Horneg- eye problems,” News Release, April, 2018. ger, and D. Comaniciu, “Multi-scale deep reinforcement learning [75] Z. Li, Y. He, S. Keel, W. Meng, R. T. Chang, and M. He, “Efficacy for real-time 3d-landmark detection in ct scans,” IEEE transactions of a deep learning system for detecting glaucomatous optic on pattern analysis and machine intelligence, vol. 41, no. 1, pp. 176– neuropathy based on color fundus photographs,” Ophthalmology, 189, 2019. 2018. [94] S. Hoo-Chang, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, [76] M. Christopher, A. Belghith, C. Bowd, J. A. Proudfoot, M. H. J. Yao, D. Mollura, and R. M. Summers, “Deep convolutional Goldbaum, R. N. Weinreb, C. A. Girkin, J. M. Liebmann, and neural networks for computer-aided detection: Cnn architectures, L. M. Zangwill, “Performance of deep learning architectures and dataset characteristics and transfer learning,” IEEE transactions on transfer learning for detecting glaucomatous optic neuropathy medical imaging, vol. 35, no. 5, p. 1285, 2016. in fundus photographs,” Scientific reports, vol. 8, no. 1, p. 16685, [95] K. Sirinukunwattana, S. E. A. Raza, Y.-W. Tsang, D. R. Snead, 2018. I. A. Cree, and N. M. Rajpoot, “Locality sensitive deep learning [77] P. Khojasteh, L. A. P. Junior,´ T. Carvalho, E. Rezende, B. Aliah- for detection and classification of nuclei in routine colon cancer mad, J. P. Papa, and D. K. Kumar, “Exudate detection in fundus histology images,” IEEE transactions on medical imaging, vol. 35, images using deeply-learnable features,” Computers in biology and no. 5, pp. 1196–1206, 2016. medicine, vol. 104, pp. 62–69, 2019. [96] A. A. A. Setio, F. Ciompi, G. Litjens, P. Gerke, C. Jacobs, S. J. [78]R.K alvi¨ ainen¨ and H. Uusitalo, “Diaretdb1 diabetic retinopathy Van Riel, M. M. W. Wille, M. Naqibullah, C. I. Sanchez,´ and database and evaluation protocol,” in Medical Image Understand- B. van Ginneken, “Pulmonary nodule detection in ct images: false ing and Analysis, vol. 2007. Citeseer, 2007, p. 61. positive reduction using multi-view convolutional networks,” [79] E. Decenciere,` G. Cazuguel, X. Zhang, G. Thibault, J.-C. Klein, IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1160–1169, F. Meyer, B. Marcotegui, G. Quellec, M. Lamard, R. Danno et al., 2016. “Teleophta: Machine learning and image processing methods for [97] J. Xu, L. Xiang, Q. Liu, H. Gilmore, J. Wu, J. Tang, and A. Madab- teleophthalmology,” Irbm, vol. 34, no. 2, pp. 196–203, 2013. hushi, “Stacked sparse autoencoder (ssae) for nuclei detection on [80] W. Zhu, Y. S. Vang, Y. Huang, and X. Xie, “Deepem: Deep breast cancer histopathology images,” IEEE transactions on medical 3d convnets with em for weakly supervised pulmonary nodule imaging, vol. 35, no. 1, pp. 119–130, 2016. detection,” arXiv preprint arXiv:1805.05373, 2018. [98] D. Wang, A. Khosla, R. Gargeya, H. Irshad, and A. H. Beck, [81] A. A. A. Setio, A. Traverso, T. De Bel, M. S. Berens, C. van den “Deep learning for identifying metastatic breast cancer,” arXiv Bogaard, P. Cerello, H. Chen, Q. Dou, M. E. Fantacci, B. Geurts preprint arXiv:1606.05718, 2016. et al., “Validation, comparison, and combination of algorithms [99] T. Kooi, G. Litjens, B. van Ginneken, A. Gubern-Merida,´ C. I. for automatic detection of pulmonary nodules in computed to- Sanchez,´ R. Mann, A. den Heeten, and N. Karssemeijer, “Large mography images: the luna16 challenge,” Medical image analysis, scale deep learning for computer aided detection of mammo- vol. 42, pp. 1–13, 2017. graphic lesions,” Medical image analysis, vol. 35, pp. 303–312, 2017. 23

[100] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, and demented older adults,” Journal of cognitive neuroscience, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya et al., “Chexnet: vol. 19, no. 9, pp. 1498–1507, 2007. Radiologist-level pneumonia detection on chest x-rays with deep [118] J. Kleesiek, G. Urban, A. Hubert, D. Schwarz, K. Maier-Hein, learning,” arXiv preprint arXiv:1711.05225, 2017. M. Bendszus, and A. Biller, “Deep mri brain extraction: a 3d [101] Y. Liu, K. Gadepalli, M. Norouzi, G. E. Dahl, T. Kohlberger, convolutional neural network for skull stripping,” NeuroImage, A. Boyko, S. Venugopalan, A. Timofeev, P. Q. Nelson, G. S. Cor- vol. 129, pp. 460–469, 2016. rado et al., “Detecting cancer metastases on gigapixel pathology [119] X. Zhao, Y. Wu, G. Song, Z. Li, Y. Zhang, and Y. Fan, “A images,” arXiv preprint arXiv:1703.02442, 2017. deep learning model integrating fcnns and crfs for brain tumor [102] M. Ghafoorian, N. Karssemeijer, T. Heskes, M. Bergkamp, segmentation,” Medical image analysis, vol. 43, pp. 98–111, 2018. J. Wissink, J. Obels, K. Keizer, F.-E. de Leeuw, B. van Ginneken, [120] T. Nair, D. Precup, D. L. Arnold, and T. Arbel, “Exploring E. Marchiori et al., “Deep multi-scale location-aware 3d convo- uncertainty measures in deep networks for multiple sclerosis lutional neural networks for automated detection of lacunes of lesion detection and segmentation,” in International Conference presumed vascular origin,” NeuroImage: Clinical, vol. 14, pp. 391– on Medical Image Computing and Computer-Assisted Intervention. 399, 2017. Springer, 2018, pp. 655–663. [103] Q. Dou, H. Chen, L. Yu, J. Qin, and P.-A. Heng, “Multilevel [121] A. G. Roy, S. Conjeti, N. Navab, and C. Wachinger, “Inherent contextual 3-d cnns for false positive reduction in pulmonary brain segmentation quality control from fully convnet monte nodule detection,” IEEE Transactions on Biomedical Engineering, carlo sampling,” arXiv preprint arXiv:1804.07046, 2018. vol. 64, no. 7, pp. 1558–1567, 2017. [122] R. Robinson, O. Oktay, W. Bai, V. V. Valindria, M. M. Sanghvi, [104] J. Zhang, M. Liu, and D. Shen, “Detecting anatomical landmarks N. Aung, J. M. Paiva, F. Zemrak, K. Fung, E. Lukaschuk et al., from limited medical imaging data using two-stage task-oriented “Subject-level prediction of segmentation failure using real-time deep neural networks,” IEEE Transactions on Image Processing, convolutional neural nets,” 2018. vol. 26, no. 10, pp. 4753–4764, 2017. [123] V. K. Singh, S. Romani, H. A. Rashwan, F. Akram, N. Pandey, [105] A. Katzmann, A. Muehlberg, M. Suehling, D. Noerenberg, J. W. M. Sarker, M. Kamal, J. T. Barrena, A. Saleh, M. Arenas et al., Holch, V. Heinemann, and H.-M. Gross, “Predicting lesion “Conditional generative adversarial and convolutional networks growth and patient survival in colorectal cancer patients using for x-ray breast mass segmentation and shape classification,” deep neural networks,” 2018. arXiv preprint arXiv:1805.10207, 2018. [106] Q. Meng, C. Baumgartner, M. Sinclair, J. Housden, M. Rajchl, [124] J. Zhang, A. Saha, B. J. Soher, and M. A. Mazurowski, “Au- et al. A. Gomez, B. Hou, N. Toussaint, V. Zimmer, J. Tan , “Auto- tomatic deep learning-based normalization of breast dynamic Data Driven matic shadow detection in 2d ultrasound images,” in contrast-enhanced magnetic resonance images,” arXiv preprint Treatment Response Assessment and Preterm, Perinatal, and Paediatric arXiv:1807.02152, 2018. Image Analysis. Springer, 2018, pp. 66–75. [125] K. Men, T. Zhang, X. Chen, B. Chen, Y. Tang, S. Wang, Y. Li, and [107] Y. Horie, T. Yoshio, K. Aoyama, S. Yoshimizu, Y. Horiuchi, J. Dai, “Fully automatic and robust segmentation of the clinical A. Ishiyama, T. Hirasawa, T. Tsuchida, T. Ozawa, S. Ishihara target volume for radiotherapy of breast cancer using big data et al., “Diagnostic outcomes of esophageal cancer by artificial in- and deep learning,” Physica Medica, vol. 50, pp. 13–19, 2018. telligence using convolutional neural networks,” Gastrointestinal endoscopy, vol. 89, no. 1, pp. 25–32, 2019. [126] J. Lee and R. M. Nishikawa, “Automated mammographic breast density estimation using a fully convolutional network,” Medical [108] K. Yasaka, H. Akai, O. Abe, and S. Kiryu, “Deep learning with physics, vol. 45, no. 3, pp. 1178–1190, 2018. convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced ct: a preliminary study,” Radiology, [127] Y. Zhang and A. Chung, “Deep supervision with additional vol. 286, no. 3, pp. 887–896, 2017. labels for retinal vessel segmentation task,” arXiv preprint [109] D. Zhang, J. Wang, J. H. Noble, and B. M. Dawant, “Accurate arXiv:1806.02132, 2018. detection of inner ears in head cts using a deep volume-to- [128] J. Staal, M. D. Abramoff,` M. Niemeijer, M. A. Viergever, and volume regression network with false positive suppression and B. Van Ginneken, “Ridge-based vessel segmentation in color a shape-based constraint,” arXiv preprint arXiv:1806.04725, 2018. images of the retina,” IEEE transactions on medical imaging, vol. 23, [110] O.¨ C¸ic¸ek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ron- no. 4, pp. 501–509, 2004. neberger, “3d u-net: learning dense volumetric segmentation [129] M. M. Fraz, P. Remagnino, A. Hoppe, B. Uyyanonvara, A. R. from sparse annotation,” in International Conference on Medical Rudnicka, C. G. Owen, and S. A. Barman, “An ensemble Image Computing and Computer-Assisted Intervention. Springer, classification-based approach applied to retinal blood vessel seg- 2016, pp. 424–432. mentation,” IEEE Transactions on Biomedical Engineering, vol. 59, [111] P. Rajpurkar, J. Irvin, A. Bagul, D. Ding, T. Duan, H. Mehta, no. 9, pp. 2538–2548, 2012. B. Yang, K. Zhu, D. Laird, R. L. Ball et al., “Mura dataset: To- [130] A. Hoover, V. Kouznetsova, and M. Goldbaum, “Locating blood wards radiologist-level abnormality detection in musculoskeletal vessels in retinal images by piecewise threshold probing of a radiographs,” arXiv preprint arXiv:1712.06957, 2017. matched filter response,” IEEE Transactions on Medical imaging, [112] Y. Li and W. Ping, “Cancer metastasis detection with neural vol. 19, no. 3, pp. 203–210, 2000. conditional random field,” arXiv preprint arXiv:1806.07064, 2018. [131] J. De Fauw, J. R. Ledsam, B. Romera-Paredes, S. Nikolov, [113] P. Bandi,´ O. Geessink, Q. Manson, M. van Dijk, M. Balkenhol, N. Tomasev, S. Blackwell, H. Askham, X. Glorot, B. ODonoghue, M. Hermsen, B. E. Bejnordi, B. Lee, K. Paeng, A. Zhong et al., D. Visentin et al., “Clinically applicable deep learning for diagno- “From detection of individual metastases to classification of sis and referral in retinal disease,” Nature medicine, vol. 24, no. 9, lymph node status at the patient level: the camelyon17 chal- p. 1342, 2018. lenge,” IEEE Transactions on Medical Imaging, 2018. [132] T. J. Jebaseeli, C. A. D. Durai, and J. D. Peter, “Segmentation of [114] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. retinal blood vessels from ophthalmologic diabetic retinopathy Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kit- images,” Computers & Electrical Engineering, vol. 73, pp. 245–258, tler et al., “Skin lesion analysis toward melanoma detection: A 2019. challenge at the 2017 international symposium on biomedical [133] X. Liu, J. Cao, T. Fu, Z. Pan, W. Hu, K. Zhang, and J. Liu, “Semi- imaging (isbi), hosted by the international skin imaging collab- supervised automatic segmentation of layer and fluid region in oration (isic),” in Biomedical Imaging (ISBI 2018), 2018 IEEE 15th retinal optical coherence tomography images using adversarial International Symposium on. IEEE, 2018, pp. 168–172. learning,” IEEE Access, vol. 7, pp. 3046–3061, 2019. [115] M. Forouzanfar, N. Forghani, and M. Teshnehlab, “Parameter [134] J. Duan, J. Schlemper, W. Bai, T. J. Dawes, G. Bello, G. Doumou, optimization of improved fuzzy c-means clustering algorithm A. De Marvao, D. P. ORegan, and D. Rueckert, “Deep nested level for brain mr image segmentation,” Engineering Applications of sets: Fully automated segmentation of cardiac mr images in pa- Artificial Intelligence, vol. 23, no. 2, pp. 160–168, 2010. tients with pulmonary hypertension,” in International Conference [116] R. Dey and Y. Hong, “Compnet: Complementary segmentation on Medical Image Computing and Computer-Assisted Intervention. network for brain mri extraction,” arXiv preprint arXiv:1804.00521, Springer, 2018, pp. 595–603. 2018. [135] W. Bai, M. Sinclair, G. Tarroni, O. Oktay, M. Rajchl, G. Vail- [117] D. S. Marcus, T. H. Wang, J. Parker, J. G. Csernansky, J. C. Morris, lant, A. M. Lee, N. Aung, E. Lukaschuk, M. M. Sanghvi et al., and R. L. Buckner, “Open access series of imaging studies (oasis): “Human-level cmr image analysis with deep fully convolutional cross-sectional mri data in young, middle aged, nondemented, networks,” arXiv preprint arXiv:1710.09289, 2017. 24

[136] P. Krahenb¨ uhl¨ and V. Koltun, “Efficient inference in fully con- using an adversarial image-to-image network,” in International nected crfs with gaussian edge potentials,” in Advances in neural Conference on Medical Image Computing and Computer-Assisted In- information processing systems, 2011, pp. 109–117. tervention. Springer, 2017, pp. 507–515. [137] W. Bai, H. Suzuki, C. Qin, G. Tarroni, O. Oktay, P. M. Matthews, [158] N. Lessmann, B. van Ginneken, P. A. de Jong, and I. Isgum,ˇ “Iter- and D. Rueckert, “Recurrent neural networks for aortic image ative fully convolutional neural networks for automatic vertebra sequence segmentation with sparse annotations,” in International segmentation,” arXiv preprint arXiv:1804.04383, 2018. Conference on Medical Image Computing and Computer-Assisted In- [159] D. Jin, Z. Xu, Y. Tang, A. P. Harrison, and D. J. Mollura, “Ct- tervention. Springer, 2018, pp. 586–594. realistic lung nodule simulation from 3d conditional genera- [138] A. Chartsias, T. Joyce, G. Papanastasiou, S. Semple, M. Williams, tive adversarial networks for robust lung segmentation,” arXiv D. Newby, R. Dharmakumar, and S. A. Tsaftaris, “Factorised preprint arXiv:1806.04051, 2018. spatial representation learning: application in semi-supervised [160] A. P. Harrison, Z. Xu, K. George, L. Lu, R. M. Summers, and D. J. myocardial segmentation,” arXiv preprint arXiv:1803.07031, 2018. Mollura, “Progressive and multi-path holistically nested neural [139] H. Kervadec, J. Dolz, M. Tang, E. Granger, Y. Boykov, and I. B. networks for pathological lung segmentation from ct images,” in Ayed, “Constrained-cnn losses forweakly supervised segmenta- International Conference on Medical Image Computing and Computer- tion,” arXiv preprint arXiv:1805.04628. Assisted Intervention. Springer, 2017, pp. 621–629. [140] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A [161] X. Xu, Q. Lu, L. Yang, S. Hu, D. Chen, Y. Hu, and Y. Shi, “Quan- deep neural network architecture for real-time semantic segmen- tization of fully convolutional networks for accurate biomedical tation,” arXiv preprint arXiv:1606.02147, 2016. image segmentation,” Preprint at https://arxiv. org/abs/1803.04907, [141] T. Joyce, A. Chartsias, and S. A. Tsaftaris, “Deep multi-class 2018. segmentation without ground-truth labels,” 2018. [162] K. Sirinukunwattana, J. P. Pluim, H. Chen, X. Qi, P.-A. Heng, [142] R. LaLonde and U. Bagci, “Capsules for object segmentation,” Y. B. Guo, L. Y. Wang, B. J. Matuszewski, E. Bruni, U. Sanchez arXiv preprint arXiv:1804.04241, 2018. et al., “Gland segmentation in colon histology images: The glas [143] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between challenge contest,” Medical image analysis, vol. 35, pp. 489–502, capsules,” in Advances in Neural Information Processing Systems, 2017. 2017, pp. 3856–3866. [163] L. Yang, Y. Zhang, J. Chen, S. Zhang, and D. Z. Chen, “Suggestive [144] C.-M. Nam, J. Kim, and K. J. Lee, “Lung nodule segmentation annotation: A deep active learning framework for biomedical with convolutional neural network trained by simple diameter image segmentation,” in International Conference on Medical Image information,” 2018. Computing and Computer-Assisted Intervention. Springer, 2017, pp. [145] N. Burlutskiy, F. Gu, L. K. Wilen, M. Backman, and P. Micke, 399–407. “A deep learning framework for automatic diagnosis in lung [164] D. Liu and T. Jiang, “Deep reinforcement learning for sur- cancer,” arXiv preprint arXiv:1807.10466, 2018. gical gesture segmentation and classification,” arXiv preprint [146] H. R. Roth, C. Shen, H. Oda, M. Oda, Y. Hayashi, K. Misawa, and arXiv:1806.08089, 2018. K. Mori, “Deep learning and its application to medical image [165] S. M. R. Al Arif, K. Knapp, and G. Slabaugh, “Spnet: Shape segmentation,” Medical Imaging Technology, vol. 36, no. 2, pp. 63– prediction using a fully convolutional neural network,” in In- 71, 2018. ternational Conference on Medical Image Computing and Computer- [147] A. Taha, P. Lo, J. Li, and T. Zhao, “Kid-net: Convolution networks Assisted Intervention. Springer, 2018, pp. 430–439. for kidney vessels segmentation from ct-volumes,” arXiv preprint [166] S. Pereira, A. Pinto, V. Alves, and C. A. Silva, “Brain tumor seg- arXiv:1806.06769, 2018. mentation using convolutional neural networks in mri images,” [148] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolu- IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1240–1251, tional neural networks for volumetric medical image segmenta- 2016. tion,” in 3D Vision (3DV), 2016 Fourth International Conference on. [167] P. Moeskops, M. A. Viergever, A. M. Mendrik, L. S. de Vries, IEEE, 2016, pp. 565–571. M. J. Benders, and I. Isgum,ˇ “Automatic segmentation of mr brain [149] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Mi- images with a convolutional neural network,” IEEE transactions sawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz et al., on medical imaging, vol. 35, no. 5, pp. 1252–1261, 2016. “Attention u-net: Learning where to look for the pancreas,” arXiv [168] P. Liskowski and K. Krawiec, “Segmenting retinal blood vessels preprint arXiv:1804.03999, 2018. with deep neural networks,” IEEE transactions on medical imaging, [150] E. Gibson, F. Giganti, Y. Hu, E. Bonmati, S. Bandula, K. Gu- vol. 35, no. 11, pp. 2369–2380, 2016. rusamy, B. R. Davidson, S. P. Pereira, M. J. Clarkson, and D. C. [169] J.-Z. Cheng, D. Ni, Y.-H. Chou, J. Qin, C.-M. Tiu, Y.-C. Chang, C.- Barratt, “Towards image-guided pancreas and biliary endoscopy: S. Huang, D. Shen, and C.-M. Chen, “Computer-aided diagnosis automatic multi-organ segmentation on abdominal ct with dense with deep learning architecture: applications to breast lesions in dilated networks,” in International Conference on Medical Image us images and pulmonary nodules in ct scans,” Scientific reports, Computing and Computer-Assisted Intervention. Springer, 2017, vol. 6, p. 24454, 2016. pp. 728–736. [170] M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, [151] M. P. Heinrich, O. Oktay, and N. Bouteldja, “Obelisk-one kernel Y. Bengio, C. Pal, P.-M. Jodoin, and H. Larochelle, “Brain tumor to solve nearly everything: Unified 3d binary convolutions for segmentation with deep neural networks,” Medical image analysis, image analysis,” 2018. vol. 35, pp. 18–31, 2017. [152] M. P. Heinrich and O. Oktay, “Briefnet: deep pancreas segmenta- [171] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. tion using binary sparse convolutions,” in International Conference Kane, D. K. Menon, D. Rueckert, and B. Glocker, “Efficient multi- on Medical Image Computing and Computer-Assisted Intervention. scale 3d cnn with fully connected crf for accurate brain lesion Springer, 2017, pp. 329–337. segmentation,” Medical image analysis, vol. 36, pp. 61–78, 2017. [153] M. P. Heinrich, M. Blendowski, and O. Oktay, “Ternarynet: faster [172] L. Fang, D. Cunefare, C. Wang, R. H. Guymer, S. Li, and S. Farsiu, deep model inference without gpus for medical 3d segmentation “Automatic segmentation of nine retinal layer boundaries in oct using sparse and binary convolutions,” International journal of images of non-exudative amd patients using deep learning and computer assisted radiology and surgery, pp. 1–10, 2018. graph search,” Biomedical optics express, vol. 8, no. 5, pp. 2732– [154] Y. Zhang, S. Miao, T. Mansi, and R. Liao, “Task driven generative 2744, 2017. modeling for unsupervised domain adaptation: Application to [173] B. Ibragimov and L. Xing, “Segmentation of organs-at-risks in x-ray image segmentation,” arXiv preprint arXiv:1806.07201, 2018. head and neck ct images using convolutional neural networks,” [155] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to- Medical physics, vol. 44, no. 2, pp. 547–557, 2017. image translation using cycle-consistent adversarial networks,” [174] M. Sarker, M. Kamal, H. A. Rashwan, S. F. Banu, A. Saleh, arXiv preprint, 2017. V. K. Singh, F. U. Chowdhury, S. Abdulwahab, S. Romani, [156] N. Tong, S. Gou, S. Yang, D. Ruan, and K. Sheng, “Fully automatic P. Radeva et al., “Slsdeep: Skin lesion segmentation based on multi-organ segmentation for head and neck cancer radiotherapy dilated residual and pyramid pooling networks,” arXiv preprint using shape representation model constrained fully convolu- arXiv:1805.10241, 2018. tional neural networks,” Medical physics, vol. 45, no. 10, pp. 4558– [175] D. Gutman, N. C. Codella, E. Celebi, B. Helba, M. Marchetti, 4567, 2018. N. Mishra, and A. Halpern, “Skin lesion analysis toward [157] D. Yang, D. Xu, S. K. Zhou, B. Georgescu, M. Chen, S. Grbic, melanoma detection: A challenge at the international symposium D. Metaxas, and D. Comaniciu, “Automatic liver segmentation on biomedical imaging (isbi) 2016, hosted by the international 25

skin imaging collaboration (isic),” arXiv preprint arXiv:1605.01397, [196] B. Hou, N. Miolane, B. Khanal, M. Lee, A. Alansary, S. McDon- 2016. agh, J. Hajnal, D. Rueckert, B. Glocker, and B. Kainz, “Deep pose [176] Z. Mirikharaji and G. Hamarneh, “Star shape prior in fully estimation for image-based registration,” 2018. convolutional networks for skin lesion segmentation,” in Inter- [197] B. Hou, B. Khanal, A. Alansary, S. McDonagh, A. Davidson, national Conference on Medical Image Computing and Computer- M. Rutherford, J. V. Hajnal, D. Rueckert, B. Glocker, and B. Kainz, Assisted Intervention. Springer, 2018, pp. 737–745. “Image-based registration in canonical atlas space,” 2018. [177] Y. Yuan, “Automatic skin lesion segmentation with fully [198] ——, “3d reconstruction in canonical co-ordinate space from convolutional-deconvolutional networks,” arXiv preprint arbitrarily oriented 2d images,” IEEE Transactions on Medical arXiv:1703.05165, 2017. Imaging, 2018. [178] F. Ambellan, A. Tack, M. Ehlke, and S. Zachow, “Automated [199] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. segmentation of knee bone and cartilage combining statistical Dalca, “An unsupervised learning model for deformable medi- shape knowledge and convolutional neural networks: Data from cal image registration,” in Proceedings of the IEEE Conference on the osteoarthritis initiative,” Medical Image Analysis, 2018. Computer Vision and Pattern Recognition, 2018, pp. 9252–9260. [179] A. Klein, J. Andersson, B. A. Ardekani, J. Ashburner, B. Avants, [200] P. Costa, A. Galdran, M. I. Meyer, M. Niemeijer, M. Abramoff,` M.-C. Chiang, G. E. Christensen, D. L. Collins, J. Gee, P. Hellier A. M. Mendonc¸a, and A. Campilho, “End-to-end adversarial et al., “Evaluation of 14 nonlinear deformation algorithms applied retinal image synthesis,” IEEE transactions on medical imaging, to human brain mri registration,” Neuroimage, vol. 46, no. 3, pp. vol. 37, no. 3, pp. 781–791, 2018. 786–802, 2009. [201] D. Mahapatra, B. Antony, S. Sedai, and R. Garnavi, “Deformable [180] R. Bajcsy and S. Kovaciˇ c,ˇ “Multiresolution elastic matching,” medical image registration using generative adversarial net- Computer vision, graphics, and image processing, vol. 46, no. 1, pp. works,” in Biomedical Imaging (ISBI 2018), 2018 IEEE 15th Inter- 1–21, 1989. national Symposium on. IEEE, 2018, pp. 1449–1453. [181] J.-P. Thirion, “Image matching as a diffusion process: an analogy [202] H. Tang, A. Pan, Y. Yang, K. Yang, Y. Luo, S. Zhang, and with maxwell’s demons,” Medical image analysis, vol. 2, no. 3, pp. S. H. Ong, “Retinal image registration based on robust non-rigid 243–260, 1998. point matching method,” Journal of Medical Imaging and Health [182] M. F. Beg, M. I. Miller, A. Trouve,´ and L. Younes, “Computing Informatics, vol. 8, no. 2, pp. 240–249, 2018. large deformation metric mappings via geodesic flows of diffeo- [203] L. Pan, F. Shi, W. Zhu, B. Nie, L. Guan, and X. Chen, “Detection morphisms,” International journal of computer vision, vol. 61, no. 2, and registration of vessels for longitudinal 3d retinal oct images pp. 139–157, 2005. using surf,” in Medical Imaging 2018: Biomedical Applications in Molecular, Structural, and Functional Imaging, vol. 10578. Interna- [183] J. Ashburner, “A fast diffeomorphic image registration algo- tional Society for Optics and Photonics, 2018, p. 105782P. rithm,” Neuroimage, vol. 38, no. 1, pp. 95–113, 2007. [204] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust [184] B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee, “Sym- features,” in European conference on computer vision. Springer, metric diffeomorphic image registration with cross-correlation: 2006, pp. 404–417. evaluating automated labeling of elderly and neurodegenerative brain,” Medical image analysis, vol. 12, no. 1, pp. 26–41, 2008. [205] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis [185] B. Glocker, N. Komodakis, G. Tziritas, N. Navab, and N. Para- and automated cartography,” Communications of the ACM, vol. 24, gios, “Dense image registration through mrfs and efficient linear no. 6, pp. 381–395, 1981. programming,” Medical image analysis, vol. 12, no. 6, pp. 731–741, [206] D. Mahapatra, “Elastic registration of medical images with gans,” 2008. arXiv preprint arXiv:1805.02369, 2018. [186] A. V. Dalca, A. Bobu, N. S. Rost, and P. Golland, “Patch-based [207] K. A. Eppenhof, M. W. Lafarge, P. Moeskops, M. Veta, and discrete registration of clinical brain images,” in International J. P. Pluim, “Deformable image registration using convolutional Workshop on Patch-based Techniques in Medical Imaging. Springer, neural networks,” in Medical Imaging 2018: Image Processing, vol. 2016, pp. 60–67. 10574. International Society for Optics and Photonics, 2018, p. [187] J. Krebs, T. Mansi, H. Delingette, L. Zhang, F. C. Ghesu, S. Miao, 105740S. A. K. Maier, N. Ayache, R. Liao, and A. Kamen, “Robust non- [208] E. Castillo, R. Castillo, J. Martinez, M. Shenoy, and T. Guerrero, rigid registration through agent-based action learning,” in In- “Four-dimensional deformable image registration using trajec- ternational Conference on Medical Image Computing and Computer- tory modeling,” Physics in Medicine & Biology, vol. 55, no. 1, p. Assisted Intervention. Springer, 2017, pp. 344–352. 305, 2009. [188] M.-M. Rohe,´ M. Datar, T. Heimann, M. Sermesant, and X. Pennec, [209] J. Vandemeulebroucke, O. Bernard, S. Rit, J. Kybic, P. Clarysse, “Svf-net: Learning deformable image registration using shape and D. Sarrut, “Automated segmentation of a motion mask to matching,” in International Conference on Medical Image Computing preserve sliding motion in deformable registration of thoracic and Computer-Assisted Intervention. Springer, 2017, pp. 266–274. ct,” Medical physics, vol. 39, no. 2, pp. 1006–1015, 2012. [189] H. Sokooti, B. de Vos, F. Berendsen, B. P. Lelieveldt, I. Isgum,ˇ [210] B. D. de Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Star- and M. Staring, “Nonrigid image registration using multi-scale ing, and I. Isgum,ˇ “A deep learning framework for unsupervised 3d convolutional neural networks,” in International Conference affine and deformable image registration,” Medical image analysis, on Medical Image Computing and Computer-Assisted Intervention. vol. 52, pp. 128–143, 2019. Springer, 2017, pp. 232–239. [211] J. Zheng, S. Miao, Z. J. Wang, and R. Liao, “Pairwise domain [190] X. Yang, R. Kwitt, M. Styner, and M. Niethammer, “Quicksilver: adaptation module for cnn-based 2-d/3-d registration,” Journal Fast predictive image registration–a deep learning approach,” of Medical Imaging, vol. 5, no. 2, p. 021204, 2018. NeuroImage , vol. 158, pp. 378–396, 2017. [212] J. Lv, M. Yang, J. Zhang, and X. Wang, “Respiratory motion [191] S. C. van de Leemput, M. Prokop, B. van Ginneken, and R. Man- correction for free-breathing 3d abdominal mri using cnn-based niesing, “Stacked bidirectional convolutional lstms for 3d non- image registration: a feasibility study,” The British journal of contrast ct reconstruction from spatiotemporal 4d ct,” 2018. radiology, vol. 91, no. xxxx, p. 20170788, 2018. [192] D. Nie, X. Cao, Y. Gao, L. Wang, and D. Shen, “Estimating ct [213] J. Lv, W. Huang, J. Zhang, and X. Wang, “Performance of u- image from mri data using 3d fully convolutional networks,” in net based pyramidal lucas-kanade registration on free-breathing Deep Learning and Data Labeling for Medical Applications. Springer, multi-b-value diffusion mri of the kidney,” The British journal of 2016, pp. 170–178. radiology, vol. 91, no. 1086, p. 20170813, 2018. [193] K. Bahrami, F. Shi, I. Rekik, and D. Shen, “Convolutional neural [214] P. Yan, S. Xu, A. R. Rastinehad, and B. J. Wood, “Adversarial im- network for reconstruction of 7t-like images from 3t mri using age registration with application for mr and trus image fusion,” appearance and anatomical features,” in Deep Learning and Data arXiv preprint arXiv:1804.11024, 2018. Labeling for Medical Applications. Springer, 2016, pp. 39–47. [215] Y. Hu, M. Modat, E. Gibson, N. Ghavami, E. Bonmati, C. M. [194] K. Lønning, P. Putzky, M. W. Caan, and M. Welling, “Recurrent Moore, M. Emberton, J. A. Noble, D. C. Barratt, and T. Ver- inference machines for accelerated mri reconstruction,” 2018. cauteren, “Label-driven weakly-supervised learning for multi- [195] A. Sheikhjafari, M. Noga, K. Punithakumar, and N. Ray, “Un- modal deformarle image registration,” in Biomedical Imaging (ISBI supervised deformable image registration with fully connected 2018), 2018 IEEE 15th International Symposium on. IEEE, 2018, pp. generative neural network,” 2018. 1070–1074. 26

[216] G. Zhao, B. Zhou, K. Wang, R. Jiang, and M. Xu, “Respond-cam: [236] T. Araujo,´ G. Aresta, E. Castro, J. Rouco, P. Aguiar, C. Eloy, Analyzing deep models for 3d imaging data by visualizations,” A. Polonia,´ and A. Campilho, “Classification of breast cancer arXiv preprint arXiv:1806.00102, 2018. histology images using convolutional neural networks,” PloS one, [217] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, vol. 12, no. 6, p. e0177544, 2017. D. Batra et al., “Grad-cam: Visual explanations from deep net- [237] H. D. Couture, J. Marron, C. M. Perou, M. A. Troester, and works via gradient-based localization.” in ICCV, 2017, pp. 618– M. Niethammer, “Multiple instance learning for heterogeneous 626. images: Training a cnn for histopathology,” arXiv preprint [218] T. Elss, H. Nickisch, T. Wissel, R. Bippus, M. Morlock, and arXiv:1806.05083, 2018. M. Grass, “Motion estimation in coronary ct angiography images [238] M. A. Troester, X. Sun, E. H. Allott, J. Geradts, S. M. Cohen, using convolutional neural networks,” 2018. C.-K. Tse, E. L. Kirk, L. B. Thorne, M. Mathews, Y. Li et al., [219] S. Miao, Z. J. Wang, and R. Liao, “A cnn regression approach “Racial differences in pam50 subtypes in the carolina breast for real-time 2d/3d registration,” IEEE transactions on medical cancer study,” JNCI: Journal of the National Cancer Institute, vol. imaging, vol. 35, no. 5, pp. 1352–1363, 2016. 110, no. 2, 2018. [220] G. Wu, M. Kim, Q. Wang, B. C. Munsell, and D. Shen, “Scalable [239] P. Sudharshan, C. Petitjean, F. Spanhol, L. E. Oliveira, L. Heutte, high-performance image registration framework by unsuper- and P. Honeine, “Multiple instance learning for histopathological vised deep feature representations learning,” IEEE Transactions breast cancer image classification,” Expert Systems with Applica- on Biomedical Engineering, vol. 63, no. 7, pp. 1505–1516, 2016. tions, vol. 117, pp. 103–111, 2019. [221] M. Simonovsky, B. Gutierrez-Becker,´ D. Mateus, N. Navab, and [240] K. Roy, D. Banik, D. Bhattacharjee, and M. Nasipuri, “Patch- N. Komodakis, “A deep metric for multimodal registration,” in based system for classification of breast histology images us- International Conference on Medical Image Computing and Computer- ing deep learning,” Computerized Medical Imaging and Graphics, Assisted Intervention. Springer, 2016, pp. 10–18. vol. 71, pp. 90–103, 2019. [222] X. Yang, R. Kwitt, and M. Niethammer, “Fast predictive image [241] N. Antropova, H. Abe, and M. L. Giger, “Use of clinical mri registration,” in Deep Learning and Data Labeling for Medical Appli- maximum intensity projections for improved breast lesion clas- cations. Springer, 2016, pp. 48–57. sification with deep convolutional neural networks,” Journal of Medical Imaging, vol. 5, no. 1, p. 014503, 2018. [223] Q. Zhang, Y. Xiao, W. Dai, J. Suo, C. Wang, J. Shi, and H. Zheng, “Deep learning based classification of breast tumors with shear- [242] D. Ribli, A. Horvath,´ Z. Unger, P. Pollner, and I. Csabai, “Detect- wave elastography,” Ultrasonics, vol. 72, pp. 150–157, 2016. ing and classifying lesions in mammograms with deep learning,” Scientific reports, vol. 8, no. 1, p. 4165, 2018. [224] D. Nie, R. Trullo, J. Lian, C. Petitjean, S. Ruan, Q. Wang, and [243] I. C. Moreira, I. Amaral, I. Domingues, A. Cardoso, M. J. Cardoso, D. Shen, “Medical image synthesis with context-aware generative and J. S. Cardoso, “Inbreast: toward a full-field digital mammo- adversarial networks,” in International Conference on Medical Image graphic database,” Academic radiology, vol. 19, no. 2, pp. 236–248, Computing and Computer-Assisted Intervention. Springer, 2017, pp. 2012. 417–425. [244] Y. Zheng, C. Yang, and A. Merkulov, “Breast cancer screening [225] E. Kang, J. Min, and J. C. Ye, “A deep convolutional neural using convolutional neural network and follow-up digital mam- network using directional wavelets for low-dose x-ray ct recon- mography,” in Computational Imaging III, vol. 10669. Interna- struction,” Medical physics, vol. 44, no. 10, 2017. tional Society for Optics and Photonics, 2018, p. 1066905. [226] B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and [245] H. Pratt, F. Coenen, D. M. Broadbent, S. P. Harding, and Y. Zheng, I. Isgum,ˇ “End-to-end unsupervised deformable image registra- “Convolutional neural networks for diabetic retinopathy,” Proce- tion with a convolutional neural network,” in Deep Learning in dia Computer Science, vol. 90, pp. 200–205, 2016. Medical Image Analysis and Multimodal Learning for Clinical Decision [246] M. S. Ayhan and P. Berens, “Test-time data augmentation for Support. Springer, 2017, pp. 204–212. estimation of heteroscedastic aleatoric uncertainty in deep neural [227] P. Radau, Y. Lu, K. Connelly, G. Paul, A. Dick, and G. Wright, networks,” 2018. “Evaluation framework for algorithms segmenting short axis [247] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in cardiac mri,” The MIDAS Journal-Cardiac MR Left Ventricle Seg- deep residual networks,” in European conference on computer vision. mentation Challenge, vol. 49, 2009. Springer, 2016, pp. 630–645. [228] X. Li, N. C. Dvornek, J. Zhuang, P. Ventola, and J. S. Duncan, [248] M. Mateen, J. Wen, S. Song, Z. Huang et al., “Fundus image clas- “Brain biomarker interpretation in asd using deep learning and sification using vgg-19 architecture with pca and svd,” Symmetry, fmri,” arXiv preprint arXiv:1808.08296, 2018. vol. 11, no. 1, p. 1, 2019. [229] E. H. Asl, M. Ghazal, A. Mahmoud, A. Aslantas, A. Shalaby, [249] R. Dey, Z. Lu, and Y. Hong, “Diagnostic classification of lung M. Casanova, G. Barnes, G. Gimelfarb, R. Keynton, and A. El Baz, nodules using 3d neural networks,” in Biomedical Imaging (ISBI “Alzheimers disease diagnostics by a 3d deeply supervised 2018), 2018 IEEE 15th International Symposium on. IEEE, 2018, adaptable convolutional network.” pp. 774–778. [230] W. Yan, H. Zhang, J. Sui, and D. Shen, “Deep chronnectome [250] A. Nibali, Z. He, and D. Wollersheim, “Pulmonary nodule clas- learning via full bidirectional long short-term memory networks sification with deep residual networks,” International journal of for mci diagnosis,” in International Conference on Medical Image computer assisted radiology and surgery, vol. 12, no. 10, pp. 1799– Computing and Computer-Assisted Intervention. Springer, 2018, 1808, 2017. pp. 249–257. [251] M. Gao, U. Bagci, L. Lu, A. Wu, M. Buty, H.-C. Shin, H. Roth, G. Z. [231] A. S. Heinsfeld, A. R. Franco, R. C. Craddock, A. Buchweitz, and Papadakis, A. Depeursinge, R. M. Summers et al., “Holistic clas- F. Meneguzzi, “Identification of autism spectrum disorder using sification of ct attenuation patterns for interstitial lung diseases deep learning and the abide dataset,” NeuroImage: Clinical, vol. 17, via deep convolutional neural networks,” Computer Methods in pp. 16–23, 2018. Biomechanics and Biomedical Engineering: Imaging & Visualization, [232] M. Soussia and I. Rekik, “A review on image-and network-based vol. 6, no. 1, pp. 1–6, 2018. brain data analysis techniques for alzheimer’s disease diagnosis [252] Y. Song, W. Cai, H. Huang, Y. Zhou, D. D. Feng, Y. Wang, reveals a gap in developing predictive methods for prognosis,” M. J. Fulham, and M. Chen, “Large margin local estimate with arXiv preprint arXiv:1808.01951, 2018. applications to medical image classification,” IEEE transactions on [233] B. Gutierrez-Becker and C. Wachinger, “Deep multi-structural medical imaging, vol. 34, no. 6, pp. 1362–1377, 2015. shape analysis: Application to neuroanatomy,” arXiv preprint [253] Y. Song, W. Cai, Y. Zhou, and D. D. Feng, “Feature-based image arXiv:1806.01069, 2018. patch approximation for lung tissue classification,” IEEE Trans. [234] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep Med. Imaging, vol. 32, no. 4, pp. 797–808, 2013. learning on point sets for 3d classification and segmentation,” [254] A. Depeursinge, A. Vargas, A. Platon, A. Geissbuhler, P.-A. Po- Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, vol. 1, letti, and H. Muller,¨ “Building a reference multimedia database no. 2, p. 4, 2017. for interstitial lung diseases,” Computerized medical imaging and [235] R. Awan, N. A. Koohbanani, M. Shaban, A. Lisowska, and N. Ra- graphics, vol. 36, no. 3, pp. 227–238, 2012. jpoot, “Context-aware learning using transferable features for [255] C. Biffi, O. Oktay, G. Tarroni, W. Bai, A. De Marvao, G. Doumou, classification of breast cancer histology images,” in International M. Rajchl, R. Bedair, S. Prasad, S. Cook et al., “Learning inter- Conference Image Analysis and Recognition. Springer, 2018, pp. pretable anatomical features through deep generative models: 788–795. Application to cardiac remodeling,” in International Conference 27

on Medical Image Computing and Computer-Assisted Intervention. “The multimodal brain tumor image segmentation benchmark Springer, 2018, pp. 464–471. (brats),” IEEE transactions on medical imaging, vol. 34, no. 10, p. [256] C. Brestel, R. Shadmi, I. Tamir, M. Cohen-Sfaty, and E. Elnekave, 1993, 2015. “Radbot-cxr: Classification of four clinical finding categories in [275] M. Heath, K. Bowyer, D. Kopans, R. Moore, and W. P. chest x-ray using deep learning,” 2018. Kegelmeyer, “The digital database for screening mammography,” [257] X. Wang, H. Chen, C. Gan, H. Lin, Q. Dou, Q. Huang, M. Cai, and in Proceedings of the 5th international workshop on digital mammog- P.-A. Heng, “Weakly supervised learning for whole slide lung raphy. Medical Physics Publishing, 2000, pp. 212–218. cancer image classification.” [276] G. Quellec, M. Lamard, P. M. Josselin, G. Cazuguel, B. Cochener, [258] N. Coudray, P. S. Ocampo, T. Sakellaropoulos, N. Narula, and C. Roux, “Optimal wavelet transform for the detection of M. Snuderl, D. Fenyo,¨ A. L. Moreira, N. Razavian, and A. Tsiri- microaneurysms in retina photographs.” IEEE Transactions on gos, “Classification and mutation prediction from non–small cell Medical Imaging, vol. 27, no. 9, pp. 1230–41, 2008. lung cancer histopathology images using deep learning,” Nature [277] E. Decenciere,` X. Zhang, G. Cazuguel, B. Lay, B. Cochener, medicine, vol. 24, no. 10, p. 1559, 2018. C. Trone, P. Gain, R. Ordonez, P. Massin, A. Erginay et al., “Feed- [259] J. M. Tomczak, M. Ilse, M. Welling, M. Jansen, H. G. Coleman, back on a publicly distributed image database: the messidor M. Lucas, K. de Laat, M. de Bruin, H. Marquering, M. J. van der database,” Image Analysis & Stereology, vol. 33, no. 3, pp. 231–234, Wel et al., “Histopathological classification of precursor lesions of 2014. esophageal adenocarcinoma: A deep multiple instance learning [278] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, approach,” 2018. “Chestx-ray8: Hospital-scale chest x-ray database and bench- [260] M. Ilse, J. M. Tomczak, and M. Welling, “Attention-based deep marks on weakly-supervised classification and localization of multiple instance learning,” arXiv preprint arXiv:1802.04712, 2018. common thorax diseases,” in Computer Vision and Pattern Recogni- [261] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger, and tion (CVPR), 2017 IEEE Conference on. IEEE, 2017, pp. 3462–3471. H. Greenspan, “Synthetic data augmentation using gan for im- [279] M. Niemeijer, J. Staal, B. van Ginneken, M. Loog, and M. D. proved liver lesion classification,” in Biomedical Imaging (ISBI Abramoff, “Comparative study of retinal vessel segmentation 2018), 2018 IEEE 15th International Symposium on. IEEE, 2018, methods on a new publicly available database,” in Medical Imag- pp. 289–293. ing 2004: Image Processing, vol. 5370. International Society for [262] Z. Xu, Y. Huo, J. Park, B. Landman, A. Milkowski, S. Grbic, Optics and Photonics, 2004, pp. 648–657. and S. Zhou, “Less is more: Simultaneous view classification [280] A. F. Fotenos, A. Snyder, L. Girton, J. Morris, and R. Buckner, and landmark detection for abdominal ultrasound images,” arXiv “Normative estimates of cross-sectional and longitudinal brain preprint arXiv:1805.10376, 2018. volume decline in aging and ad,” Neurology, vol. 64, no. 6, pp. [263] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, 1032–1039, 2005. and S. Thrun, “Dermatologist-level classification of skin cancer [281] J. C. Morris, “The clinical dementia rating (cdr): current version with deep neural networks,” Nature, vol. 542, no. 7639, p. 115, and scoring rules.” Neurology, 1993. 2017. [282] R. L. Buckner, D. Head, J. Parker, A. F. Fotenos, D. Marcus, J. C. [264] M. Combalia and V. Vilaplana, “Monte-carlo sampling applied to Morris, and A. Z. Snyder, “A unified approach for morphometric multiple instance learning for whole slide image classification,” and functional data analysis in young, old, and demented adults 2018. using automated atlas-based head size normalization: reliability [265] J. Antony, K. McGuinness, N. E. O’Connor, and K. Moran, “Quan- and validation against manual measurement of total intracranial tifying radiographic knee osteoarthritis severity using deep con- volume,” Neuroimage, vol. 23, no. 2, pp. 724–738, 2004. volutional neural networks,” in Pattern Recognition (ICPR), 2016 [283] E. H. Rubin, M. Storandt, J. P. Miller, D. A. Kinscherf, E. A. Grant, 23rd International Conference on. IEEE, 2016, pp. 1195–1200. J. C. Morris, and L. Berg, “A prospective study of cognitive [266] O. Paserin, K. Mulpuri, A. Cooper, A. J. Hodgson, and R. Garbi, function and onset of dementia in cognitively healthy elders,” “Real time rnn based 3d ultrasound scan adequacy for develop- Archives of neurology, vol. 55, no. 3, pp. 395–401, 1998. mental dysplasia of the hip,” in International Conference on Medical [284] Y. Zhang, M. Brady, and S. Smith, “Segmentation of brain mr Image Computing and Computer-Assisted Intervention. Springer, images through a hidden markov random field model and the 2018, pp. 365–373. expectation-maximization algorithm,” IEEE transactions on medi- [267] M. Anthimopoulos, S. Christodoulidis, L. Ebner, A. Christe, cal imaging, vol. 20, no. 1, pp. 45–57, 2001. and S. Mougiakakou, “Lung pattern classification for interstitial [285] J. Suckling, J. Parker, D. Dance, S. Astley, I. Hutt, C. Boggis, lung diseases using a deep convolutional neural network,” IEEE I. Ricketts, E. Stamatakis, N. Cerneaz, S. Kok et al., “Mammo- transactions on medical imaging, vol. 35, no. 5, pp. 1207–1216, 2016. graphic image analysis society (mias) database v1. 21,” 2015. [268] M. Kallenberg, K. Petersen, M. Nielsen, A. Y. Ng, P. Diao, C. Igel, [286] D. F. Pace, A. V. Dalca, T. Geva, A. J. Powell, M. H. Moghari, and C. M. Vachon, K. Holland, R. R. Winkel, N. Karssemeijer et al., P. Golland, “Interactive whole-heart segmentation in congenital “Unsupervised deep learning applied to breast density segmen- heart disease,” in International Conference on Medical Image Com- tation and mammographic risk scoring,” IEEE transactions on puting and Computer-Assisted Intervention. Springer, 2015, pp. medical imaging, vol. 35, no. 5, pp. 1322–1331, 2016. 80–88. [269] B. Q. Huynh, H. Li, and M. L. Giger, “Digital mammographic [287] S. Azizi, P. Mousavi, P. Yan, A. Tahmasebi, J. T. Kwak, S. Xu, tumor classification using transfer learning from deep convolu- B. Turkbey, P. Choyke, P. Pinto, B. Wood et al., “Transfer learning tional neural networks,” Journal of Medical Imaging, vol. 3, no. 3, from rf to b-mode temporal enhanced ultrasound features for p. 034501, 2016. prostate cancer detection,” International journal of computer assisted [270] Z. Yan, Y. Zhan, Z. Peng, S. Liao, Y. Shinagawa, S. Zhang, D. N. radiology and surgery, vol. 12, no. 7, pp. 1111–1121, 2017. Metaxas, and X. S. Zhou, “Multi-instance deep learning: Discover [288] T. Kooi, B. van Ginneken, N. Karssemeijer, and A. den Heeten, discriminative local anatomies for bodypart recognition,” IEEE “Discriminating solitary cysts from soft tissue lesions in mam- transactions on medical imaging, vol. 35, no. 5, pp. 1332–1343, 2016. mography using a pretrained deep convolutional neural net- [271] W. Sun, T.-L. B. Tseng, J. Zhang, and W. Qian, “Enhancing deep work,” Medical physics, vol. 44, no. 3, pp. 1017–1027, 2017. convolutional neural network scheme for breast cancer diagnosis [289] R. K. Samala, H.-P. Chan, L. M. Hadjiiski, M. A. Helvie, K. H. with unlabeled data,” Computerized Medical Imaging and Graphics, Cha, and C. D. Richter, “Multi-task transfer learning deep convo- vol. 57, pp. 4–9, 2017. lutional neural network: application to computer-aided diagnosis [272] S. Christodoulidis, M. Anthimopoulos, L. Ebner, A. Christe, and of breast cancer on mammograms,” Physics in Medicine & Biology, S. Mougiakakou, “Multi-source transfer learning with convolu- vol. 62, no. 23, p. 8894, 2017. tional neural networks for lung pattern analysis,” arXiv preprint [290] A. R. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese, arXiv:1612.02589, 2016. “Taskonomy: Disentangling task transfer learning,” in Proceedings [273] K. Lekadir, A. Galimzianova, A.` Betriu, M. del Mar Vila, L. Igual, of the IEEE Conference on Computer Vision and Pattern Recognition, D. L. Rubin, E. Fernandez,´ P. Radeva, and S. Napel, “A convo- 2018, pp. 3712–3722. lutional neural network for automatic characterization of plaque [291] N. Akhtar, A. S. Mian, and F. Porikli, “Joint discriminative composition in carotid ultrasound.” IEEE J. Biomedical and Health bayesian dictionary and classifier learning.” in CVPR, vol. 4, 2017, Informatics, vol. 21, no. 1, pp. 48–55, 2017. p. 7. [274] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Fara- [292] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online dictionary hani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest et al., learning for sparse coding,” in Proceedings of the 26th annual 28

international conference on machine learning. ACM, 2009, pp. 689– Syed Islam completed his PhD with Distinction 696. in Computer Engineering from the University of [293] Y. Bar, I. Diamant, L. Wolf, and H. Greenspan, “Deep learning Western Australia (UWA) in 2011. He received with non-medical training used for chest pathology identifica- his MSc in Computer Engineering from King tion,” in Medical Imaging 2015: Computer-Aided Diagnosis, vol. Fahd University of Petroleum and Minerals in 9414. International Society for Optics and Photonics, 2015, p. 2005 and BSc in Electrical and Electronic En- 94140V. gineering from Islamic Institute of Technology in [294] F. Ciompi, B. de Hoop, S. J. van Riel, K. Chung, E. T. Scholten, 2000. He was a Research Assistant Professor M. Oudkerk, P. A. de Jong, M. Prokop, and B. van Ginneken, at UWA from 2011 to 2015, a Research Fellow “Automatic classification of pulmonary peri-fissural nodules in at Curtin University from 2013 to 2015 and a computed tomography using an ensemble of 2d views and Lecturer at UWA from 2015-2016. Since 2016, a convolutional neural network out-of-the-box,” Medical image he has been working as Lecturer in Computer Science at Edith Cowan analysis, vol. 26, no. 1, pp. 195–202, 2015. University. He has published around 53 research articles and got nine [295] U. Lopes and J. F. Valiati, “Pre-trained convolutional neural net- public media releases. He obtained 14 competitive research grants for works as feature extractors for tuberculosis detection,” Computers his research in the area of Image Processing, Computer Vision and in biology and medicine, vol. 89, pp. 135–143, 2017. Medical Imaging. He has co-supervised to completion seven honours [296] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, “Un- and postgrad students and currently supervising one MS and seven PhD derstanding neural networks through deep visualization,” arXiv students. He is serving as Associate Editor of 13 international journals, preprint arXiv:1506.06579, 2015. Technical Committee Member of 24 conferences and regular reviewer [297] L. Zhang, A. Gooya, and A. F. Frangi, “Semi-supervised assess- of 26 journals. He is also serving seven professional bodies including ment of incomplete lv coverage in cardiac mri using generative IEEE and Australian Computer Society (Senior Member). adversarial nets,” in International Workshop on Simulation and Synthesis in Medical Imaging. Springer, 2017, pp. 61–68. [298] F. Calimeri, A. Marzullo, C. Stamile, and G. Terracina, “Biomed- ical data augmentation using generative adversarial neural net- works,” in International Conference on Artificial Neural Networks. Springer, 2017, pp. 626–634. [299] A. Lahiri, K. Ayush, P. K. Biswas, and P. Mitra, “Generative adversarial learning for reducing manual annotation in semantic segmentation on large scale miscroscopy images: Automated Naveed Akhtar received his PhD in Computer vessel segmentation in retinal fundus image as test case,” in Vision from The University of Western Australia Conference on Computer Vision and Pattern Recognition Workshops, (UWA) and Master degree in Computer Science 2017, pp. 42–48. from Hochschule Bonn-Rhein-Sieg, Germany [300] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining (HBRS). His research in Computer Vision and and harnessing adversarial examples (2014),” arXiv preprint Pattern Recognition is regularly published in the arXiv:1412.6572. prestigious venues of the field, including IEEE [301] H. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, CVPR and IEEE TPAMI. He has also served as A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, A. Enk et al., a reviewer for these, and twelve other research “Man against machine: diagnostic performance of a deep learn- venues in the broader field of Machine Learning. ing convolutional neural network for dermoscopic melanoma He is currently also serving as an Associate recognition in comparison to 58 dermatologists,” Annals of On- Editor of the IEEE Access. During his PhD, he was a recipient of multiple cology, vol. 29, no. 8, pp. 1836–1842, 2018. scholarships, and winner of the Canon Extreme Imaging Competition in [302] G. A. Bowen, “Document analysis as a qualitative research 2015. Currently, he is a Research Fellow at UWA. Previously, he has method,” Qualitative research journal, vol. 9, no. 2, pp. 27–40, 2009. also served on the same position at the Australian National University. [303] G. G. Chowdhury, “Natural language processing,” Annual review His current research interests include Medical Imaging, Adversarial of information science and technology, vol. 37, no. 1, pp. 51–89, 2003. Machine Learning, Hyperspectral Image Analysis, and 3D Point Cloud [304] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and Analysis. J. Xiao, “Lsun: Construction of a large-scale image dataset us- ing deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365, 2015. [305] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al., “Mastering the game of go without human knowledge,” Nature, vol. 550, no. 7676, p. 354, 2017. Naeem Khalid Janjua is a Lecturer at the School of Science, Edith Cowan University, Perth. He is an Associate Editor for Interna- tional Journal of Computer System Science and Engineering (IJCSSE) and International Journal of Intelligent systems (IJEIS). He has published an authored book, a book chapter, and vari- ous articles in international journals and refer- eed conference proceedings. His areas of active Fouzia Altaf is currently pursuing her PhD at research are defeasible reasoning, argumenta- the School of Science, Edith Cowan University tion, ontologies, data modelling, cloud comput- (ECU), Australia. Previously, she completed her ing, machine learning and data mining. He works actively in the domain Master of Philosophy and Master of Science of business intelligence and Web-based intelligent decision support degrees from the University of the Punjab, Pak- systems. istan. She is the recipient of Research Training Program scholarship at ECU. Her current re- search pertains to the areas of Deep Learning, Medical Image Analysis and Computer Vision.