1 Segmentation of Skin Lesions and their Attributes Using Multi-Scale Convolutional Neural Networks and Domain Specific Augmentations Mostafa Jahanifar*, Neda Zamani Tajeddin*, Navid Alemi Koohbanani*, Ali Gooya, and Nasir Rajpoot Abstract—Computer-aided diagnosis systems for classification task) from the lesion, to finally decide about the type of of different type of skin lesions have been an active field of skin lesions (classification task). In this framework, precise research in recent decades. It has been shown that introducing le- segmentation of lesion boundary is a critical task which can sions and their attributes masks into lesion classification pipeline can greatly improve the performance. In this paper, we propose lead to representative features to differentiate between normal a framework by incorporating transfer learning for segmenting and malignant lesions. lesions and their attributes based on the convolutional neural Lesions attributes, also known as dermoscopic structures, networks. The proposed framework is based on the encoder- are meaningful visual patterns in the skin lesion texture. Each decoder architecture which utilizes a variety of pre-trained of them represents a specific diagnostic symptom, and based networks in the encoding path and generates the prediction map by combining multi-scale information in decoding path using a on the lesion type and malignancy severity their appearance pyramid pooling manner. To address the lack of training data and may change [5]. Some lesions may not contain any specific increase the proposed model generalization, an extensive set of dermoscopic pattern while some may include textural pattern novel domain-specific augmentation routines have been applied related to several attributes. There are different types of to simulate the real variations in dermoscopy images. Finally, attributes defined in dermatology. Five of the most clinically by performing broad experiments on three different data sets obtained from International Skin Imaging Collaboration archive interesting attributes are pigment networks, globules/dot, Milia (ISIC2016, ISIC2017, and ISIC2018 challenges data sets), we like cysts, negative networks, and streaks. In clinical practices, show that the proposed method outperforms other state-of-the-art apart from the general and local appearance characteristics of approaches for ISIC2016 and ISIC2017 segmentation task and the lesion, dermatologists look for these attributes to better st achieved the 1 rank on the leader-board of ISIC2018 attribute decide about the lesion type and its malignancy level. Hence, detection task. CAD systems can also benefit from the useful information Index Terms—melanoma, attribute segmentation, lesion seg- that attributes deliver, to achieve a better automatic recognition mentation, deep learning, transfer learning, augmentation, ISIC performance. challenge Although modern CNNs are able to extract relevant features from the raw dermoscopic images and classify them without I. INTRODUCTION the need of lesions or their attributes segmentation map [6], it has been shown that incorporating additional information can KIN lesion analysis through image processing and ma- significantly improve lesion classification task [7], [8]. Yu et al. chine learning techniques has attracted a lot of attention S [7] showed that incorporating lesion mask can elevate lesion in the last decades [1]. With the advent of deep convolutional classification and they achieved 1st rank in the ISIC2016: neural networks (CNNs) and different scientific competitions Melanoma Recognition challenge. Moreover, Gonzalez Diaz on this topic in the recent years [2], [3], numerous powerful [8] proposed a lesion classification framework that incorpo- arXiv:1809.10243v3 [cs.CV] 29 Mar 2019 computational methods have been emerged to solve different rates both lesion segmentation and attributes probability maps problems in this area. In general, the final goal of using which was able to achieve state-of-the-art results using only computer aided diagnosis (CAD) systems for skin lesion one single model by adding this additional information. Such analysis is to automatically detect any abnormalities or skin researches and their promising results prove the importance of diseases. The methods in the literature usually follow a certain lesion mask and its attributes segmentation tasks for diagnosis analysis pipeline [4]: first, lesion area in the image is detected purposes. (segmentation task) which secondly helps other computerized There are various hurdles in automatic analysis of dermo- algorithms to extract discriminative feature (feature extraction scopic images due to the artefact in image capturing setup, M. Jahanifar and N. Zamani Tajeddin were with the Department of sample preparations and lesion appearance itself. During im- Biomedical Engineering, Tarbiat Modares University. M. Jahanifar is now age capturing from skin lesion samples using a dermatoscope with the Department of Research and Development, NRP company, Tehran, different image artifact can affect the output quality. These Iran (e-mails: fm.jahanifar,[email protected]). N. Alemi Koohbanani and N. Rajpoot are with the Tissue Image Analytics artefacts are hair occlusion, color charts, ruler markers, low (TIA) Lab at the Department of Computer Science, University of Warwick, image contrast and sharpness, uneven illumination, gel bub- UK (emails: fn.alemi-koohbanani,[email protected]). bles, marker annotations, clothing obstacles, color inconstancy, Ali Gooya is with the Department of Electronic and Electrical Engineering, University of Sheffield, Sheffield, England (e-mail: a.gooya@sheffield.ac.uk) and lens artifact (dark corners). Besides, skin lesions can show *These authors contributed equally to this work large variability in appearance based on their type. Severe 2 cases may have fuzzy borders, complex or vague texture, low we proposed to construct the saliency map of the lesion in a contrast against the skin, amorphous geometry, and multi- supervised and multi-scale manner based on the shape, color, colored face. Therefore, designing a general algorithm to and textures features extracted from super-pixel segments of overcome all these hurdles is hardly achievable. Attribute the images. In [11] several regression models in different segmentation task is even more challenging due to their small scales predict initial saliency maps based on extracted features structures and weak appearance. and the final saliency map of the lesion is achieved through Here, we are looking for powerful CNN models to segment combining them. In a recent paper, Garcia et al. [14] proposed lesion and its related attributes from dermoscopic images. In a hybrid segmentation method with two main parts: pixel fuzzy order to avoid the complications during training a CNN from classification and histogram thresholding. Their method can scratch, we propose to use transfer learning in a novel segmen- be consider as a saliency detection approach like [11] since tation framework. Our proposed segmentation network benefits authors in [14] extracted some pixel-wise color and texture from multi-scale convolutional and pyramid pooling blocks to features from image and allocate each pixel to one of lesion, detect desired small and big objects from the image. Another skin, or other probability maps (saliency maps). Afterward, novel aspect of the present work is extensive use of augmen- they employed a novel histogram-based thresholding algorithm tation routines. Considering the domain-specific knowledge, on the probability maps to achieve the lesion segmentation. we design some exquisite augmentations techniques to imitate Ahn et al. [13] also took a supervised saliency detection the appearance variation in dermoscopy images. Prediction approach. They inverted the lesion detection into a background maps are then generated by ensembling the outputs of different detection problem. In their method, the background map is models and different test time augmentation. To obtain the reconstructed by analysing image’s super-pixels in a multi- binary mask of the lesion or dermoscopy attribute, optimal scale framework. Then, they proposed that lesion area can thresholding and post-processing tasks are applied on the final be drawn from super-pixels having the highest reconstruction ensemble prediction map. error. They pre-processed the input images to remove hairs Rest of the paper is organized as follows: In section II, and color chart from the images and incorporated thresholding we broadly cover the recent literature for segmenting skin and post-processing operations on the output [13]. In an lesion and its attributes which is followed by description of unsupervised fashion, Fan et al. [12] introduced two saliency our approach to tackle these problems in section III. Section detection routines based on color and brightness information IV comprises some explanation about the data set, evaluation and proposed a method to fuse these saliency maps in order metrics, and implementation details. Experimental results for to construct the final lesion prediction. Then, the segmenta- lesion segmentation and attributes detection tasks on three tion process is done through a histogram-based thresholding different benchmark data sets are reported in section IV. algorithm. Finally, the discussion and conclusion of this research are 2) Unsupervised hybrid methods: Unsupervised hybrid drawn in sections V and VI, respectively. methods that usually contain several steps