Bio-Inspired Deep-CNN Pipeline for Skin Cancer Early Diagnosis

computation

Article Bio-Inspired Deep-CNN Pipeline for Skin Cancer Early Diagnosis

Francesco Rundo 1,* , Giuseppe Luigi Banna 2 and Sabrina Conoci 1 1 STMicroelectronics ADG—Central R&D, Stradale Primosole 50, 95121 Catania, Italy 2 Division of Medical Oncology, Cannizzaro Hospital, 95126 Catania, Italy * Correspondence: [email protected]; Tel.: +39-095-7404566

Received: 21 June 2019; Accepted: 17 August 2019; Published: 22 August 2019

Abstract: Skin cancer is the most common type of cancer, as also among the riskiest in the medical oncology field. Skin cancer is more common in people who work or practice outdoor sports and those that expose themselves to the sun. It may also develop years after radiographic therapy or exposure to substances that cause cancer (e.g., arsenic ingestion). Numerous tumors can affect the skin, which is the largest organ in our body and is made up of three layers: the epidermis (superficial layer), the dermis (middle layer) and the subcutaneous tissue (deep layer). The epidermis is formed by different types of cells: melanocytes, which have the task of producing melanin (a pigment that protects against the damaging effects of sunlight), and the more numerous keratinocytes. The keratinocytes of the deepest layer are called basal cells and can give rise to basal cell carcinomas. We are interested in types of skin cancer that originate from melanocytes, i.e., the so-called melanomas, because it is the most aggressive. The dermatologist, during a complete visit, evaluates the personal and family history of the patient and carries out an accurate visual examination of the skin, thanks to the use of epi-luminescence (or dermoscopy), a special technique for enlarging and illuminating the skin. This paper mentions one of the most widely used diagnostic methods due to its simplicity and validity—the ABCDE method (Asymmetry, edge irregularity, Color Variegation, Diameter, Evolution). This methodology, based on “visual” investigation by the dermatologist and/or oncologist, has the advantage of not being invasive and quite easy to perform. This approach is affected by the opinion of who (physicians) applies it. For this reason, certain diagnosis of cancer is made, however, only with a biopsy, a procedure during which a portion of tissue is taken and then analyzed under a microscope. Obviously, this is particularly invasive for the patient. The authors of this article have analyzed the development of a method that obtains with good accuracy the early diagnosis of skin neoplasms using non-invasive, but at the same time, robust methodologies. To this end, the authors propose the adoption of a deep learning pipeline based on morphological analysis of the skin lesion. The results obtained and compared with previous approaches confirm the good performance of the proposed pipeline.

Keywords: deep learning; skin cancer; melanoma; STM32

1. Introduction The stage of skin cancer indicates how widespread the disease is in the body and is a very important parameter for determining prognosis and deciding on the type of treatment to be undertaken. Therefore, diagnosis and relative early staging are of fundamental importance to increase a patient’s chances of survival. Basal cell and spino-cellular carcinomas, unlike other skin tumors such as melanoma, metastasize only in rare cases and many years after their appearance. As a result, they are usually removed when they are still located. In persons with immune system issues, spino-cellular carcinomas can be at high risk of metastasis and for this we proceed with staging with the TNM system: tumors can be classiﬁed into four stages (I, II, III and IV) based on the size and position of

Computation 2019, 7, 44; doi:10.3390/computation7030044 www.mdpi.com/journal/computation Computation 2019, 7, 44 2 of 16 the disease (T), involvement of lymph nodes (N) and presence of metastases (M). It is clear, therefore, that the search for an efficient and accurate approach that allows dermatologists and/or oncologists to quickly discriminate skin lesion (the so-called “nevus”) suspicions (which therefore deserve further investigation) is a subject of considerable attention by researchers in the medical field. Basal cell and spino-cellular carcinomas, if treated in the initial phases, recover in almost all cases and can often be treated completely, thanks to surgery and/or local treatments. Therefore, surgery is generally the first choice treatment for these tumors. Radiotherapy and systemic chemotherapy (to the whole body) are not used very often for skin carcinomas: chemotherapy is useful in cases where the tumor has reached lymph nodes. Therefore, much emphasis is placed on early diagnosis, especially for the most aggressive histology of melanoma. Among the commonly used methods for early diagnosis of skin tumors is the “ABCDE Method”; based on the abbreviation, the following nevus characteristics are monitored: Asymmetry, Borders, variegation of the Color, Diameter, Evolution of the lesion cuts. Basically, a physician who applies the ABCDE criterion in the analysis of a skin lesion performs a visual statistical evaluation of the nevus based on the heuristic (and experience-based) evaluation of the characteristics mentioned. Clearly, this strategy suffers from the subjectivity of the clinician or low sensitivity and specificity. Consequently, analysis by the ABCDE method is followed by the necessary biopsy confirmation that often, however, belies the apparent oncological risk that instead the dermatologist had estimated by means of the ABCDE analysis. In order to correctly address this issue, the authors propose an automatic pipeline discrimination system based on the analysis of dermoscopic images so as to classify the low-risk lesion (benign lesions or low risk of oncological progression) from those at high risk (high risk of oncological progression), trying to find a good trade-off between sensitivity and specificity. Scientific research is making significant progress in the aspect of early nevus diagnosis. A method that currently shows itself to be particularly promising is the so-called “ex vivo confocal microscopy” (performed on excised tissue), which allows us to anticipate the time taken for standard histological analyzes, which is about 10 days. The specificity—the ability to exclude the presence of the disease—of this method is particularly high. Thus, this new method allows to intervene with the biopsy only if necessary and not as a precaution in case of suspected malignancy. However, methods based on recent deep learning techniques continue to be much investigated by researchers around the world. The authors recently developed two different implementations of deep systems for skin lesion discrimination [1,2]. The obtained results are promising. A subsequent extension of the proposed approach is described in [3] and in which performances have been strongly increased, thanks to revision of the hand-crafted image features, adopted together with the use of Stacked Deep Autoencoder as classifiers of such features. All the algorithmic versions proposed by the authors in the previous referenced works have been successfully validated using the public dataset (containing a set of dermoscopic images of benign and malignant nevi) called PH2 [4]. However, in the scientific literature, different algorithmic approaches have been proposed for the early and robust diagnosis of skin cancer. Among these, it is worth mentioning methods based on classical pattern recognition techniques combined with the statistical study of dermoscopic images [5,6]. In [6–9], some methods for classifying skin cancer are described, which are based on the adaptive analysis of the so-called global and local characteristics of lesions. Using soft computing techniques, the authors of these pipelines have successfully discriminated skin lesions by post-processing these features. Also the use of clustering methods based on Support Vector Machines (SVM), Artificial Neural Networks (ANNs), K-closer, Naive-Bayes Algorithm have shown remarkable discriminative abilities, compared to the classical methods based on the search for specific patterns in dermoscopic images [8,9]. A very promising approach has been proposed in [7], in which the features of ad hoc hand-crafted dermoscopic image features are combined with an in-depth learning algorithm capable of extracting additional learned functions, as well as classifying the entire set of features with respect to the characterization of benign versus malignant lesions. However, the analysis of the results obtained in [7] shows a fairly low sensitivity/specificity ratio. In [8], the authors propose an interesting novel Computation 2019, 7, 44 3 of 16

Computation 2019, 7, x FOR PEER REVIEW 3 of 16 multiple convolution neural network model (MCNN) to classify different disease types in dermoscopic images,propose where an interesting several models novel weremultiple trained convolution separately neural using annetwork additive model sample (MCNN) learning to strategy. classify Thedifferent results disease are promising, types in asdermoscopic confirmed images, by authors where in [ 8several]. An interesting models were consideration trained separately was made using by suchan additive authors sample in [9] in learning which the strategy. skin lesion The images results dataset are promising, was augmented as confirmed before by it is authors used in in the [8]. deep An learninginteresting pipeline. consideration The image was dataset made augmentationby such authors showed in [9] an in interestingwhich the performanceskin lesion images increase dataset with respectwas augmented to the same before approach it is appliedused in in the the deep same lear datasetning withoutpipeline. augmentation The image dataset pre-processing. augmentation More detailsshowed are an provided interesting in [performance9]. increase with respect to the same approach applied in the same datasetThe without authors augmentation intend to propose pre-processi an innovativeng. More methoddetails are of provided nevus discrimination in [9]. for the early diagnosisThe ofauthors high-risk intend malignancy to propose lesions an (with innovative particular method attention of nevus to the histologydiscrimination called melanomafor the early as thediagnosis most aggressive). of high-risk Considering malignancy thatlesions melanoma (with particular originates attention in the melanocytic to the histology cells ofcalled the epidermis,melanoma weas characterizedthe most aggressive). the proposed Considering hand-crafted that features melano thatma wereorigina particularlytes in the attentive melanocytic to the cells superficial of the melanocyticepidermis, we distribution characterized of the the nevus. proposed In fact, hand-craft we designeded features ad-hoc that hand-crafted were particula imagerly features attentive that to allowthe superficial us to characterize melanocytic the self-similaritydistribution of ofthe melanocytic nevus. In fact, cells we in designed addition toad-hoc the degree hand-crafted of symmetry image andfeatures regularity that allow of the us edges to cha ofracterize the lesion the of self-similarity the skin, as specified of melanocytic in the nextcells paragraphs.in addition to In the this degree way, weof symmetry bio-inspired and the regularity proposed of algorithm, the edges of since the thelesion characterization of the skin, as ofspecified the nevus in the is correlatednext paragra tophs. the analyticalIn this way, modeling we bio-inspired of the physical the properties proposed of algori the melanocyticthm, since cellsthe characterization of the analyzed lesion. of the Moreover, nevus is thecorrelated algorithmic to the part analytical linked to modeling the use ofof Cellular the physical Neural properties Networks of (CNNs) the melanocytic was hypothesized cells of the to analyzeanalyzed each lesion. pixel Moreover, of the lesion the (which algorithmic groups part several linked melanocytic to the use cells) of byCellular means Neural of the processing Networks defined(CNNs)by was the hypothesized suitably configured to analyze CNNs’ each pixel cell state of the equation. lesion (which Therefore, groups each several CNN’s melanocytic setup allows cells) theby means characterization of the processing of a specific defined morphological by the suitably feature configured of the CNNs’ analyzed cell lesion,state equation. thus enabling Therefore, the discriminatoreach CNN’s setup system allows based the on convolutivecharacterization multi-level of a specific deep architecture morphological to learn feature these of morphological the analyzed featureslesion, thus and enabling from these the to discriminator provide an internal system functional based on mappingconvolutive useful multi-level for classifying deep architecture the analyzed to skinlearn lesion. these Themorphological proposed method features will and be from compared these to with provide methods an internal present functional in scientific mapping literature useful and previousfor classifying algorithmic the analyzed versions skin proposed lesion. by The the proposed authors [ 1method–3]. In orderwill be to guaranteecompared awith robust methods basis ofpresent comparison, in scientific it was literature decided toand apply previous and validatealgorithmic the proposedversions proposed pipeline inby the the same authors database [1–3]. ofIn dermoscopicorder to guarantee images, a i.e.,robust the basis PH2 databaseof compa [rison,4]. it was decided to apply and validate the proposed pipeline in the same database of dermoscopic images, i.e., the PH2 database [4]. 2. Materials and Methods 2. Materials and Methods Figure1 shows the proposed pipeline. Each block is detailed in the following paragraphs. The pipelineFigure 1 proposed shows the here proposed aims to pipeline. analyzethe Each dermoscopic block is deta imageiled in of the the following nevus generating paragraphs. features The thatpipeline are characteristic proposed here of theaims lesion to analyze of the skin,the dermoscopic which allows image the convolutional of the nevus ge classificationnerating features system tha tot characterizeare characteristic the high of riskthe lesion of oncological of the skin, progression which allows or confirm the convolutional its benignity. Inclassification this way, we system want toto providecharacterize physicians the high with risk a practicalof oncological and innovative progression tool or that confirm allows its them benignity. to diagnose In this with way, su weffi cientwant evidenceto provide the physicians neoplastic with progression a practical of aand new in growth of skin.

FigureFigure 1.1. TheThe proposedproposed bio-inspiredbio-inspired skinskin lesionlesion discriminationdiscrimination pipeline. pipeline.

2.1. SC-CNNs Pre-Processing Block The input dermoscopy lesion image D(x,y) is usually a simple RGB colour image acquired by classical medical optical dermatoscope. Computation 2019, 7, x FOR PEER REVIEW 4 of 16 Computation 2019, 7, 44 4 of 16 Dermatoscopy, dermoscopy or epiluminescence, is a non-invasive technique aimed at the early diagnosis of melanoma based on an optical instrument called a dermatoscope, which allows to 2.1. SC-CNNs Pre-Processing Block observe sub-cutaneous patterns not visible to the naked eye, favoring their recognition. The optical input dermoscopy dermatoscope lesion is a small image manualD(x,y) istool usually based aon simple a lens, RGB able colourto provide image magnifications acquired by classicalmainly between medical 10 optical and dermatoscope.20 times, specially illuminated with incident light. It is now demonstrated how Dermatoscopy,dermoscopy increases dermoscopy the diagnostic or epiluminescence, sensitivity for is amelanoma non-invasive compared technique to simple aimed vision at the earlywith diagnosisthe naked of eye melanoma of 20–30%, based allowing on an opticalearly diagnosis. instrument Experiences called a dermatoscope, over the last which decade allows have to made observe it sub-cutaneouspossible to extend patterns the application not visible of to this the nakeddiagnostic eye, technique favoring their to non-pigmented recognition. skin lesions. Today dermatoscopyThe optical is dermatoscope particularly effective is a small in manual the recognition tool based of on non-melanocytic a lens, able to provide skin tumors magnifications such as mainlymelanoma, between basal 10cell and carcinoma 20 times, and specially Bowen’s illuminated disease and with of other incident cutaneous light. Itneo-formations is now demonstrated such as howseborrheic dermoscopy keratoses, increases actinic the kerato diagnosticses, dermatofibroma, sensitivity for and melanoma clear cell compared acanthoma. to simple Figure vision 2 shows with a theclassical naked dermoscopy eye of 20–30%, image allowing of a nevus. early We diagnosis. are interested Experiences in a method over that the lastallows decade us to have discriminate made it possibleoncologically to extend the lesion the application of the skin of using this diagnostica low computational technique tocomplexity non-pigmented method. skin lesions. Today dermatoscopyFor this reason, is particularly the proposed effective pipeline in the will recognition process the of non-melanocyticgray-level component skin tumorsof the starting such as melanoma,dermoscopic basal RGB cell colour carcinoma image. andTo do Bowen’s that, the disease source and dermoscopic of other cutaneous colour images neo-formations are converted such into as seborrheicYCbCr format keratoses, [10–12] actinic from which keratoses, we will dermatofibroma, estract only the and luminance clear cell component acanthoma. (gray Figure levels2 shows image). a classicalWe denoted dermoscopy this gray-levels image ofimage a nevus. as Y Wep(x,y are). It interested is fed as in“input” a method and that“state” allows to a us State-Controlled to discriminate oncologicallyCellular Neural the Networks lesion of the(SC- skinCNNs) using 2D a matrix low computational having a size complexity equal to the method. source image Yp(x,y) i.e., m × n.

Figure 2. A typical dermoscopy image of a skin nevus.

ForThe thisfirst reason,version theof the proposed Cellular pipeline Neural will(or Nonlinear) process the Network gray-level (CNN) component was proposed of the startingby L.O dermoscopicChua and L. RGBYang colour[13]. The image. CNNs To docan that, be defined the source as a dermoscopic high speed, colourlocal in imagesterconnected are converted computing into YCbCrarray of format analog [10 –processors.12] from which The weCNN will dynamic estract only is defined the luminance through component so-called (gray cloning levels template image). Wematrixes. denoted The this CNN gray-levels cells interact image with as eachYp(x ,othery). It within is fed as their “input” neighbourhood, and “state” defined to a State-Controlled by a heuristic Cellularradius. Neural Networks (SC-CNNs) 2D matrix having a size equal to the source image Yp(x,y) i.e., m n. × Each CNN cell has an input, a state, and an output, which is correlated to the state (usually by meansThe of firstPieweise version Linear of the (PWL) Cellular function). Neural The (or Nonlinear)CNN can be Network implemented (CNN) both was in proposed digital (trough by L.O ChuaFPGA andtechnology) L. Yang [and13]. analog The CNNs (VLSI can or UVLSI be defined technology) as a high for speed, which local it is able interconnected to perform computinghigh speed array“near ofreal-time” analog processors. computations. The CNN Some dynamic stability is defined results through by means so-called of Lyapunov cloning template theorems matrixes. and Theconsideration CNN cells about interact the with dynamics each other of the within CNNs their can neighbourhood, be found in [13]. defined Several by extensions a heuristic of radius. classical CNNEach have CNN been cellreported has an in input, the literature, a state, and among an output, which whichit is worth is correlated mentioning to the the state so-called (usually State- by meansControlled of Pieweise CNNs Linearintroduced (PWL) (SC-CNNs) function). Theby Arena CNN canet al. be in implemented [14]. This model both indirectly digital (troughexplicates FPGA the technology)dependency andof dynamic analog (VLSIevolution or UVLSI of the technology)cell to the “state” for which of the it issingle able cell. to perform We refer high to SC-CNNs speed “near in real-time”the following computations. mathematical Some formulations. stability results By assi bygning means each of Lyapunov normalized theorems gray-level and considerationlesion image aboutYp(x,y) the (ofdynamics size m × ofn) theto input CNNs and can state be found of the in SC-CNNs, [13]. Several several extensions image of processing classical CNN tasks have may been be reportedperformed in according the literature, to the among defined which cloning it is templates worth mentioning instructions the [14] so-called. Equation State-Controlled (1) defines the CNNs state introducedequation of (SC-CNNs)a (m × n) SC-CNNs. by Arena Nr et(i,j al.) represents in [14]. Thisthe neighbourhood model directly of explicates each cell theC(i,j dependency) with radius ofr; dynamicxij represents evolution the state, ofthe yij the cell output to the “state”(mapped of to the the single state cell. xij by We means refer toof SC-CNNsa classical inPWL the equation) following, mathematicaland uij the input formulations. of the cell C By(i,j). assigning The cloning each templates normalized suitable gray-level to define lesion the image dynamicYp(x ,ofy) the (of sizeSC- m n) to input and state of the SC-CNNs, several image processing tasks may be performed according CNNs× are respectively A(i,j;k,l), B(i,j;k,l), C(i,j;k,l), and the constant bias I. The coefficients C and Rx

Computation 2019, 7, x FOR PEER REVIEW 5 of 16

Computation 2019, 7, 44 5 of 16 are correlated to the capacitor and resistance of the analog circuit used to implement the single SC- CNNs cell [13,14]. to the defined cloning𝑑𝑥(𝑡 templates) 1 instructions [14]. Equation (1) defines the state equation of a (m n) 𝐶 =− 𝑥 × SC-CNNs. Nr(i,j) represents𝑑𝑡 the𝑅 neighbourhood of each cell C(i,j) with radius r; xij represents the state, yij the output (mapped to the state+𝐴xij by means( of𝑖, 𝑗;a classical 𝑘, 𝑙)𝑦(𝑡) PWL+ equation)𝐵(𝑖,, and 𝑗; 𝑘, 𝑙)u𝑢ijthe(𝑡) input of the cell C(i,j). The cloning templates suitable(, to)∈ define(,) the dynamic of( the,)∈ SC-CNNs(,) are respectively A(i,j;k,l),

B(i,j;k,l), C(i,j;k,l), and the constant+ bias I. The 𝐶 coe(𝑖,ffi 𝑗;cients 𝑘, 𝑙)𝑥(𝑡C) and+ 𝐼R x are correlated to the capacitor and resistance of the analog circuit used( to,) implement∈(,) the single SC-CNNs cell [13,14]. (1) (1𝑖𝑀,1𝑗𝑁) dxij(t) 1 P P C = xij + A(i, j; k, l)y (t) + B(i, j; k, l)u (t) dt − Rx 1 kl kl 𝑦 (𝑡) =C(k,l)(𝑥Nr(i(,j𝑡)) +1−𝑥 (𝑡) −1)C( k,l) Nr(i,j) P 2 ∈ ∈ + C(i, j; k, l)xkl(t) + I C(k,l) Nr(i,j) ∈ (1) 𝑁(𝑖,) 𝑗 =𝐶(𝑘, 𝑙); (𝑚𝑎𝑥(|𝑘−𝑖(1 |,i|𝑖−𝑗M|), 1𝑟,1𝑘𝑀,1𝑙𝑁j N) ) 1≤ ≤ ≤ ≤ As anticipated, the SC-CNNs ycomparedij(t) = 2 xtoij( thet) + classical1 xij( tCNNs) 1 model introduced by Chua and n − − o Yang make it possibleNr (toi, jexplain) = Cr (thek, l )direct; max dependenk i , i cej betweenr, 1 kthe Mstate, 1 ofl theN cell with that of the | − | − ≤ ≤ ≤ ≤ ≤ adjacent cells belonging to its own neighborhood. In this way, we were able to characterize different imageAs processing anticipated, tasks the SC-CNNswith respect compared to theto sa theme classicalexecutable CNNs using model the introducedCNN model by Chuaoriginally and Yangproposed make by it Chua possible and to Yang. explain Through the direct SC-CNNs, dependence we can between better exploit the state the of dependence the cell with between that of thethe adjacentdynamics cells of the belonging state of individual to its own neighborhood.cells in the network. In this In way, this weway, were considering able to characterize that the intensity different of imagethe single processing pixel of tasks the withdermoscopic respect to image the same is associ executableated with using the the state CNN of model the single originally cell of proposed the SC- byCNNs, Chua through and Yang. the Through appropriate SC-CNNs, configurations we can better of the exploit cloning the dependencetemplates, we between are able the dynamicsto highlight of thecertain state morphological of individual characteristics cells in the network. of the nevus In this or way, eliminate considering others. that the intensity of the single pixelBefore of the dermoscopicprocessing the image acquired is associated pre-processed with the nevus state image, of the it single is important cell of the to segment SC-CNNs, the through Region theof Interest appropriate (ROI) configurations of the image of(i.e., the the cloning nevus) templates, with respect we are to able background to highlight (skin). certain Considering morphological that characteristicsskin lesion segmentation of the nevus is oranother eliminate issue others. currently under investigation by the authors, we decided to focusBefore on processingthe only objective the acquired of discriminating pre-processed against nevus image,skin lesions it is important and for this to segment reason thewe Regionused a ofsegmentation Interest (ROI) mask of the provided image (i.e., by the the nevus) dermoscopy with respect images to background database we (skin). used Considering for validating that skinthe lesionproposed segmentation approach isi.e., another the PH issue2 database currently [2]. under investigation by the authors, we decided to focus on theThe only SC-CNNs objective pre-processing of discriminating pipeline against is skin detailed lesions in and Figure for this 3, while reason Equation we used a(2) segmentation reports the masksetup providedof the used by pre-processing the dermoscopy cloning images templates database we[2,15]. used for validating the proposed approach i.e., the PH2 database [2]. 000 2.8 0.20 0.20 000 The SC-CNNs𝐴 pre-processing=000 ;𝐵 pipeline =0.20 is detailed 2.71 0.20 in Figure ;𝐶3, = while000 Equation; 𝐼=0.65 (2) reports the setup(2) of the used pre-processing000 cloning templates2.71 [ 0.202,15]. 0.20 000 The cloning templates in (2) are useful to configure SC-CNNs in order to perform ad-hoc  0 0 0   2.8 0.20 0.20   0 0 0  adaptive time-transient image stabilization (motion artifacts, dermo-gel, etc.). The cloning templates’ A =  0 0 0  ; B =  0.20 2.71 0.20  ; C =  0 0 0 ; I = 0.65 (2) setup used in (2) improves stabilization  with respect to the previous version used in [1–3]. The  0 0 0   2.71 0.20 0.20   0 0 0  dermoscopy gray level image processed by SC-CNNs is denoted as YSCC(x,y). Figures 4a–d show the mask and output images related to the SC-CNN pre-processing task.

Figure 3. The pre-processing block based on SC-CNNs. Figure 3. The pre-processing block based on SC-CNNs. The cloning templates in (2) are useful to conﬁgure SC-CNNs in order to perform ad-hoc adaptive time-transient image stabilization (motion artifacts, dermo-gel, etc.). The cloning templates’ setup used in (2) improves stabilization with respect to the previous version used in [1–3]. The dermoscopy gray level image processed by SC-CNNs is denoted as YSCC(x,y). Figure4a–d show the mask and output images related to the SC-CNN pre-processing task.

Computation 2019, 7, 44 6 of 16 Computation 2019, 7, x FOR PEER REVIEW 6 of 16

2 Figure 4.4. ((aa)) OriginalOriginal RGB RGB dermoscopy dermoscopy image, image, (b) ( image-maskb) image-mask provided provided by PH by database,PH2 database, (c) SC-CNNs (c) SC- pre-processedCNNs pre-processed image, image, (d) segmented (d) segmented SC-CNNs SC-CNNs pre-processed pre-proc imageessed withimage details. with details. 2.2. Morphological Pattern and SC-CNNs Features Block 2.2. Morphological Pattern and SC-CNNs Features Block The aim of this block is to provide a mathematical representation of the ABCDE rule aforementioned The aim of this block is to provide a mathematical representation of the ABCDE rule in this paper. The mathematical modeling of the ABCDE heuristic rule is obtained by using a aforementioned in this paper. The mathematical modeling of the ABCDE heuristic rule is obtained combination of hand-crafted image features with Deep Learning generated features. Preliminarily, by using a combination of hand-crafted image features with Deep Learning generated features. the proposed set of hand crafted image features is analyzed. The proposed image features are re-scaled Preliminarily, the proposed set of hand crafted image features is analyzed. The proposed image via logarithmic function with the aim to reduce the range of their variability. We propose the main features are re-scaled via logarithmic function with the aim to reduce the range of their variability. representative subset of the hand-crafted features we developed in our previous pipeline [3]. We We propose the main representative subset of the hand-crafted features we developed in our added new features and replaced others, as we have noted that the overal performance of the method previous pipeline [3]. We added new features and replaced others, as we have noted that the overal was improved. In detail, we added the 7-th order moment (feature F11) and a modiﬁed version of performance of the method was improved. In detail, we added the 7-th order moment (feature F11) the “cosine similarity” feature (F16) with respect to the same used in previous work [3]. We dropped and a modified version of the “cosine similarity” feature (F16) with respect to the same used in features F14, F18–F21 proposed in the previous version of the pipeline [3], as they do not bring any previous work [3]. We dropped features F14, F18–F21 proposed in the previous version of the pipeline improvement in the overall discrimination performance of the algorithm herein described. We proceed [3], as they do not bring any improvement in the overall discrimination performance of the algorithm in the description of the used features in the shown algorithm by indicating with nr and mr the herein described. We proceed in the description of the used features in the shown algorithm by dimension of the bounding box enclosing the ROI (Figure4d), respectively, while p(i,j) represents the indicating with nr and mr the dimension of the bounding box enclosing the ROI (Figure 4d), gray-level intensity of each pre-processed image pixel: respectively, while p(i,j) represents the gray-level intensity of each pre-processed image pixel: X X 1 ∑∑mr nr ( ) F1 =𝐹 log=log( ( 𝑝 𝑖,p(𝑗i,)j )) (3)(3) m∙n i=1 j=1 · 1 𝐹 =𝑙𝑜𝑔 ( Xm(|𝑝Xn (𝑖,𝑗) −𝐹|)) 1 (4) F2 = log(𝑚𝑟 ∙ 𝑛𝑟 ( p(i, j) F1 )) (4) mr nr − i=1 j=1 · 1 m n 𝐹 =log( 1 XX (𝑝(𝑖, 𝑗) −𝐹) ) (5) F = log (𝑚𝑟 ∙ 𝑛𝑟 (p(i, j) F )2) (5) 3 mr nr − 1 · i=1 j=1 𝐹 = p|𝐹| (6) F4 = F3 (6) 𝜋 1 | | m n 𝐹 =log( ∙ X(|𝑝X (𝑖, 𝑗) −𝐹|)) (7) 2π 𝑚∙𝑛1 F5 = log ( ( p(i, j) F1 )) (7) 2 ·m n − · i=1 j=1 𝐹 =log( q|𝐹 −(𝐹) | (8) 2 F6 = log( F3 (F6) (8) 1 −|𝑝(𝑖, 𝑗) −𝐹| 𝐹 =log (9) 𝑚𝑟 ∙ 𝑛𝑟 𝐹

Computation 2019, 7, 44 7 of 16

  3  Xm Xn p(i, j) F   1  1   F7 = log  −   (9) Computation 2019, 7, x FOR PEER REVIEW mr nr  F   7 of 16 · i=1 j=1 4   m n !4  11 X X 𝑚m n𝑛 p(𝑝i(,𝑖,j)𝑗) −𝐹F1  F =  i j −  𝐹 =log8 log  (𝑖− ) ∙ ( 𝑗 − )  (10)(10) 𝑚𝑟mr nr ∙ 𝑛𝑟 − 22 · − 22 F4𝐹  · i=1j=1  5 m n    1 1 X X 𝑚m n𝑛  p|𝑝(i(,𝑖,j)𝑗) −𝐹F1 |  F = log i j  −   (11) 𝐹 =log9  (𝑖− ) ∙ ( 𝑗 − )   (11) 𝑚𝑟mr nr ∙ 𝑛𝑟 −22 · − 22 F4𝐹  · i=1j=1   m n !6  X X ( )   11 𝑚m n𝑛 p𝑝i(,𝑖,j 𝑗) −𝐹F1  𝐹 F=log10 = log  (𝑖− i ) ∙ ( j𝑗 − ) −  (12) mr nr − 2 · − 2 F4  (12) 𝑚𝑟 ∙ 𝑛𝑟 i=1 j=1 2 2 𝐹 ·   m n !7  X X m n p(i, j) F   11 𝑚 𝑛 𝑝(𝑖, 𝑗) −𝐹1  F11 = log i j −  (13) 𝐹 =log mr nr(𝑖−− 2) ∙· (𝑗−−2 ) F4  (13) 𝑚𝑟· ∙ 𝑛𝑟 i=1 j=1 2 2 𝐹 m n 1 XX m n = ( 1 ( 𝑚 𝑛 ( ( ))2)) F12 log i j p i, j (14) 𝐹 =log( mr nr(𝑖−− 2∙·𝑗 − 2∙· (𝑝(𝑖,𝑗)) )) (14) 𝑚𝑟· ∙ 𝑛𝑟 i=1 j=1 2 2 The following Figure5 shows the contribution of the hand-crafted feature F12 for the right The following Figure 5 shows the contribution of the hand-crafted feature F12 for the right characterization of melanocyte distribution over the skin epidermis as well as “ad-hoc” measure of characterization of melanocyte distribution over the skin epidermis as well as “ad-hoc” measure of nevus area/dermoscopy area ratio (ﬁxed input dermo-image size): nevus area/dermoscopy area ratio (fixed input dermo-image size):

Figure 5. Graphical representation of the nevus image processing done by feature F12. Figure 5. Graphical representation of the nevus image processing done by feature F12.

m n 1 XX m n 2 F13 = log (1 ( i 𝑚 j 𝑛 (p(i, j) (i j) ))) (15) mr nr − 2 · − 2 · · − 𝐹 =log( (𝑖− ∙𝑗 − ∙(𝑝(𝑖, 𝑗) ∙ (𝑖−𝑗) ))) (15) 𝑚𝑟· ∙ 𝑛𝑟 i=1 j=1 2 2 mXk Xn k 1 −− m n F14 = log (1 ( i 𝑚 j 𝑛 ( p(i, j) p(i + k, j + k) ))) (16) 𝐹 =log( mr nr(𝑖−− 2∙·𝑗 − 2∙· (|𝑝(𝑖, 𝑗)−−𝑝(𝑖+𝑘,𝑗 +𝑘)|))) (16) 𝑚𝑟 ∙· 𝑛𝑟 i=1 j=1 2 2 1 1 4 1 2 F15 = log m n1 1 (F7) + 1(F8 3) (17) · ·12·mr nr· 16· − 𝐹 =𝑙𝑜𝑔 𝑚∙𝑛∙ ∙ · ∙(𝐹) + ∙ (𝐹 −3) (17)   12m 𝑚𝑟 ∙ 𝑛𝑟 16  round( 2 ) n 2  1  P P m n p(i,j) p(i+1,j) (i j)  F = log  ( i j ) q − · −  16  mr nr  2 2 | |   · = = − |·| − · ( ( ))2 ( ( ))2  · i 1 j 1 p i,j + p i+1,j  n  (18) ⎛ 1 m round( 2 ) 𝑚 𝑛 |𝑝(𝑖, 𝑗) −𝑝(𝑖+1,𝑗2)| ∙ (𝑖−𝑗) 𝐹 =log ∙ ⎛ (|𝑖− P P |∙|𝑗−m |n)∙ p(i,j) p(i,j+1) (i j)  ⎞ ⎜ + i j q − · − ) 𝑚𝑟 ∙ 𝑛𝑟  2 2 2 2 | |  i=1j=1 − · − · (p(i,j))2+(p(i,j+1))2  ⎝ 𝑝(𝑖,) 𝑗 +𝑝(𝑖+1,𝑗) ⎠ ⎝ (18) The following Figure6 shows the contribution of the hand-crafted feature F16 for the right characterization of “similarity and symmetry”𝑚 of the𝑛 skin|𝑝 lesion(𝑖,) 𝑗 −𝑝 i.e.,(𝑖,𝑗 weighted)| +1 ∙ (𝑖−𝑗 average) ⎞ measure of ⎛ ⎞ nevus asymmetry (“A” of+ the(𝑖− ABCDE rule) with∙𝑗− speciﬁc) ∙ weights for border part of nevus;⎟ speciﬁc 2 2 ⎝ 𝑝(𝑖, 𝑗) +𝑝(𝑖,𝑗 +1) ⎠ ⎠

The following Figure 6 shows the contribution of the hand-crafted feature F16 for the right characterization of “similarity and symmetry” of the skin lesion i.e., weighted average measure of nevus asymmetry (“A” of the ABCDE rule) with specific weights for border part of nevus; specific weighted average measure of symmetry ratio between the borders and nevus core and, finally,

Computation 2019, 7, 44 8 of 16

Computation 2019, 7, x FOR PEER REVIEW 8 of 16 weighted average measure of symmetry ratio between the borders and nevus core and, ﬁnally, speciﬁc specificweighted weighted measure measure of symmetry of symmetry distribution distribution of the near of pixelsthe near of nevuspixels segmentedof nevus segmented dermo-image: dermo- image:

Figure 6. Graphical representation of the nevus image processing made by feature F16. Figure 6. Graphical representation of the nevus image processing made by feature F16. As said, the feature F1 to F6 are classical statistical features suitable to characterize melanocyte distributionAs said, in the terms feature of a F statistical1 to F6 are process. classical The statistical features featuresF7 to F15 suitableare a modified to characterize version ofmelanocyte statistical distributionmoments, such in terms that they of a arestatistical useful toprocess. characterize The features the dynamic F7 to F evolution15 are a modified (borders, version radius, of colour statistical and moments,asimmetry) such of the that nevus. they are The useful feature to Fcharacteri16 is a modifiedze the dynamic version of evolution the so-called (borders, “cosine radius, similarity”, colour andused asimmetry) to analyze theof the intrinsic nevus. similarity The feature and F16 symmetry is a modified of the version nevus, of while the so-called the feature “cosineF15 is similarity”, a modified usedadvanced to analyze version the of intrinsic the “Jarque-Bera similarity an index”,d symmetry which of is the able nevus, to point-out while the kurtosis feature and F15 is skewness a modified of advancedtime-series version and is oftenof the applied “Jarque-Bera in the fieldindex”, of financialwhich is markets.able to point-out In the proposed kurtosis work, and skewness we adapted of time-seriesthe Jarque-Bera and indexis often to applied 2D image in analysisthe field with of financial regards markets. to the kurtosis In the andproposed skewness work, features we adapted for the theanalyzed Jarque-Bera pre-processed index to dermoscopic2D image analysis image. with We regards used k = to9 the in Equationkurtosis and (16). skewness features for the analyzedIn order pre-processed to improve dermoscopic the performance image. of the We previous used k = version 9 in Equation of the proposed (16). pipeline, the authors analyzedIn order features to improve obtained the by meansperformance of a modified of the versionprevious of version SC-CNNs of i.e.,the theproposed so-called pipeline, NonLinear the authorsSC-CNNs analyzed (NLSC-CNNs). features Formally,obtained anby NLSC-CNNsmeans of a modified can be mathematically version of SC-CNNs represented i.e., the as follows:so-called NonLinear SC-CNNs (NLSC-CNNs). Formally, an NLSC-CNNs can be mathematically represented dxij(t) P P as follows: C = 1 x + A(i, j; k, l)y (t) + B(i, j; k, l)u (t) dt Rx ij kl kl − C(k,l) Nr(i,j) C(k,l) Nr(i,j) ( ) ∈ ∈ 𝑑𝑥 𝑡 +1 P ( ) ( ) 𝐶 =− 𝑥 C i, j; k, l xkl t 𝑑𝑡 𝑅C(k,l) Nr(i,j) ∈ + P ( ) ( ) ( ) + (19) +𝐴D i, j(;𝑖,k, l 𝑗;y 𝑘,ij 𝑙)𝑦t,(y𝑡)kl +t 𝐵I (𝑖,𝑗; 𝑘,𝑙)𝑢(𝑡) C(k,l) Nr(i,j) (∈,)∈(,) (,)∈(,) (1 i M, 1 j N) + 𝐶1≤( 𝑖,≤ 𝑗; 𝑘, 𝑙)𝑥≤ (𝑡≤) yij(t) = 2 xij(t) + 1 xij(t) 1 (,)∈(,) − − (19) A nonlinear template D(+i,j ;k,l) has been 𝐷(𝑖,𝑗; added 𝑘,𝑙)(𝑦 to the(𝑡),𝑦 model(𝑡)) reported+ 𝐼 in Equation (1) in order to exploit the nonlinearity relationship(,)∈(, between) the outputs of single cell with its neighborhood. The NLSC-CNNs allow to characterize(1𝑖𝑀,1𝑗𝑁) the dependence between the output of the single cell with those of the cells belonging to its neighborhood.1 In this way, it was possible to highlight the dynamic 𝑦 (𝑡) = (𝑥 (𝑡) +1−𝑥 (𝑡) −1) correlation between the processing of individual2 pixel that contain, as mentioned, groups of melanocytic cells. In this way, through the designed NLSC-CNNs, we can study the morphological dynamics of A nonlinear template D(i,j;k,l) has been added to the model reported in Equation (1) in order to melanocytic cells (output of the single cell of the NLSC-CNNs architecture) for each type of processing exploit the nonlinearity relationship between the outputs of single cell with its neighborhood. The defined by the designed cloning template. This result would not have been possible with the CNNs NLSC-CNNs allow to characterize the dependence between the output of the single cell with those model proposed by Chua and Yang nor with that defined by classic SC-CNNs, where the dependence of the cells belonging to its neighborhood. In this way, it was possible to highlight the dynamic between the cell outputs with those of its neighborhood is not directly explained. The NLSC-CNN correlation between the processing of individual pixel that contain, as mentioned, groups of architecture has been used for several applications, including video and image processing [14–16]. melanocytic cells. In this way, through the designed NLSC-CNNs, we can study the morphological The authors have analyzed the target of robust skin lesion discrimination and, after several optimization dynamics of melanocytic cells (output of the single cell of the NLSC-CNNs architecture) for each type tests, designed the following further deep learned features: of processing defined by the designed cloning template. This result would not have been possible with the CNNs model proposed by Chua and Yang nor with that defined by classic SC-CNNs, where  0 0 0   0.8 0.21 0.20   0 0 0   0 0.3 0  the dependence between  the cell outputs with those of its neighborhood  is not directly explained. A =  0 0 0  ; B =  0.20 0.71 0.20  ; C =  0 0 0 ; D =  0.3 0 0.3 ; I = 0.65 (20)         The NLSC-CNN 0 0 0 architecture 0.71 has 0.20 been 0.31 used for several 0 0 applications, 0   0including 0.3 0video and image processing [14–16]. The authors have analyzed the target of robust skin lesion discrimination and, after several optimization tests, designed the following further deep learned features:

Computation 2019, 7, 44 9 of 16

Computation 2019, 7, x FOR PEER REVIEW 9 of 16          0 0 0   0.23 0.95 0.01   0 0 0   0 0 0          A =000 0 0 0  ; B =0.8 0.20 0.21 0.14 0.20 0.11  ; C000=  0 0 0 ; D00.30=  0 0 0 ; I = 0.35 (21)         𝐴 =000 0 0 0;𝐵 =0.20  0.23 0.71 0.20 0.20 0.20 ;𝐶 =000  0;𝐷 0 0  =0.3 00 0.3 0; 0 𝐼=0.65 (20) 000 0.71 0.20 0.31 000 00.30          0000 0 0   0.180.23 0.20 0.95 0.20 0.01  0000 0 0  000 0 0.1 0          A =𝐴=0000 0 0  ; B;𝐵=  =0.200.20 0.22 0.14 0.20 0.11 ;;𝐶C =  =000 0 0 0;𝐷; D = =000 0.1; 0 𝐼=0.35 0.1 ; I = 0.65(21) (22)          0000 0 0   0.660.23 0.20 0.20 0.20 0.20  0000 0 0  000 0 0.1 0  000 0.18 0.20 0.20  000 00.10  𝐴 =0000 0 0 ;𝐵 =0.20  0.13 0.22 0.35 0.20 0.13 ;𝐶 =000  0.1;𝐷 0.2 0 =0.1 0 0 0.1 0; 0𝐼=0.65 (22)         A = 0000 0 0  ; B =0.66 0.20 0.20 0.14 0.20 0.72  ; C =000 0 0.1 0 00.10; D =  0 0 0 ; I = 0.15 (23)          0000 0 0  0.13 0.65 0.35 0.50 0.13 0.51  0.1 0.2 0.2 0 0 0  000 0 0 0  𝐴 = 000 ;𝐵 =0.20 0.14 0.72 ;𝐶 = 00.10;𝐷 =000 ; 𝐼=0.15 (23)  0 0 0   0.22 0.44 0.32   0.2 0 0.2   0 0 0   000 0.65 0.50 0.51  0.200  000  A =   B =   C =   D =   I =  0000 0 0  ; 0.22 0.25 0.44 0.97 0.32 0.59  ; 0.2 0 0 0.3 0.2 0 ; 000 0 0 0 ; 0.35 (24)         𝐴 =0000 0 0 ;𝐵 =0.25 0.71 0.97 0.33 0.59 0.41 ;𝐶 = 00.300.2 0;𝐷 0.2 =0000 0; 𝐼=0.35 0 (24)  000 0.71 0.33 0.41  0.2 0 0.2  000   0 0 0   0.78 0.88 0.65   0 0 0   0 0.5 0   000 0.78 0.88 0.65  000  00.50  A =  0 0 0  ; B =  0.20 0.21 0.20  ; C =  0 0 0 ; D =  0.5 0 0.5 ; I = 0.25 (25) 𝐴 = ;𝐵 = ;𝐶 =  ;𝐷 =  ; 𝐼=0.25  000 0.20 0.21 0.20  000  0.5 0 0.5  (25) 0000 0 0 0.330.33 0.45 0.45 0.41 0.41 0000 0 0 00.500 0.5 0 AtAt the the end end of the described analysis, we we collected 16 16 hand-crafted features features that that we we arranged arranged in in a 32 32 matrix by means of a bi-cubic resizing algorithm [12] as well as six deep learned features a 32 × 32 matrix by means of a bi-cubic resizing algorithm [12] as well as six deep learned features generatedgenerated by by the the NLS-CNNs NLS-CNNs properly configured. conﬁgured. InIn the the following following Figure Figure 77,, thethe authorsauthors reportreport somesome ofof thethe generatedgenerated features:features:

FigureFigure 7. 7. SomeSome instances instances of of the the dermoscopic dermoscopic image image fe featuresatures generated generated by by a a previous previous block. block.

2.3.2.3. The The Deep Deep Learning Learning Classification Classification Block Block TheThe goal goal of of this this block block is is the the proper proper classificati classificationon of offeatures features previously previously described described in order in order to oncologicallyto oncologically discriminate discriminate the theanalyzed analyzed skin skin lesion. lesion. Specifically, Specifically, the objective the objective of this of thispart partof the of proposedthe proposed pipeline pipeline is to is discriminate to discriminate skin skin lesion lesions—benigns—benign (i.e., (i.e., with with a alow low risk risk of of oncological progression)progression) as compared to malignant (i.e., with a high risk of malign progression). OnOn the the basis basis of of this this lesion lesion discrimination discrimination and and in agreement in agreement with with the theattending attending physician, physician, the algorithmthe algorithm will willprovide provide a follow-up a follow-up rate ratefor the for patient. the patient. InIn this this work, work, the the authors authors tried tried to to further further improv improvee the the performance performance of the of theprevious previous version version of the of pipelinethe pipeline [1–3] [1 –by3] byusing using Deep Deep Convolutional Convolutional Neur Neuralal Network Network (ConvNN) (ConvNN) architecture architecture for for further further processingprocessing ofof imageimage features features coming coming from from previous previo blocks.us blocks. The nextThe sections next sections will explain will theexplain structure the structureof the Deep of the ConvNN Deep ConvNN that we propose that wein propose this work: in this work: • Input Layers 32 × 3232 × 11 • × × • Convolution Layer (kernels 3 × 33 × 1,1, 8-filters, 8-filters, Padding) Padding) • × × • Gradient Normalization Layer • • ReLU Layer • • MaxPooling Layer (2 × 2)2) • × • Convolution Layer (kernels 3 × 33 × 1,1, 16-filters, 16-filters, Padding) Padding) •• ReLU Layer × × ReLU Layer •• MaxPooling Layer (2 × 2) MaxPooling Layer (2 2) •• Convolution Layer (kernels× 3 × 3 × 1, 32-filters, Padding) • ReLU Layer The proposed pipeline is shown in the following figure:

Computation 2019, 7, 44 10 of 16

Convolution Layer (kernels 3 3 1, 32-filters, Padding) • × × ReLU Layer • The proposed pipeline is shown in the following figure: In Figure8 the proposed Deep Convolutional Neural Network (ConvNN) architecture is reported. As noticed from the input layer description, the input size is 32 32. Consequently, we resized each × dermoscopic image of the analyzed nevi (included in the PH2 dataset) to a size of 32 32; therefore, × the 2D structure of the used NLSC-CNNs will be composed of 32 32 cells having dynamics defined × by the cloning templates described in Equations (20)–(25). Furthermore, the described 16 hand-crafted image features had previously been arranged in a 32 32 size pattern, becoming in fact a sort of × “morphologic pattern” suitable for identifying each of the analyzed ones. All the described resizing was done using a classical bi-cubic algorithm [12]. In the following paragraphs, each layer of ConvNNs isComputation described: 2019, 7, x FOR PEER REVIEW 10 of 16

Figure 8. The proposed Deep Convolutional Neural Network pipeline. Figure 8. The proposed Deep Convolutional Neural Network pipeline. Input Layer: These numbers correspond to the height, width, and channel size. As said, the input In Figure 8 the proposed Deep Convolutional Neural Network (ConvNN) architecture is data (CNNs processed features) consists of grayscale images, so channel size (color channel) is one. reported. As noticed from the input layer description, the input size is 32 × 32. Consequently, we Convolutional Layer: In the convolutional layer, the first argument is filter size, which is the resized each dermoscopic image of the analyzed nevi (included in the PH2 dataset) to a size of 32 × height and width of the kernel filters the training function uses while scanning the CNNs lesion features 32; therefore, the 2D structure of the used NLSC-CNNs will be composed of 32 × 32 cells having images. In this case, the number 3 indicates that we have used a filter size of 3-by-3. The second dynamics defined by the cloning templates described in Equations (20)–(25). Furthermore, the argument is the number of filters, which is the number of neurons that connect to the same region described 16 hand-crafted image features had previously been arranged in a 32 × 32 size pattern, of the input. This parameter determines the dimension of the so-called “feature maps” generated by becoming in fact a sort of “morphologic pattern” suitable for identifying each of the analyzed ones. the ConvNNs. All the described resizing was done using a classical bi-cubic algorithm [12]. In the following ReLU Layer: The convolutional layer is followed by a nonlinear activation function. We decided paragraphs, each layer of ConvNNs is described: to use the most common activation function i.e., the rectified linear unit (ReLU). MaxInput Pooling Layer: These Layer :numbers In the Deep correspond ConvNN to paradigm,the height, wi thedth, convolutional and channel layers size. As (with said, activation the input functions)data (CNNs are processed sometimes features) followed consists by a down-sampling of grayscale images, operation so channel that reduces size (color the spatial channel) size is ofone. the featureConvolutional map and removes Layer redundant: In the convolutional spatial information. layer, the The first max argument pooling layer is filter returns size, the which maximum is the valuesheight ofand rectangular width of regionsthe kernel of feature filters inputthe training image comingfunction from uses the while previous scanning layer. the CNNs lesion featuresThe images. proposed In pipeline this case, performs the number deep 3 learning indicate ofs that the skinwe have lesion used image a filter features size that of 3-by-3. come from The thesecond previous argument block is and the at number the end theof filters, learned which features is the will number be fed intoof neurons further layersthat connect that are to part the ofsame the overallregion skinof the lesion input. discrimination This parameter system, determines as described: the dimension of the so-called “feature maps” generated by the ConvNNs. Fully-Connected Layer (2 Output Neurons) • ReLU Layer: The convolutional layer is followed by a nonlinear activation function. We decided Softmax Layer •to use the most common activation function i.e., the rectified linear unit (ReLU). Classification Layer • Max Pooling Layer: In the Deep ConvNN paradigm, the convolutional layers (with activation functions)The following are sometimes paragraphs followed describe by a down-samplin the above layers:g operation that reduces the spatial size of the featureFully map Connected and removes Layer: redundant This layer combines spatial information. all the features The learned max by pooling the previous layer layersreturns across the themaximum image tovalues identify of rectangular larger patterns. regions The of last feature fully input connected image layer coming combines from the the previous features tolayer. classify the images.The proposed In this case, pipeline in the performs last fully deep connected learning layer, of the we skin defined lesion the image number features of classes that in come the target from datathe previous i.e., two, block as described and at the in the end previous the learned section features (benign will nevus be fed versus into further malignant layers ones). that are part of the overallSoftmax skin Layer: lesionThe discrimi outputnation of the system, softmax as layer described: consists of positive numbers that sum up to one, which can then be used as classification probabilities by the classification layer. • Fully-Connected Layer (2 Output Neurons) • Softmax Layer • Classification Layer The following paragraphs describe the above layers: Fully Connected Layer: This layer combines all the features learned by the previous layers across the image to identify larger patterns. The last fully connected layer combines the features to classify the images. In this case, in the last fully connected layer, we defined the number of classes in the target data i.e., two, as described in the previous section (benign nevus versus malignant ones). Softmax Layer: The output of the softmax layer consists of positive numbers that sum up to one, which can then be used as classification probabilities by the classification layer. Classification Layer: The final layer is the classification layer. This layer uses the probabilities returned by the softmax activation function for each input to assign the input to one of the mutually exclusive classes and compute loss and performance indexes. The classification output is further processed by the Autonomous Diagnosis System, which suggests a “follow-up rate” according to the detected oncological risk to be agreed with the attending physician. Specifically, the system will

Computation 2019, 7, 44 11 of 16

Classification Layer: The final layer is the classification layer. This layer uses the probabilities returned by the softmax activation function for each input to assign the input to one of the mutually exclusive classes and compute loss and performance indexes. The classification output is further processed by the Autonomous Diagnosis System, which suggests a “follow-up rate” according to the detected oncological risk to be agreed with the attending physician. Specifically, the system will suggest prompt contact with the attending physician if it classifies the analyzed nevus at high risk of malignancy or will suggest a follow-up rate of more than six months if it considers that the analyzed skin lesion is at low risk of oncological progression. Obviously, the physician will be able to decide the follow-up rate on the basis of appropriate medical assessments, which are in fact based on the preventive evaluation performed by the algorithm.

3. Results The proposed method has been trained and tested by using classified skin lesion images included in a database called “PH2”[4]. The PH2 database consists of several dermoscopic images containing benign nevus as well as malignant ones (melanoma) with different morphologic features. The PH2 dataset has been developed for research purposes and to allow a robust comparison between the methods of discriminative analysis of skin lesions for oncological purposes. The dermoscopic images included in the PH2 dataset were acquired during daily clinical activity performed at the Dermatology Service of Hospital Pedro Hispano, Matosinhos, Portugal. As reported in the related documentation [4], the dermoscopic images were acquired through the Tuebinger Mole Analyzer system using a magnification of 20 . The dataset contains 8-bit RGB color images with resolution of × 768 560 pixels. It includes dermoscopic images of melanocytic lesions, including benign lesions × (common, atypical and dysplastic nevus) and several cancerous skin lesions (i.e., melanomas) with different morphologies (diameter, asymmetry, edge regularity, etc.). The PH2 database includes medical annotation of each reported image. The medical diagnosis for each reported lesion was performed by an expert dermatologist. The PH2 dataset was thus properly organized. About 65% of the PH2 dataset images were used as a training set. In the training set thus composed, dermoscopic images of benign skin lesions with different morphological features (common, atypical and dysplastic nevus) are included. Moreover, several dermoscopic images of cancerous lesions (melanomas) have been included in the training set but with different characteristics (diameter, edges, regularity of the contours, etc.). Therefore, the used training set consists of about 130 8-bit RGB dermoscopic images of benign and malignant lesions. In this way, we tried to increase the generalization capability of the proposed deep learning system. The remaining set of images (35%) of the PH2 dataset (also composed of images of benign lesions and melanomas) were used for the testing and validation of the proposed pipeline. In order to provide a robust and efficient benchmarking of the proposed approach, we decided to include a comparison with several similar pipelines validated on the same PH2 database. In order to perform a fair comparison of the proposed method with respect to other approaches, we evaluated our approach by computing the same benchmark indicators as computed in the compared methods [4]. Specifically, we computed the ‘sensibility’ indicator as well ‘specificity’ and cost function ‘C’, which can be computed as reported in the following equation:

c10(1 SE)+c01(1 SP) C= − − c10+c01

c10=1.5 :c01=1

The parameter C includes the term C10 as the weight coefficient of wrong classification (the pipeline classifies “malignant” or high risk a skin lesion, which actually is benign i.e., a false negative FN), while C01 represents the weight coefficient for a False Positive (FP) wrong classification (the pipeline classifies “benign” or low-risk a nevus, which actually is malignant or high-risk). From a simple mathematical analysis of the model used for computing C, it is clear that the system considers a False Positive more Computation 2019, 7, 44 12 of 16

dangerous with respect to a False Negative for reasons that appear to be easily understandable, because they are linked to the extremely high danger of failure to report a malignant lesion, rather than a wrong classiﬁcation of benign lesion, which could lead to closer controls and nothing more. Table1 reports performance benchmark comparison with respect to several approaches applied to the same PH2 database [17,18]. In Table2, further performance indicators related to the proposed approach are reported.

Computation 2019, 7, x FOR PEER REVIEW Table 1. Benchmarks comparison for the proposed12 of 16 pipeline. ComputationComputation 2019, 7, 2019x FOR, 7 ,PEER x FOR REVIEW PEER REVIEW 12 of 1612 of 16 Computation 2019, 7, x FOR PEER REVIEW 12 of 16 Computation 2019In, Table7, x FOR 3, PEER the REVIEWauthors reported analyzed skin lesionsMethod and [4, 16related] classification Sensitivity 12made of Speciﬁcity 16 by the C proposed pipeline. In TableIn Table3, the 3,authors the authors reported reported analyzed analyzed skin lesions skin lesions and related and related classification classification made madeby the by the In Table 3, the authors reported analyzed skin lesionsLM [2] and related classification 97% made 95% by the 0.0682 In Table 3, the authors reportedproposed analyzedproposed pipeline. skin pipeline. lesions andSAE related [3] classification 98.5% made by the 98.1% 0.0166 proposed pipeline. Computation 2019, 7, x FOR PEER REVIEW 12 of 16 proposed pipeline. Table 1. Benchmarks comparison for Proposedthe proposed pipeline. 99.1% 98.5% 0.0114 TableGM 1.Table Benchmarks (Color) 1. Benchmarks comparison comparison 90% for the forproposed the 89% proposed pipeline. 0.1040pipeline. TableIn 1.Table Benchmarks 3, the authorscomparison reported for the proposedanalyzedC pipeline. skin lesions and related classification made by the Table 1. Benchmarks comparison for the proposedLM (Color) pipeline. 93% 84% 0.1060 proposed pipeline. 0.0682 C C LM (Textures)C 88% 76% 0.1680 GF-C6–kNNC 0.0166 100% 71%0.06820.0682 0.1160 0.0682 Table 1.GF-C1–kNN Benchmarks0.0682 comparison0.0114 88% for the proposed 81% pipeline.0.0166 0.0166 0.1490 0.0166 G–AdaBoost0.0166 0.1040 96% 77%0.01140 0.117.0114 0.0114 C LF-BoF0.0114 0.1060 98% 79%0.10400.1040 0.100 0.1040 0.0682 0.1040 0.1680 0.10600.1060 0.1060 0.0166 Table 2. Performance0.1060 measure0.1160 indicators of the proposed0.16800.1680 approach. 0.1680 0.0114 0.1680 0.1490 0.11600 .1160 0.1160 0.1040 0.1160Method Accuracy F1-Score0.14900.1490 0.1490 0.1060 0.1490Proposed 99.00% 99.07%0.1680 0.1160 In Table3, the authors reported analyzed skin lesions and0.1490 related classiﬁcation made by the proposed pipeline.

Table 3. Classiﬁed nevus versus classiﬁcation made by the proposed pipeline.

PH2 Image Proposed Pipeline Classiﬁcation in PH2 Follow-Up Rate

Non-melanoma Common Nevus >=6 months (Benign Nevus) (Benign Nevus)

Non-melanoma Dysplastic Nevus >=6 months (Benign Nevus) (Benign Nevus)

Melanoma Melanoma Contact physician (Cancer lesion) (Cancer lesion) immediately

Melanoma Melanoma Contact physician (Cancer lesion) (Cancer lesion) immediately Computation 2019, 7, 44 13 of 16

In Table2, we reported further performance measure indicators, speciﬁcally, the accuracy index of the overall pipeline as well as the F1-score (also known as F-measure) [19]. Moreover, we showcased theComputation ROC diagram 2019, 7, x inFOR Figure PEER 9REVIEW. 13 of 16

Figure 9. The ROC diagram of the proposed method. Figure 9. The ROC diagram of the proposed method. Both from the analysis of the ROC reported in Figure9 and the common performance indices Both from the analysis of the ROC reported in Figure 9 and the common performance indices (Sensitivity, Specificity, F1-score and Accuracy) reported in Tables1 and2, it is evident that the proposed (Sensitivity, Specificity, F1-score and Accuracy) reported in Table 1 and Table 2, it is evident that the method has remarkable capabilities of identification (sensitivity) and discrimination (specificity) of proposed method has remarkable capabilities of identification (sensitivity) and discrimination skin lesions. With accuracy close to 100%, the proposed pipeline was able to identify skin lesions at (specificity) of skin lesions. With accuracy close to 100%, the proposed pipeline was able to identify high risk of malignancy, thus allowing the dermatologist/oncologist to promptly schedule treatment. skin lesions at high risk of malignancy, thus allowing the dermatologist/oncologist to promptly It is possible to verify this by analyzing the malignant lesions present in the PH2 dataset (used for the schedule treatment. It is possible to verify this by analyzing the malignant lesions present in the PH2 training and testing of the proposed method); there are dermoscopic images of cancerous lesions at dataset (used for the training and testing of the proposed method); there are dermoscopic images of the limit of the indications in the mentioned ABCDE criterion, which can escape visual inspection cancerous lesions at the limit of the indications in the mentioned ABCDE criterion, which can escape by expert physicians (we refer to cases of melanoma with a small diameter or edges not completely visual inspection by expert physicians (we refer to cases of melanoma with a small diameter or edges irregular). The proposed pipeline was also able to correctly identify malignant lesions, highlighting to not completely irregular). The proposed pipeline was also able to correctly identify malignant lesions, the physician the high risk of the nevus, as well as suggesting a follow-up rate. Moreover, in order to highlighting to the physician the high risk of the nevus, as well as suggesting a follow-up rate. avoid unnecessary biopsies, a lot of work has also been done on the discriminative capacity (specificity) Moreover, in order to avoid unnecessary biopsies, a lot of work has also been done on the of the proposed pipeline. Even in the case of dysplastic or atypical nevus (included both in the training discriminative capacity (specificity) of the proposed pipeline. Even in the case of dysplastic or set and in the set of dermoscopic images used for testing/validation) the proposed algorithm was able atypical nevus (included both in the training set and in the set of dermoscopic images used for to correctly characterize the benignity of the lesion by appropriately evaluating the similarity features testing/validation) the proposed algorithm was able to correctly characterize the benignity of the of the analyzed lesion (thanks to the designed hand-crafted features). The use of the features generated lesion by appropriately evaluating the similarity features of the analyzed lesion (thanks to the by the convolution system has allowed to significantly improve the specificity of the pipeline proposed designed hand-crafted features). The use of the features generated by the convolution system has compared to the previous versions, as the used ConvNN allowed to extract latent features otherwise allowed to significantly improve the specificity of the pipeline proposed compared to the previous not detectable by the classic techniques based on hand-crafted features, which allowed the classifier to versions, as the used ConvNN allowed to extract latent features otherwise not detectable by the properly separate the two classes of skin lesions (benign versus malignant). classic techniques based on hand-crafted features, which allowed the classifier to properly separate 4.the Discussion two classes and of skin Conclusions lesions (benign versus malignant).

4. DiscussionA simple and comparison Conclusions between benchmark results of the methods reported in Tables1 and2 shows promising performance of the herein reported method with respect to other similar pipelines, A simple comparison between benchmark results of the methods reported in Table 1 and Table also including the previous version of the same pipeline [1–3]. An innovative combination between 2 shows promising performance of the herein reported method with respect to other similar pipelines, re-designed hand-crafted lesion image features with further features generated by the NLSC-CNNs also including the previous version of the same pipeline [1–3]. An innovative combination between system and innovative Deep Convolutional Neural Network framework considerably improves the re-designed hand-crafted lesion image features with further features generated by the NLSC-CNNs overall discrimination performance of the system. As reported in Table1, it is clear that several system and innovative Deep Convolutional Neural Network framework considerably improves the approaches proposed in scientiﬁc literature increase sensitivity of the pipelines to the disadvantage overall discrimination performance of the system. As reported in Table 1, it is clear that several of ‘speciﬁcity’ or vice versa. Unfortunately, a major drawback of poor skin lesion discrimination approaches proposed in scientific literature increase sensitivity of the pipelines to the disadvantage of ‘specificity’ or vice versa. Unfortunately, a major drawback of poor skin lesion discrimination specificity is more nevus biopsy for patients. The proposed approach tries to achieve an excellent trade-off between sensitivity and specificity, thus offering the physician a practical tool that can be used in everyday medical practice. The development of the proposed approach as well as training and validation over the PH2 database was made in MATLAB 2018b full toolboxes environment with

Computation 2019, 7, 44 14 of 16 specificity is more nevus biopsy for patients. The proposed approach tries to achieve an excellent trade-off between sensitivity and specificity, thus offering the physician a practical tool that can be used in everyday medical practice. The development of the proposed approach as well as training and validation over the PH2 database was made in MATLAB 2018b full toolboxes environment with ad-hoc NLSC-Cellular Neural Network functions library. The time performance of the proposed method is acceptable, as the proposed pipeline is able to analyze a single nevus in about 2.5/3 s (we tested the pipeline in a PC Intel Core having 12 Cores 64 Gbyte of RAM and a GPU Nvidia GTX 2030). The proposed pipeline is currently being ported from a MATLAB environment to an embedded platform based on STM32 [19]. We are extending the skin lesions image dataset in order to better validate the discrimination performance of the proposed approach. Moreover, we have been studying the use of physiological signals acquired in the area of the skin lesion to extract ad-hoc dynamic features to be used for nevus classification [20–25] as well as an ad-hoc chemotherapy approach, driven by features detected trough the proposed approach [26]. Specifically, through the innovative approaches used by the authors to characterize physiological signals related to blood-arterial flow activity (including the PhotoPlethysmoGraphic (PPG) signal analysis), we have been analyzing the underlying evolution of skin lesions for early detection of the lymphatic or hematic invasiveness of this lesion because vascular or lymphatic invasion is one of the most discriminating indicators of the malignancy or benignity. A recent extension of the proposed method is being validated on a larger dataset than the one used in this article, which includes over 10,000 suitably cataloged skin lesions. Specifically, the authors, in addition to using recent deep learning approaches based on convolutive architecture with dilation, are exploiting a particular cross-correlation measure formulated originally by the authors themselves in [27] and now modified to be applied to discriminative analysis of skin lesions. In a future work, the authors aim to describe the preliminary results they have achieved using this innovative approach.

5. Patent Francesco Rundo, Giuseppe Luigi Banna. Methods and System for Analyzing Skin Lesions, IT Patent Application Number 102016000121060, 29 November 2016—USA Application Number 15607178, 26 May 2017.

Author Contributions: Conceptualization, methodology, software, validation, investigation, F.R.; formal analysis, G.L.B., writing—original draft preparation, F.R.; writing—review and editing, S.C. Funding: This research received no external funding. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Conoci, S.; Rundo, F.; Petralia, S.; Battiato, S. Advanced skin lesion discrimination pipeline for early melanoma cancer diagnosis towards PoC devices. In Proceedings of the IEEE 2017 European Conference on Circuit Theory and Design (ECCTD), Catania, Italy, 4–6 September 2018. 2. Rundo, F.; Conoci, S.; Banna, G.L.; Stanco, F.; Battiato, S. Bio-Inspired Feed-Forward System for Skin Lesion Analysis, Screening and Follow-Up. In Image Analysis and Processing—ICIAP 2017; Lecture Notes in Computer Science (LNCS Including Subseries Lecture Notes in Artiﬁcial Intelligence and Lecture Notes in Bioinformatics); Springer International Publishing: Cham, Switzerland, 2017; Volume 10485, pp. 399–409. 3. Rundo, F.; Conoci, S.; Banna, G.L.; Ortis, A.; Stanco, F.; Battiato, S. Evaluation of Levenberg–Marquardt neural networks and stacked autoencoders clustering for skin lesion analysis, screening and follow-up. IET Comput. Vis. 2018, 12, 957–962. [CrossRef] 4. Mendonça, T.; Ferreira, P.M.; Marques, J.S.; Marcal, A.R.; Rozeira, J. PH2-A dermoscopic image database for research and benchmarking. In Proceedings of the 35th International Conference of the IEEE Engineering in Medicine and Biology Society, Osaka, Japan, 3–7 July 2013. Computation 2019, 7, 44 15 of 16

5. Sathiya, S.B.; Kumar, S.S.; Prabin, A. A survey on recent computer-aided diagnosis of Melanoma. In Proceedings of the 2014 International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT), Kanyakumari, India, 10–11 July 2014. 6. Xie, F.; Fan, H.; Li, Y.; Jiang, Z.; Meng, R.; Bovik, A. Melanoma Classification on Dermoscopy Images Using a Neural Network Ensemble Model. IEEE Trans. Med. Imaging 2017, 36, 849–858. [CrossRef][PubMed] 7. Majtner, T.; Yildirim-Yayilgan, S.; Hardeberg, J.Y. Combining deep learning and hand-crafted features for skin lesion classification. In Proceedings of the Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA), Oulu, Finland, 12–15 December 2016. 8. Guo, Y.; Ashour, A.S.; Si, L.; Mandalaywala, D.P. Multiple Convolutional Neural Network for Skin Dermoscopic Image Classification. In Proceedings of the IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Louisville, KY, USA, 6–8 December 2018; pp. 365–369. 9. Ayan, E.; Ünver, H.M. Data augmentation importance for classification of skin lesions via deep learning. In Proceedings of the 2018 Electric Electronics, Computer Science, Biomedical Engineerings’ Meeting (EBBT), Istanbul, Turkey, 18–19 April 2018; pp. 1–4. 10. Celebi, M.E.; Wen, Q.; Hwang, S.; Iyatomi, H.; Schaefer, G. Lesion border detection in dermoscopy images using ensembles of thresholding methods. Skin Res. Technol. 2013, 19, e252–e258. [CrossRef][PubMed] 11. Fukushima, K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 1980, 36, 93–202. [CrossRef] 12. Gonzalez, R.C.; Woods, R.E. Digital Image Processing, 3rd ed.; Prentice Hall: Upper Saddle River, NJ, USA, 2007. 13. Chua, L.O.; Yang, L. Cellular Neural Networks: Theory. IEEE Trans. Circuits Syst. 1988, 35, 1257–1272. [CrossRef] 14. Arena, P.; Baglio, S.; Fortuna, L.; Manganaro, G. Dynamics of state controlled CNNs. In Proceedings of the 1996 IEEE International Symposium on Circuits and Systems, Circuits and Systems Connecting the World, ISCAS 96, Atlanta, GA, USA, 15 May 1996. 15. Hagan, M.T.; Menhaj, M. Training feed-forward networks with Marquardt algorithm. IEEE Trans. Neural Netw. 1994, 5, 989–993. [CrossRef][PubMed] 16. Duan, S.; Hu, X.; Dong, Z.; Wang, L.; Mazumder, P. Memristor-Based Cellular Nonlinear/Neural Network: Design, Analysis, and Applications. IEEE Trans. Neural Netw. Learn. Syst. 2015, 26, 1202–1213. [CrossRef] [PubMed] 17. Bengio, Y. Learning Deep Architectures for AI. Found. Trends Mach. Learn. 2009, 2, 1–127. [CrossRef] 18. Barata, C.; Ruela, M.; Francisco, M.; Mendonça, T.; Marques, J.S. Two Systems for the Detection of Melanomas in Dermoscopy Images using Texture and Color Features. IEEE Syst. J. 2013, 8, 965–979. [CrossRef] 19. Derczynski, L. Complementarity, F-score, and NLP Evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23–28 May 2016. 20. STM32 32-bit ARM Cortex MCUs. Available online: http://www.st.com/en/microcontrollers/stm32-32-bit- arm-cortexmcus.html?querycriteria=productId=SC1169 (accessed on 1 July 2019). 21. Rundo, F.; Conoci, S.; Ortis, A.; Battiato, S. An Advanced Bio-Inspired PhotoPlethysmoGraphy (PPG) and ECG Pattern Recognition System for Medical Assessment. Sensors 2018, 18, 405. [CrossRef][PubMed] 22. Rundo, F.; Ortis, A.; Battiato, S.; Conoci, S. Advanced Bio-Inspired System for Noninvasive Cuff-Less Blood Pressure Estimation from Physiological Signal Analysis. Computation 2018, 6, 46. [CrossRef] 23. Mazzillo, M.; Maddiona, L.; Rundo, F.; Sciuto, A.; Libertino, S.; Lombardo, S.; Fallica, G. Characterization of sipms with nir long-pass interferential and plastic filters. IEEE Photonics J. 2018, 10.[CrossRef] 24. Rundo, F.; Petralia, S.; Fallica, G.; Conoci, S. A nonlinear pattern recognition pipeline for PPG/ECG medical assessments. In Sensors. CNS 2018; Lecture Notes in Electrical Engineering; Springer International Publishing: Cham, Switzerland, 2019; Volume 539, pp. 473–480. 25. Vinciguerra, V.; Ambra, E.; Maddiona, L.; Romeo, M.; Mazzillo, M.; Rundo, F.; Fallica, G.; di Pompeo, F.; Chiarelli, A.M.; Zappasodi, F.; et al. PPG/ECG multisite combo system based on SiPM technology. In Sensors. CNS 2018; Lecture Notes in Electrical Engineering; Springer International Publishing: Cham, Switzerland, 2019; Volume 539, pp. 105–109. 26. Banna, G.L.; Camerini, A.; Bronte, G.; Anile, G.; Addeo, A.; Rundo, F.; Zanghi, G.; Lal, R.; Libra, M. Oral metronomic vinorelbine in advanced non-small cell lung cancer patients unfit for chemotherapy. Anticancer Res. 2018, 38, 3689–3697. [CrossRef][PubMed] Computation 2019, 7, 44 16 of 16

27. Ortis, A.; Rundo, F.; Di Giore, G.; Battiato, S. Adaptive Compression of Stereoscopic Images. In Image Analysis and Processing—ICIAP 2013; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2013; Volume 8156, pp. 391–399.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).