Classification of Bainitic Structures Using Textural Parameters

metals

Article Classiﬁcation of Bainitic Structures Using Textural Parameters and Machine Learning Techniques

Martin Müller 1,2,*, Dominik Britz 1,2 , Laura Ulrich 1, Thorsten Staudt 3 and Frank Mücklich 1,2

1 Department of Materials Science, Chair of Functional Materials, Saarland University, 66123 Saarbrücken, Germany; [email protected] (D.B.); [email protected] (L.U.); [email protected] (F.M.) 2 Material Engineering Center Saarland, Saarland University, 66123 Saarbrücken, Germany 3 AG der Dillinger Hüttenwerke, Werkstraße 1, 66763 Dillingen/Saar, Germany; [email protected] * Correspondence: [email protected]; Tel.: +49-681-3027-0548

Received: 7 April 2020; Accepted: 9 May 2020; Published: 12 May 2020

Abstract: Bainite is an essential constituent of modern high strength steels. In addition to the still great challenge of characterization, the classification of bainite poses difficulties. Challenges when dealing with bainite are the variety and amount of involved phases, the fineness and complexity of the structures and that there is often no consensus among human experts in labeling and classifying those. Therefore, an objective and reproducible characterization and classification is crucial. To achieve this, it is necessary to analyze the substructure of bainite using scanning electron microscope (SEM). This work will present how textural parameters (Haralick features and local binary pattern) calculated from SEM images, taken from specifically produced benchmark samples with defined structures, can be used to distinguish different bainitic microstructures by using machine learning techniques (support vector machine). For the classification task of distinguishing pearlite, granular, degenerate upper, upper and lower bainite as well as martensite a classification accuracy of 91.80% was achieved, by combining Haralick features and local binary pattern.

Keywords: bainite; microstructure classiﬁcation; textural parameters; Haralick parameters; local binary pattern; machine learning; support vector machine

1. Introduction Bainite is a typical constituent of modern high strength steels, combining high strength and high toughness and thereby making it interesting for many applications. In order to adjust desired strength or toughness levels in these steels, it is essential to know and understand what types of bainite are present depending on chemical composition and processing steps. The characterization or classification of bainite, however, is a difficult task because of the variety and amount of involved phases as well as the fineness and complexity of the structures. Moreover, the lack of consensus among human experts in labeling and classifying bainitic structures further complicates this task. There is also no consistent nomenclature to describe or label bainitic microstructures, and many different classification schemes can be found in literature. There are two concepts for classification schemes, the first is to describe the arrangement of ferrite and carbon-rich phase as a whole, using one integral expression. The oldest and most common known descriptions of this kind are upper and lower bainite [1]. Other existing classification schemes were suggested by Ohmori et al. [2], Bramfitt and Speer [3] or by Lotter and Hougardy [4] and Aaronson et al. [5]. Furthermore, summaries of several classification schemes can be found in [5] and [6]. One of the most well-known schemes is proposed by Zajac et al. [7]. Five different bainitic structures are distinguished. Their schematic structures are shown in Figure1. Upper bainite consists of lath-like ferrite with cementite on the lath boundaries. Instead of

Metals 2020, 10, 630; doi:10.3390/met10050630 www.mdpi.com/journal/metals Metals 2020, 10, 630 2 of 19 Metals 2020, 10, x FOR PEER REVIEW 2 of 19 cementite,martensite-retained degenerated austenite upper constituents bainite has so-called [1] (MA) martensite-retained on the lath boundaries. austenite Lower constituents bainite consists [1] (MA) of onlath-like the lath ferrite boundaries. with cementite Lower bainite precipitated consists ofinside lath-like the ferrite withlaths. cementite Degenerated precipitated lower insidebainite the is ferriteagain with laths. MA Degenerated constituents lower instead bainite of cementite. is again with Granular MA constituents bainite is defined instead as of irregular cementite. ferrite Granular with bainitecarbon-rich is deﬁned second as irregularphases distributed ferrite with carbon-richbetween th secondese irregular phases grains. distributed Typically, between the these carbon-rich irregular grains.second Typically,phase is thenot carbon-rich cementite, second but any phase tran issformation not cementite, product but any that transformation can develop product from thatcarbon-enriched can develop austenite. from carbon-enriched They investigated austenite. five types They of investigatedsecond phases: ﬁve (i) types degenerated of second pearlite phases: or (i)debris degenerated of cementite, pearlite (ii) orbainite, debris (iii) of cementite, mixture of (ii) “incomplete” bainite, (iii) mixturetransformation of “incomplete” products, transformation (iv) MA, and products,(v) martensite. (iv) MA, The and advantage (v) martensite. of this scheme The advantage is that it of covers this scheme various is thatbainitic it covers microstructures various bainitic and microstructurescan be easily used and and can understood be easily used in common and understood parlance. in common parlance.

Figure 1. Schematics of the 5 bainite types suggested by Zajac [7]. Diagram is modified according to [7]. Figure 1. Schematics of the 5 bainite types suggested by Zajac [7]. Diagram is modified according to [7]. The second concept for bainite classification is to describe the ferritic and the carbon-rich phase separately. This has been done in [8–11]. For example, in Gerdemann [9], the ferritic phase is The second concept for bainite classification is to describe the ferritic and the carbon-rich phase described by its crystallographic structure and morphology and the secondary phase by its location, separately. This has been done in [8–11]. For example, in Gerdemann [9], the ferritic phase is crystallographic structure, and morphology. From these descriptions, certain letters are taken to write described by its crystallographic structure and morphology and the secondary phase by its location, an abbreviation for the microstructure. With this classification concept it is more of a pure description crystallographic structure, and morphology. From these descriptions, certain letters are taken to of the microstructure without the subjective component of pressing it into one integral expression. write an abbreviation for the microstructure. With this classification concept it is more of a pure However, those coded descriptions are hard to use in common speech. description of the microstructure without the subjective component of pressing it into one integral Considering the aforementioned challenges when dealing with bainite, it is not surprising that expression. However, those coded descriptions are hard to use in common speech. not many approaches for an automated classification of steel microstructures, including bainite Considering the aforementioned challenges when dealing with bainite, it is not surprising that subclasses can be found in the literature. Gola et al. [12] used a combination of morphological and not many approaches for an automated classification of steel microstructures, including bainite textural parameters with a support vector machine (SVM) to classify the carbon-rich second phase subclasses can be found in the literature. Gola et al. [12] used a combination of morphological and of two phase steels into pearlite, bainite, and martensite. However, bainite subclasses were not textural parameters with a support vector machine (SVM) to classify the carbon-rich second phase of considered, as all structures that were neither pearlite nor martensite were put into one bainite class. two phase steels into pearlite, bainite, and martensite. However, bainite subclasses were not Azimi et al. [13] worked with the same data set, but used deep learning methods for classification considered, as all structures that were neither pearlite nor martensite were put into one bainite class. of pearlite, bainite, martensite, and tempered martensite, however, bainite subclasses were also not Azimi et al. [13] worked with the same data set, but used deep learning methods for classification of included. Zajac et al. [14] use misorientation angle distribution from electron backscatter diffraction pearlite, bainite, martensite, and tempered martensite, however, bainite subclasses were also not (EBSD) measurements to differentiate granular, upper and lower bainite. Tsuitsui et al. [15] used included. Zajac et al. [14] use misorientation angle distribution from electron backscatter diffraction misorientation parameters and variant pairs from EBSD data to distinguish bainite formed at high and (EBSD) measurements to differentiate granular, upper and lower bainite. Tsuitsui et al. [15] used low temperatures as well as martensite and bainite–martensite mixtures. Komenda et al. [16] used misorientation parameters and variant pairs from EBSD data to distinguish bainite formed at high trainable segmentation to distinguish ferrite, pearlite, martensite, upper and lower bainite in sintered and low temperatures as well as martensite and bainite–martensite mixtures. Komenda et al. [16] steel, however the segmentation result is shown in a light optical microscope picture and differences used trainable segmentation to distinguish ferrite, pearlite, martensite, upper and lower bainite in between martensite, lower and upper bainite are not discussed in detail. In Miyama et al. [17] sintered steel, however the segmentation result is shown in a light optical microscope picture and upper and lower bainite are distinguished by calculating morphological parameters of the cementite differences between martensite, lower and upper bainite are not discussed in detail. In Miyama et al. precipitates. Banerjee et al. [18] differentiate ferrite, bainite, and martensite by using intensity values [17] upper and lower bainite are distinguished by calculating morphological parameters of the as well as density of substructure particles, whereas Paul et al. [19] use regional contour pattern and cementite precipitates. Banerjee et al. [18] differentiate ferrite, bainite, and martensite by using local entropy to segment ferrite, martensite and bainite in dual-phase-steels. Regarding the suggested intensity values as well as density of substructure particles, whereas Paul et al. [19] use regional contour pattern and local entropy to segment ferrite, martensite and bainite in dual-phase-steels. Regarding the suggested classifications schemes for bainite mentioned earlier, there are no reported

Metals 2020, 10, 630 3 of 19 classifications schemes for bainite mentioned earlier, there are no reported automated workflows who use the differences cited in the definition of these bainite classes for a classification task. The analysis of the texture of an image is a promising way to distinguish images in general but also steel microstructure images. Image texture is the “spatial arrangement of color or intensities in an image” [20]. Image texture based analysis methods include Haralick textural parameters [21] and local binary patterns [22], amongst others. Webel et al. [23] developed a new approach based on Haralick textural parameters. Instead of calculating the parameters only at angles of 0◦, 45◦, 90◦, and 135◦, they are calculated from the 1◦ stepwise rotation of images, making the values independent of the original texture orientation. By doing this, they were able to distinguish pearlite, lower bainite, and martensite. Local binary pattern are for example used in [24] to distinguish phases in ferrite–pearlite and martensite–austenite microstructures. In order to use a machine learning classification approach, it is important to assign the ground truth in the most objective way. When dealing with complex microstructures this can be challenging. One way to make the ground truth assignment more objective is to use reference samples where there is no doubt about the present microstructures. In this work, bainitic reference samples are used and analyzed based on the workflows suggested by Webel et al. and Gola et al., which is feature extraction by using Haralick textural parameters, now also complemented by using local binary pattern, followed by machine learning classification using a support vector machine, in order to demonstrate the feasibility of a classification of bainite subclasses.

2. Materials and Methods

2.1. Material The assignment of the ground truth for machine learning classification has a subjective component that should not be underestimated. Industrial steel samples usually exhibit complex microstructures, with bainitic phase constituents often being small or inhomogeneous. This makes it difficult to extract ground truth parameters from these regions in an objective and statistically secured way. For this reason, samples were specifically produced using a quenching dilatometer (TA Instruments 805 A/D, TA Instruments, New Castle, DE, USA, Hüllhorst, Germany) to obtain bainitic microstructures. Sample material was taken from an industrial steel grade, capable of achieving a typical yield strength range of 1050–1250 MPa, after hot rolling and heat treatment. The carbon content of the material is 0.22 wt.%, alloying elements are manganese, chromium and nickel as well as microalloying additions Nb, Ti, and B, see Table1. Because of industrial collaboration, exact microalloying amounts cannot be given. This composition was chosen after extensive review of literature about bainitic steels and correlations to their chemistry and processing conditions. E.g. in Zajac et al. [14] similar steel compositions (steel group “II”) yielded predominant bainitic structures and also allowed to adjust different bainite types, including lower bainite which typically doesn’t form in low-carbon steels, depending on the cooling rate.

Table 1. Chemical composition of the used steel, in weight %.

C Mn Si Cr Ni Nb Ti B 0.22 0.90 0.40 0.20 0.20 Microalloying additions

Samples with a diameter of 4 mm and length of 10 mm were machined for use in the quenching dilatometer. After identical austenitization at 1000 ◦C for 10 min, samples were continuously cooled with diﬀerent cooling rates, ranging from 2 K/s to the maximum cooling rate of 278 K/s, in order to get diﬀerent bainitic microstructures (Figure2). Metals 2020, 10, x FOR PEER REVIEW 4 of 19

Metals 2020, 10, x FOR PEER REVIEW 4 of 19 Metals 2020, 10, 630 4 of 19

Figure 2. Time–temperature-profile of sample production experiments: after heating (1,2) samples are austenitized at 1000 °C for 10 min (3), and then continuously cooled with varying cooling rates (4).Figure 2. Time–temperature-profile of sample production experiments: after heating (1,2) samples are Figure 2. Time–temperature-profile of sample production experiments: after heating (1,2) samples austenitized at 1000 ◦C for 10 min (3), and then continuously cooled with varying cooling rates (4). 2.2. Sampleare austenitizedPreparation at 1000 °C for 10 min (3), and then continuously cooled with varying cooling rates 2.2. Sample(4). Preparation The samples were ground using 80–1200 grid SiC papers and then subjected to 6, 3, and finally, 1 μm diamondThe samples polishing were to ground obtain usingsmooth 80–1200 surfaces grid for SiC subsequent papers and etching. then subjectedMetallographic to 6,3, etching and finally, was µ2.2. Sample Preparation carried1 m diamond out by submerging polishing topolished obtain sa smoothmple surfaces surfaces into for a subsequent mixture of et etching.hanol and Metallographic nitric acid (2 vol. etching %), alsowas called carriedThe “Nital” samples out by etching. submergingwere ground polishedusing 80–1200 sample grid surfaces SiC pape intors and a mixture then subjected of ethanol to 6, and 3, and nitric finally, acid 1 (2μ vol.m diamond %), also calledpolishing “Nital” to obtain etching. smooth surfaces for subsequent etching. Metallographic etching was 2.3.carried Microscopy out by submerging polished sample surfaces into a mixture of ethanol and nitric acid (2 vol. %), 2.3.also Microscopy called “Nital” etching. Each sample was imaged in a scanning electron microscope (Supra FE-SEM, Carl Zeiss Each sample was imaged in a scanning electron microscope (Supra FE-SEM, Carl Zeiss Microscopy Microscopy GmbH, Jena, Germany) using secondary electron contrast at a magnification of 2000× with GmbH,2.3. Microscopy Jena, Germany) using secondary electron contrast at a magnification of 2000 with an image an image size of 2048 × 1536 pixels. SEM was operated at an acceleration voltage of 5 kV,× an aperture of size of 2048 1536 pixels. SEM was operated at an acceleration voltage of 5 kV, an aperture of 30 µm 30 μm toEach set ×thesample probe wascurrent imaged and a inworking a scanning distance electron of 5 mm. microscope To get reference (Supra images FE-SEM, with Carljust oneZeiss to set the probe current and a working distance of 5 mm. To get reference images with just one phase phaseMicroscopy constituent GmbH, and toJena, avoid Germany) any potential using secondaryheterogeneities, electron the contrast captured at imagesa magnification were later of cropped2000× with constituent and to avoid any potential heterogeneities, the captured images were later cropped to to 350an image × 350 sizepx, equalof 2048 to × 9.701536 × pixels. 9.70 μ SEMm. During was operated image acquisition,at an acceleration care was voltage taken of not5 kV, to ancut aperture off the of 350 350 px, equal to 9.70 9.70 µm. During image acquisition, care was taken not to cut off the greyscale30× μm tohistogram. set the probe Also, current because× and the a image working contrast distance depends of 5 mm. on the To intensityget reference of etching images and with sample just one greyscale histogram. Also, because the image contrast depends on the intensity of etching and sample preparation,phase constituent all images and were to avoid subsequently any potential norm alizedheterogeneities, regarding thetheir captured gray value images histogram. were later cropped preparation,to 350 × 350 all px, images equal were to 9.70 subsequently × 9.70 μm. normalizedDuring image regarding acquisition, their care gray was value taken histogram. not to cut off the greyscale histogram. Also, because the image contrast depends on the intensity of etching and sample 2.4.2.4. Calculation Calculation of of Haralick Haralick Parameters Parameters preparation, all images were subsequently normalized regarding their gray value histogram. TheThe textural textural parameters parameters developed developed by by Haralick Haralick et et al. al. [21] [21 ]essentially essentially describe describe how how often often a a gray gray valuevalue2.4. appears appearsCalculation in in theof the Haralick image image in inParameters a a certain spatial spatial relati relationshiponship to to another another gray gray value. value. For For this, this, the the gray gray levellevel co-occurrence co-occurrence matrix matrix (GLCM) (GLCM) of of the the image image is is calculated. calculated. From From the the GLCM GLCM several several parameters parameters representingThe textural the image parameters texture can developed be calculated. by Haralick This is illustratedet al. [21] essentiallyin Figure 3. describe how often a gray representingvalue appears the imagein the image texture in can a certain be calculated. spatial Thisrelati isonship illustrated to another in Figure gray3. value. For this, the gray level co-occurrence matrix (GLCM) of the image is calculated. From the GLCM several parameters representing the image texture can be calculated. This is illustrated in Figure 3.

Figure 3. Visualization of calculation of Haralick Parameters: input is a gray-scale image from which Figure 3. Visualization of calculation of Haralick Parameters: input is a gray-scale image from which the gray level co-occurrence matrix (GLCM) is calculated. Using mathematical matrix operations, the gray level co-occurrence matrix (GLCM) is calculated. Using mathematical matrix operations, diﬀerent parameters like contrast, correlation, energy and homogeneity can be calculated from the

GLCM.Figure Diagram 3. Visualization is modiﬁed of calculation according toof [Haralick25]. Parameters: input is a gray-scale image from which the gray level co-occurrence matrix (GLCM) is calculated. Using mathematical matrix operations,

Metals 2020, 10, x FOR PEER REVIEW 5 of 19

different parameters like contrast, correlation, energy and homogeneity can be calculated from the GLCM. Diagram is modified according to [25]. Metals 2020, 10, 630 5 of 19 While the original approach of Haralick’s analysis measures the pairing of gray values only for directions at angles of 0°, 45°, 90°, and 135°, the approach presented by Webel et al. [23] where the imageWhile is rotated the original in 1° steps approach from 0°of to Haralick’s180° and the analysis GLCM measures is measured the for pairing each ofrotation, gray values is used only in this for work.directions From at these angles rotations of 0◦, 45 the◦, 90 mean◦, and value 135◦ ,as the we approachll as the presentedamplitude, by defined Webel as et maximum al. [23] where minus the minimum,image is rotated of the in 1textural◦ steps fromparameters 0◦ to 180 are◦ and calculated. the GLCM This is measuredis done forusing each MATLAB rotation, is(R2019b, used in MathWorks,this work. From Natick, these MA, rotations USA) and the meanthe built-in value func as welltions as for the Haralick amplitude, Parameters. defined as Here, maximum the Haralick minus parametersminimum, ofcontrast, the textural correlation, parameters energy are and calculated. homogeneity This is are done used. using By MATLABusing averaged (R2019b, parameters MathWorks, of theNatick, rotation, MA, these USA) parameters and the built-in become functions rotation for invariant. Haralick For Parameters. more detailed Here, thedescriptions, Haralick parametersthe author refercontrast, to [21] correlation, and [23]. energy and homogeneity are used. By using averaged parameters of the rotation, these parameters become rotation invariant. For more detailed descriptions, the author refer to [21,23]. 2.5. Calculation of Local Binary Pattern 2.5. Calculation of Local Binary Pattern Local Binary Pattern (LBP) is a texture descriptor, originally made by Ojala et al. [22]. LBP Local Binary Pattern (LBP) is a texture descriptor, originally made by Ojala et al. [22]. LBP features features encode the neighboring context of each pixels into a histogram of the whole image which is usedencode as thefinal neighboring feature descriptor. context The of each process pixels of intocalcul aating histogram LBP features of the whole is visualized image whichin Figure is used 4. For as furtherfinal feature details descriptor. the authors The refer process to the of original calculating publication LBP features by Ojala is visualized et al. [22]. in FigureLBP features4. For furthercan be calculateddetails the by authors using refer different to the parameters original publication for the neighborhood, by Ojala et al. [that’s22]. LBP the features number can of beneighboring calculated pixelsby using (N) diandfferent the distance parameters of the for neighboring the neighborhood, pixels (R) that’s that the are number considered. of neighboring By considering pixels a circular (N) and neighborhoodthe distance of instead the neighboring of a square, pixels LBP (R)features that are become considered. rotation By invariant. considering The amain circular advantage neighborhood of LBP isinstead that it ofcan a square,easily encode LBP features fine details become in the rotation structure. invariant. However, The mainas the advantage encoding ofhappens LBP is in that a small it can scaleeasily it encodecannot finecapture details large in textures. the structure. In [26], However, multi-scale as the LBP encoding were introduced happens to in address a small scalethis problem. it cannot Multi-scalecapture large LBP textures. are created In [26], by multi-scale simply LBPconcatenating were introduced LBP features to address with this different problem. neighboring Multi-scale parametersLBP are created N and by R. simply In this concatenating work, uniform, LBP rotati featureson invariant with di LBPfferent with neighboring histogram parametersnormalization N andare used.R. In thisCalculation work, uniform, was done rotation in MATLAB invariant (R2019b, LBP withMathWorks, histogram Natick, normalization MA, USA) are with used. an open-source Calculation codewas donefor LBP. in MATLABNeighboring (R2019b, parameters MathWorks, of R = 1 Natick, and N MA,= 8 (1/8), USA) 2.4/8, with 4.2/16, an open-source and 6.2/16 codeare used for LBP.for single-scaleNeighboring LBP. parameters Additionally, of R = all1 andthose N four= 8 (1scales/8), 2.4 are/8, combined 4.2/16, and into 6.2 a/16 multi-scale are used forLBP. single-scale LBP. Additionally, all those four scales are combined into a multi-scale LBP.

Figure 4. Visualization of calculation of Local Binary Pattern (LBP): An example region of the original Figureimage is4. examined,Visualization with of neighboring calculation of parameters Local Binary of R Pattern= 1 and (LBP): N = 8. An Neighboring example region pixels of are the compared original imageto the centeris examined, pixel: pixelwith valuesneighboring smaller parameters than the center of R pixel = 1 valuesand N are= 8. assigned Neighboring to 1, pixel pixels values are comparedbigger to 0. to Binary the center values pixel: are pixel stringed values together. smaller This than allows the center a calculation pixel values of a decimal are assigned value whichto 1, pixel will valuesbe stored bigger in matrix to 0. Binary with the values same are width stringed and height together. as the This original allows image a calculation and in the of samea decimal place value as the whichinput centerwill be pixel. stored This in matrix is done with for everythe same pixel width of the and image. height The as LBPthe original matrix canimage be representedand in the same as a placehistogram as the which input will center be treated pixel. asThis the is feature done for vector every of thepixel original of the image. image. The LBP matrix can be represented as a histogram which will be treated as the feature vector of the original image. 2.6. Classification Process Using Support Vector Machine For classification, the workflow presented by Gola et al. [12,27], using a support vector machine (SVM) was adopted. An SVM classifies data by finding the best hyperplane that separates the data Metals 2020, 10, x FOR PEER REVIEW 6 of 19

2.6. Classification Process Using Support Vector Machine For classification, the workflow presented by Gola et al. [12,27], using a support vector machine (SVM)Metals 2020was, 10adopted., 630 An SVM classifies data by finding the best hyperplane that separates the data6 of 19 points of one class from the data points of another class. The implementation was done using MATLAB (R2019b, MathWorks, Natick, MA, USA) classification learner app, which allows automated points of one class from the data points of another class. The implementation was done using MATLAB classifier training of different SVM in order to find its best kernel and parameter settings. To estimate (R2019b, MathWorks, Natick, MA, USA) classification learner app, which allows automated classifier the predictive accuracy 5-fold cross-validation was used. Following parameters of the SVM, that are training of different SVM in order to find its best kernel and parameter settings. To estimate the available in MATLAB classification learner app, that were kept constant are a box constraint level of 1, predictive accuracy 5-fold cross-validation was used. Following parameters of the SVM, that are auto kernel scale mode, one-vs-one multiclass method and data standardization. available in MATLAB classification learner app, that were kept constant are a box constraint level of 1, auto kernel scale mode, one-vs-one multiclass method and data standardization. 3. Results 3. Results 3.1. Microstructures 3.1.Figure Microstructures 5 shows a continuous cooling diagram where regions for the specific microstructures are marked.Figure Table5 shows 2 lists athe continuous microstructure cooling constituen diagramts where found regions in each for sample. the specific Corresponding microstructures images, are beforemarked. cropping Table2 for lists further the microstructure analysis, are shown constituents in supplemental found in each Figures sample. S1–S4. Corresponding images, before cropping for further analysis, are shown in Supplemental Figures S1–S4.

Figure 5. Continuous cooling diagram showing diﬀerent cooling rates after austenitization at 1000 ◦C, Figurephase 5. transitions Continuous of ferrite,cooling pearlite,diagram bainite, showing and different martensite. cooling Additionally, rates after austenitization regions where diat ﬀ1000erent °C,microstructures phase transitions predominantly of ferrite, formedpearlite, are bain shown.ite, and Also, martensite. measured hardnessAdditionally, values regions (HV1) forwhere each differentcooling ratemicrostructures are added. predominantly formed are shown. Also, measured hardness values (HV1) for each cooling rate are added. Table 2. List of microstructures found in the produced samples.

Sample/Cooling RateTable 2. List Predominant of microstructures Microstructure found in the produced Other samples. Microstructure Constituents Sample/cooling2 K/s Pearlite Ferrite (polygonal, irregular) Predominant Microstructure Other Microstructure Constituents rate Rests of pearlite, beginning of 5 K/s Granular bainite 2 K/s Pearlite Ferritedegenerate (polygonal, upper irregular) bainite Rests of pearlite, beginning of degenerate 5 K/s8 K/s Granular Degenerate bainite upper bainite Upper bainite, rests of pearlite upper bainite Lath-like bainite: degenerate upper, 8 K/s10 K/s Degenerate upper bainite UpperFirst bainite, martensitic rests constituentsof pearlite upper, lower bainite Lath-like bainite: degenerate upper, upper, 10 K/s Lath-like bainite: degenerate upper, First martensitic constituents 12 K/s lower bainite Martensite upper, lower bainite Lath-like bainite: degenerate upper, upper, 12 K/s Lath-like bainite: degenerate upper, Martensite 15 K/s lower bainite Martensite upper, lower bainite Lath-like bainite: degenerate upper, upper, 15 K/s20 K/s Martensite Lath-like bainite:Martensite upper, lower bainite lower bainite 20 K/s278 K/s Martensite Fully martensitic Lath-like bainite: upper, - lower bainite 278 K/s Fully martensitic - The captured images from these samples were cropped to 350 350 px, equal to 9.70 9.70 µm, × × to get reference images with just one microstructure class. To label the bainitic structures present in the samples, the classiﬁcation scheme suggested by Zajac et al. [7] was used because it’s the most convenient Metals 2020, 10, x FOR PEER REVIEW 7 of 19

The captured images from these samples were cropped to 350 × 350 px, equal to 9.70 × 9.70 μm, to get reference images with just one microstructure class. To label the bainitic structures present in theMetals samples,2020, 10, 630the classification scheme suggested by Zajac et al. [7] was used because it’s the 7most of 19 convenient to use in common parlance and fits the best with the present bainitic structures. Labeling wasto use done in commonusing the parlance morphology and fitsand the general best withappearance the present in the bainitic SEM image, structures. based Labeling on experience was done and commonusing the morphologyknowledge andfrom general literature. appearance However, in the sometimes SEM image, higher based onresolution experience methods and common like transmissionknowledge from electron literature. microscopy However, (TEM) sometimes can be higher necessary resolution for methodsbainite identification like transmission [28]. electron Thus, focusingmicroscopy on(TEM) information can be necessaryfrom SEM for images bainite identificationcan yield some [28 ].uncertainty, Thus, focusing but on it informationis also done from in state-of-the-artSEM images can literature yield some like uncertainty, [10] and has but the it is advantage also done into state-of-the-artbe applicable in literature daily laboratory like [10] and work, has likethe advantagequality control, to be applicablewhere time-consuming in daily laboratory and expensive work, like techniques quality control, like whereTEM or time-consuming EBSD are not available. and expensive techniques like TEM or EBSD are not available. Six microstructure classes are found in the samples: pearlite, granular bainite, degenerated Six microstructure classes are found in the samples: pearlite, granular bainite, degenerated upper upper bainite, upper bainite, lower bainite, and martensite. Representative images for these six bainite, upper bainite, lower bainite, and martensite. Representative images for these six classes are classes are shown in Figure 6. The pearlitic structures are not always “textbook-like” as the cementite shown in Figure6. The pearlitic structures are not always “textbook-like” as the cementite lamellae lamellae are not consistently parallel or continuous and the gaps between lamellae are sometimes are not consistently parallel or continuous and the gaps between lamellae are sometimes not very not very pronounced. This can be attributed to the cooling that is faster than the usual cooling for pronounced. This can be attributed to the cooling that is faster than the usual cooling for ferrite-pearlite ferrite-pearlite microstructures. Granular bainite consists of irregular shaped ferrite grains with microstructures. Granular bainite consists of irregular shaped ferrite grains with islands of MAs as islands of MAs as the carbon rich phase in between. Some debris of cementite can also be found. the carbon rich phase in between. Some debris of cementite can also be found. Degenerated upper Degenerated upper bainite is composed of ferrite laths with predominantly MA constituents on the bainite is composed of ferrite laths with predominantly MA constituents on the lath boundaries. lath boundaries. Partially, there are also some cementite aggregates on the lath boundaries. Upper Partially, there are also some cementite aggregates on the lath boundaries. Upper bainite has cementite bainite has cementite precipitates on the ferrite lath boundaries, with cementite in clearly elongated precipitates on the ferrite lath boundaries, with cementite in clearly elongated form in contrast to form in contrast to the cementite aggregates found in the degenerated upper bainite. Lower bainite the cementite aggregates found in the degenerated upper bainite. Lower bainite exhibits intra-lath exhibits intra-lath cementite precipitation. These reference pictures can be used for further analysis, cementite precipitation. These reference pictures can be used for further analysis, i.e., extraction of i.e., extraction of textural features and machine learning classification. There are 15 images for class textural features and machine learning classification. There are 15 images for class pearlite, 24 for pearlite, 24 for granular bainite, 57 for degenerated upper bainite, 27 for upper bainite, 18 for lower granular bainite, 57 for degenerated upper bainite, 27 for upper bainite, 18 for lower bainite, and 18 bainite, and 18 for martensite. Pearlite and martensite were also included in the analysis because for martensite. Pearlite and martensite were also included in the analysis because those two phases those two phases can be present in granular bainite and because they present the upper and lower can be present in granular bainite and because they present the upper and lower boundaries of the boundaries of the bainitic microstructures. bainitic microstructures.

Figure 6. Representative scanning electron microscope (SEM) images of the six microstructure classes Figure 6. Representative scanning electron microscope (SEM) images of the six microstructure found in the samples, after cropping to 350 350 px (equal to 9.70 9.70 µm): (a) pearlite, (b) granular classes found in the samples, after cropping× to 350 × 350 px (equal× to 9.70 × 9.70 μm): (a) pearlite, (b) bainite, (c) degenerated upper bainite, (d) upper bainite, (e) lower bainite, and (f) martensite. Etching granular bainite, (c) degenerated upper bainite, d) upper bainite, e) lower bainite, and f) martensite. was done using 2% Nital. Etching was done using 2% Nital. 3.2. Microstructure Classification 3.2. Microstructure Classification All six microstructure classes found in the samples were considered for the classification task: pearlite (P), granular bainite (GB), degenerated upper bainite (DUB), upper bainite (UB), lower bainite (LB), and martensite (M). Classification was performed using only Haralick parameters, only local Metals 2020, 10, 630 8 of 19 binary pattern (single-scale and multi-scale), as well as a combination of both. Results are presented by confusion matrices (CM) in which overall accuracy and F1 score as well as class recalls and precisions and F1 scores for each class are shown. The accuracy is the ratio of correctly predicted examples by the total examples. Recall is defined as the ratio of true positives by the sum of true positives plus false negatives while precision is the ratio of true positives by the sum of true positives plus false positives. F1 score is a function of recall and precision which can be used if a balance between these two metric is wanted or when the data set is unbalanced which is the case in this study because of the dominance of degenerated upper bainite structures in the reference samples. F1 score is the product of precision and recall times two, divided by the sum of precision and recall. Accordingly, F1 score is calculated for each class. Overall F1 score is the mean value of F1 scores of each class.

3.3. Classification Using Haralick Parameters Table3 shows the CM for the classification using Haralick parameters. As there are some differences in class recalls and precisions, the F1 score is also added for each class. Accuracies are high for pearlite, granular and degenerated upper bainite and martensite, but lower for upper and lower bainite.

Table 3. Confusion matrix for the classiﬁcation using Haralick textural parameters with a quadratic kernel support vector machine (SVM).

Accuracy: 85.50% / F1 Score: 85.40% True True True True True True Class Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 15 0 0 0 0 0 100.00% pred. GB 0 21 1 0 2 0 87.50% pred. DUB 0 0 54 3 0 0 94.74% pred. UB 0 0 9 18 0 0 66.67% pred. LB 0 1 2 3 12 0 66.67% pred. Martensite 2 0 0 0 0 16 88.89% class recall 88.24% 95.45% 81.82% 75.00% 85.71% 100.00% - F1 score 93.75% 91.30% 87.80% 70.59% 75.00% 94.12% -

3.4. Classiﬁcation Using Local Binary Pattern

3.4.1. Single-Scale Local Binary Pattern Table4 shows the CMs for the four di fferent settings of the single-scale LBP. The 1/8 LBP does not yield particularly good classification results, especially for upper bainite there are several misclassifications. The overall accuracies of the other three single-scale LBP are good and comparable. However, precisions and recalls for the separate classes vary strongly.

3.4.2. Multi-Scale Local Binary Pattern By combining the four diﬀerent neighborhood parameters into one multi-scale LBP, the accuracy is signiﬁcantly improved to 88.70%. Table5 Accuracies for separate classes, assessed by the F1 score, are now high for all six microstructure classes.

3.5. Classification Using a Combination of Haralick Parameters and Local Binary Pattern As the Haralick classification gives a higher accuracy for granular bainite than the LBP multi-scale classification but is lower for other microstructure classes, both features are combined in the hope of improving the classification result. Table6 shows the confusion matrix. The accuracy is slightly improved to 90.60%. Metals 2020, 10, 630 9 of 19

Table 4. Confusion matrices for the classiﬁcation using single-scale local binary pattern: (a) radius R = 1, number of neighbors N = 8 (1/8), (b) 2.4/8, (c) 4.2/16, and (d) 6.2/16.

Accuracy: 74.20% / F1 score: 76.40% Accuracy: 79.90% / F1 score: 81.30% TRUE TRUE TRUE TRUE TRUE TRUE Class TRUE TRUE TRUE TRUE TRUE TRUE Class Pearlite GB DUB UB LB Martensite Precision Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 14 0 0 0 0 1 93.33% pred. Pearlite 13 0 1 0 1 0 86.67% pred. GB 0 17 4 3 0 0 70.83% pred. GB 0 17 5 2 0 0 70.83% pred. DUB 0 5 43 8 1 0 75.44% pred. DUB 0 2 49 5 0 1 85.96% pred. UB 0 1 7 16 3 0 59.26% pred. UB 0 4 4 19 0 0 70.37% pred. LB 0 1 3 1 13 0 72.22% pred. LB 0 0 3 2 13 0 72.22% pred. Martensite 0 2 0 0 1 15 83.33% pred. Martensite 0 0 2 0 0 16 88.89% Class recall 100.00% 65.38% 75.44% 57.14% 72.22% 93.75% Class recall 100.00% 73.91% 76.56% 67.86% 92.86% 94.12% F1 score 96.55% 68.00% 75.44% 58.18% 72.22% 88.24% F1 score 92.86% 72.34% 80.99% 69.09% 81.25% 91.43% (a) LBP 1-8 (b) LBP 2.4-8 Accuracy: 83.60% / F1 score: 84.70% Accuracy: 80.50% / F1 score: 79.50% TRUE TRUE TRUE TRUE TRUE TRUE Class Tue TRUE TRUE TRUE TRUE TRUE Class Pearlite GB DUB UB LB Martensite Precision Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 14 1 0 0 0 0 93.33% pred. Pearlite 11 1 0 0 0 3 73.33% pred. GB 0 17 7 0 0 0 70.83% pred. GB 0 16 7 1 0 0 66.67% pred. DUB 0 1 55 1 0 0 96.49% pred. DUB 0 3 52 1 0 1 91.23% pred. UB 0 0 8 19 0 0 70.37% pred. UB 0 3 4 20 0 0 74.07% pred. LB 0 0 2 4 12 0 66.67% pred. LB 0 2 1 3 12 0 66.67% pred. Martensite 0 0 2 0 0 16 88.89% pred. Martensite 1 0 0 0 0 17 94.44% Class recall 100.00% 89.47% 74.32% 79.17% 100.00% 100.00% Class recall 91.67% 64.00% 81.25% 80.00% 100.00% 80.95% F1 score 96.55% 79.07% 83.97% 74.51% 80.00% 94.12% F1 score 81.48% 65.31% 85.95% 76.92% 80.00% 87.18% (c) LBP 4.2-16 (d) LBP 6.2-16 Metals 2020, 10, 630 10 of 19

Table 5. Confusion matrix for the classiﬁcation using multi-scale local binary pattern with a quadratic kernel SVM.

Accuracy: 88.70%/F1 score: 89.10% True True True True True True Class Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 15 0 0 0 0 0 100.00% pred. GB 0 20 3 0 1 0 83.33% pred. DUB 0 2 54 0 0 1 94.74% pred. UB 0 1 5 21 0 0 77.78% pred. LB 0 1 2 1 14 0 77.78% pred. Martensite 1 0 0 0 0 17 94.44% class recall 93.75% 83.33% 84.38% 95.45% 93.33% 94.44% F1 score 96.77% 83.33% 89.26% 85.71% 84.85% 94.44%

Table 6. Confusion matrix for the classiﬁcation using a combination of Haralick parameters and multi-scale local binary pattern with a quadratic kernel SVM, without feature selection.

Accuracy: 90.60%/F1 Score: 90.70% True True True True True True Class Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 14 1 0 0 0 0 93.33% pred. GB 0 22 2 0 0 0 91.67% pred. DUB 0 0 55 2 0 0 96.49% pred. UB 0 1 4 22 0 0 81.48% pred. LB 0 0 2 2 14 0 77.78% pred. Martensite 1 0 0 0 0 17 94.44% class recall 93.33% 91.67% 87.30% 84.62% 100.00% 100.00% F1 score 93.33% 91.67% 91.67% 83.02% 87.50% 97.14%

Combining Haralick parameters and multiclass LBP gives a big set of features (64). For a better generalization, correlative features are removed. After doing this, only 31 features were kept. By this, the classiﬁcation result could even be slightly improved to 91.80% (Table7).

Table 7. Confusion matrix for the classiﬁcation using a combination of Haralick parameters and multi-scale local binary pattern with a quadratic kernel SVM, after removing correlative features.

Accuracy: 91.80%/F1 Score: 91.90% True True True True True True Class Pearlite GB DUB UB LB Martensite Precision pred. Pearlite 15 0 0 0 0 0 100.00% pred. GB 0 23 1 0 0 0 95.83% pred. DUB 0 1 55 1 0 0 96.49% pred. UB 0 1 4 21 1 0 77.78% pred. LB 0 0 1 2 15 0 83.33% pred. Martensite 1 0 0 0 0 17 94.44% class recall 93.75% 92.00% 90.16% 87.50% 93.75% 100.00% F1 score 96.77% 93.88% 93.22% 82.35% 88.24% 97.14%

4. Discussion The work of Azimi et al. [13] can be viewed as the state of the art in microstructure classification of low-carbon steels. The only drawback of their approach is that Deep Learning is inherently difficult to interpret. Gola et al. [12] worked with the same data set, but used morphological and textural parameters in a combination with a support vector machine for classification. This approach can be Metals 2020, 10, x FOR PEER REVIEW 11 of 19

4.1. Classification Results Table 8 summarizes the classification results of all the used features. With the confusion matrices showed earlier, this clearly shows that features calculated from the image texture can be Metals 2020, 10, 630 11 of 19 used to successfully distinguish complex bainitic microstructures. Best results, an excellent classification accuracy of 91.80%, were obtained by combining Haralick parameters and a bettermulti-scale linked LBP, to thefollowed metallurgical by a feature processes selection. and materialPicture IDs properties. allow to Golatrace etback al. from showed classification that both morphologicalresult to textural parameters parameters of the to investigatedthe original objectsmicrostructure and morphological images, which and textural permits parameters to asses which of the substructuremicrostructure of images these objects where are correc importanttly or wrongly for their classified. classification, Figure asall 7 shows three parameter examples groupsfor correctly were representedclassified images almost for equally all six microstructure after feature selection. classes. In continuing this approach and expanding it to bainitic structures the focus is now on using textural parameters as they showed to be very promising. Table 8. Overview of classification results for the different features that were used to classify. 4.1. Classification Results Haralick + LBP Haralick + Table8 summarizesLBP the classificationLBP LBP results ofLBP all the usedLBP features. With the confusionMulti-Scale matrices Features Haralick LBP showed earlier, this clearly1/8 shows2.4/8 that4.2/16 features 6.2/16 calculated Multi-Scale from the image texture canwith be Feature used to Multi-Scale successfully distinguish complex bainitic microstructures. Best results, an excellent classificationSelection Accuracyaccuracy of 91.80%,85.50% were74.20% obtained 79.90% by combining83.60% 80.50% Haralick parameters 88.70% and a 90.60% multi-scale LBP, 91.80% followed byF1 ascore feature 85.40% selection. 76.40% Picture IDs 81.30% allow 84. to70% trace back79.50% from classification 89.10% result 90.70% to textural parameters 91.90% to the original microstructure images, which permits to asses which microstructure images where correctlyThe ormain wrongly application classified. option Figure for 7this shows suggested examples classification for correctly approach classified based images on forreference all six microstructuresamples would classes. be the training of a model which could then be used as a pre-classification respectively labeling for other classification tasks. For a task of classifying complex steel Table 8. Overview of classification results for the different features that were used to classify. microstructures, i.e., bainite, the assignment of the ground truth can be quite objective because bainitic phase constituents are often small or inhomogeneous, and there is oftenHaralick no consensus+ LBP Haralick + LBP LBP LBP LBP LBP Multi-Scale amongFeatures human Haralick experts in labeling and classifying bainitic structures. This makesLBP it difficult to extract 1/8 2.4/8 4.2/16 6.2/16 Multi-Scale with Feature Multi-Scale ground truth parameters from these regions in an objective and statistically securedSelection way. By incorporating these reference microstructures, the ground-truth assignment will be much more Accuracy 85.50% 74.20% 79.90% 83.60% 80.50% 88.70% 90.60% 91.80% objective.F1 score 85.40% 76.40% 81.30% 84.70% 79.50% 89.10% 90.70% 91.90%

Figure 7. Examples of correctly classified images for each microstructure class. Figure 7. Examples of correctly classified images for each microstructure class. The main application option for this suggested classification approach based on reference samples wouldConsidering be the training the ofamount a model of which analyzed could thenpictures be used and asthe a pre-classification variety of structures respectively even labelingin one formicrostructure other classification class which tasks. cause For a taska broad of classifying distribution complex of values, steel microstructures,it is difficult to i.e.,draw bainite, definite the assignmentconclusions ofabout the how ground the truthmicrostructures can be quite correlate objective with because the calculated bainitic phasetextural constituents parameters are and often the smallclassification or inhomogeneous, result. Actually, and this there is isone often reason no consensuswhy machine among learning human methods experts are in needed labeling to and be classifyingable to classify bainitic these structures. microstruc Thistures. makes But it still, diffi cultsome to indications extract ground can truthbe found parameters when comparing from these regions in an objective and statistically secured way. By incorporating these reference microstructures, the ground-truth assignment will be much more objective. Metals 2020, 10, x FOR PEER REVIEW 12 of 19 textural parameters and microstructures. Figure 8 shows four Haralick parameters, i.e., the mean values of contrast, correlation and energy as well as the amplitude of correlation. Black dots represent the single values of every single image, red dots assign the mean value of all images. Contrast is a measure of the local variations in an image [21]. Correlation is a measure of how correlated a pixel is to its neighbor over the whole image, i.e., the joint probability occurrence of the specified pixel pairs [29]. Energy measures the uniformity of the gray level distribution in an image [30]. Few entries in the GLCM that have high probability, lead to a high energy value. Homogeneity measures the closeness of the distribution of elements in the GLCM to the GLCM diagonal [29], i.e., only entries close to the diagonal have a high impact on the value of the homogeneity. Looking at the mean value of contrast, values for pearlite and martensite are significantly higher than for the bainitic structures, so that even when considering the scattering of the data, the bainitic classes could almost be separated from pearlite and martensite by setting a threshold. Low contrast values mean fewer local variations in the image. For the bainitic structures, there is always the dark bainitic ferrite as a “background”. This dark background, which has only few local variations, represents a considerable part of the image, explaining the lower overall contrast values for the bainitic structures. In a similar way, the higher correlation values for bainitic structures can beMetals explained.2020, 10, 630 Because of the “background” more “dark pixel pairs” occur, leading to higher12 of 19 correlation values. Pearlite tends to have higher contrast and correlation values compared to martensite.Considering This could the amountbe explaine ofd analyzed by the ferrite-cementite pictures and the transitions variety ofthat structures occur in the even pearlitic in one microstructuremicrostructure because class which of the cause topography a broad contrast distribution, caused of by values, using Everhart-Thornley it is difficult to draw Detector definite in theconclusions SEM. Regarding about how bainitic the microstructures structures, not much correlate tendencies with the can calculated be seen texturalbecause parameters the scattering and can the overshadowclassification the result. trend. Actually, However, this for is onethe reasonamplitude why value machine of correlation, learning methods granular are and needed lower to bainite be able tendto classify to have these lower microstructures. values than upper But still,and degenera some indicationsted upper can bainite. be found In general, when comparing amplitude texturalvalues willparameters be higher and for microstructures.structures with certain Figure preferenti8 shows foural directions Haralick and parameters, lower for i.e.,statistically the mean distributed values of structures.contrast, correlation While upper and energyand degenerated as well as the upper amplitude bainite of have correlation. a pronounced Black dots lath represent structure the of single the carbon-richvalues of every second single phase, image, the reddistribution dots assign of this the second mean value phase of is all more images. statistically Contrast distributed is a measure in granularof the local and variations lower bainite, in an causing image [lower21]. Correlation amplitude isvalue a measure for correlation. of how correlatedLower bainite a pixel still is has to itsa preferentialneighbor over direction the whole (60° arrangem image, i.e.,ent the of jointintra-lath probability precipitates), occurrence however of the it specifiedis less pronounced pixel pairs than [29]. inEnergy upper measures and degenerated the uniformity upper ofbainite. the gray Taking level distributionthis into account, in an imagepearlite [30 should]. Few show entries higher in the amplitudeGLCM that values have highthan probability,martensite. leadHowever, to a high their energy values value. partly Homogeneity overlap. The measuresreason for the this closeness is that notof theall distributionpearlitic pictures of elements have straight in the GLCMcontinuous to the cementite GLCM diagonal laths but [29 are], i.e., already only entriesa bit degenerated close to the withdiagonal less preferential have a high orientation. impact on the value of the homogeneity.

Metals 2020, 10, x FOR PEER REVIEW 13 of 19 (a) (b)

FigureFigure 8. 8. HaralickHaralick Parameters: Parameters: (a ()a )contrast contrast mean mean value, value, (b (b) )correlation correlation mean mean value, value, (c (c) )correlation correlation amplitude,amplitude, and and (d (d) )homogeneity homogeneity mean mean value. value.

LocalLooking binary at the pattern mean valueare good of contrast, at capturing values small for pearlite and fine and details martensite of images are signiﬁcantly [31], e.g., edges, higher corners,than for spots, the bainitic etc. One structures, disadvantage so that is that even they when can considering have problems the scattering to distinguish of the textures data, the that bainitic have theclasses same could small almost structures be separated but differ from in their pearlite large and structures. martensite LBPs by setting used in a threshold.this work Loware rotation contrast invariantvalues mean uniform fewer LBP. local By using variations uniform in the LBP, image. the length For of the the bainitic histogram, structures, i.e., the there feature is alwaysvector can the bedark reduced bainitic and ferrite the performance as a “background”. of classifiers This usin darkg background,these LBP features which can has be only improved few local [22,32]. variations, representsIn a try a considerableto correlate partLBP offeatures, the image, the explainingmicrostruc thetures lower and overall the classification contrast values results, for the Figure bainitic 9 showsstructures. the LBP In a1/8 similar for the way, six microstructure the higher correlation images from values Figure for bainitic 6, which structures are representative can be explained. for the sixBecause classes of and the “background”show clear differen moreces. “dark Bainitic pixel pairs”structures occur, show leading higher to highervalues correlationfor bin 0, which values. representsPearlite tends bright to havespots higher [22], than contrast pearlite and correlationand martensite, values correctly compared capturing to martensite. the arrangement This could of be brightexplained carbon-rich by the ferrite-cementite second phase in transitions a dark bainitic that occur ferrite in the background pearlitic microstructure of the bainitic because structures, of the compared to the more uniform gray level distribution of pearlite and martensite. Bins 1–7, which correspond to different edges or corners of varying positive and negative curvature [22], partly show differences which is plausible, as the images also exhibit different kind of edges. However, several bins show only marginal differences, especially for bainitic structures, already indicating that this LBP could not provide enough discrimination for the classification of bainitic structures. Figure 10 shows the averaged histograms of the local binary pattern with R = 1 and N = 8 of all images for all six microstructure classes, which reached an accuracy of just 74.20%. For an easier visualization, histograms are averaged by calculating the mean value of the separate bins for every analyzed image. This allows for some indications how the LBPs, the microstructures and the classification results correlate. Pearlite and martensite show quite different histograms while the histograms of the four bainitic structures are similar. For a better illustration and a closer look at differences, some histogram bins (bins 1, 5, 6, and 9) are separately shown in Figure 11a. Error bars for the standard deviation are added to indicate the scattering of all the images. Pearlite and martensite can be distinguished quite well, also when considering the scattering. Contrary, values for bainite are comparable most of the time, giving a hint that a differentiation of bainitic structures with these features will be difficult. Looking at the confusion matrix for this LBP in Table 4, there are strong variations between the recalls and precisions of the individual classes. Indeed, F1 scores for pearlite and martensite are high and lower for bainitic structures which fits the differences that are indicated by the histograms. Also, for the R = 2.4 and N = 8 as well as the R = 4.2 and N = 16 LBP, the results for pearlite and martensite are good and clearly better than the results for the bainitic structures. When looking at the representative SEM images for all classes, pearlite and martensite are the “densest” structures compared to bainitic structures, which have more “background” (the dark bainitic ferrite). So basically, the representative area to capture to relevant features of a microstructure is smaller for pearlite and martensite and bigger for the bainitic structures. That’s why these three single-scale LBP achieve better results for pearlite and martensite.

Metals 2020, 10, 630 13 of 19 topography contrast, caused by using Everhart-Thornley Detector in the SEM. Regarding bainitic structures, not much tendencies can be seen because the scattering can overshadow the trend. However, for the amplitude value of correlation, granular and lower bainite tend to have lower values than upper and degenerated upper bainite. In general, amplitude values will be higher for structures with certain preferential directions and lower for statistically distributed structures. While upper and degenerated upper bainite have a pronounced lath structure of the carbon-rich second phase, the distribution of this second phase is more statistically distributed in granular and lower bainite, causing lower amplitude value for correlation. Lower bainite still has a preferential direction (60◦ arrangement of intra-lath precipitates), however it is less pronounced than in upper and degenerated upper bainite. Taking this into account, pearlite should show higher amplitude values than martensite. However, their values partly overlap. The reason for this is that not all pearlitic pictures have straight continuous cementite laths but are already a bit degenerated with less preferential orientation. Local binary pattern are good at capturing small and fine details of images [31], e.g., edges, corners, spots, etc. One disadvantage is that they can have problems to distinguish textures that have the same small structures but differ in their large structures. LBPs used in this work are rotation invariant uniform LBP. By using uniform LBP, the length of the histogram, i.e., the feature vector can be reduced and the performance of classifiers using these LBP features can be improved [22,32]. In a try to correlate LBP features, the microstructures and the classification results, Figure9 shows the LBP 1/8 for the six microstructure images from Figure6, which are representative for the six classes and show clear differences. Bainitic structures show higher values for bin 0, which represents bright spots [22], than pearlite and martensite, correctly capturing the arrangement of bright carbon-rich second phase in a dark bainitic ferrite background of the bainitic structures, compared to the more uniform gray level distribution of pearlite and martensite. Bins 1–7, which correspond to different edges or corners of varying positive and negative curvature [22], partly show differences which is plausible, as the images also exhibit different kind of edges. However, several bins show only marginal differences, especially for bainitic structures, already indicating that this LBP could not provide enough discrimination for the classification of bainitic structures. Figure 10 shows the averaged histograms of the local binary pattern with R = 1 and N = 8 of all images for all six microstructure classes, which reached an accuracy of just 74.20%. For an easier visualization, histograms are averaged by calculating the mean value of the separate bins for every analyzed image. This allows for some indications how the LBPs, the microstructures and the classification results correlate. Pearlite and martensite show quite different histograms while the histograms of the four bainitic structures are similar. For a better illustration and a closer look at differences, some histogram bins (bins 1, 5, 6, and 9) are separately shown in Figure 11a. Error bars for the standard deviation are added to indicate the scattering of all the images. Pearlite and martensite can be distinguished quite well, also when considering the scattering. Contrary, values for bainite are comparable most of the time, giving a hint that a differentiation of bainitic structures with these features will be difficult. Looking at the confusion matrix for this LBP in Table4, there are strong variations between the recalls and precisions of the individual classes. Indeed, F1 scores for pearlite and martensite are high and lower for bainitic structures which fits the differences that are indicated by the histograms. Also, for the R = 2.4 and N = 8 as well as the R = 4.2 and N = 16 LBP, the results for pearlite and martensite are good and clearly better than the results for the bainitic structures. When looking at the representative SEM images for all classes, pearlite and martensite are the “densest” structures compared to bainitic structures, which have more “background” (the dark bainitic ferrite). So basically, the representative area to capture to relevant features of a microstructure is smaller for pearlite and martensite and bigger for the bainitic structures. That’s why these three single-scale LBP achieve better results for pearlite and martensite. Metals 2020, 10, x FOR PEER REVIEW 14 of 19

MetalsMetals2020 2020, ,10 10,, 630 x FOR PEER REVIEW 1414 of of 19 19

Figure 9. LBP 1/8 for representative images from Figure 5 for all six microstructure classes, showing Figure 9. LBP 1/8 for representative images from Figure5 for all six microstructure classes, showing differences in the separate bins of the histogram. diFigureﬀerences 9. LBP in the 1/8 separate for representative bins of the images histogram. from Figure 5 for all six microstructure classes, showing differences in the separate bins of the histogram.

MetalsFigure 2020, 10 10., xAveragedhistograms FOR PEER REVIEW of LBP with R = 1 and N = 8 of all images for all six microstructure classes.15 of 19 Figure 10. Averaged histograms of LBP with R = 1 and N = 8 of all images for all six microstructure Averaging performed by calculating the mean value of the separate bins for every analyzed image. classes. Averaging performed by calculating the mean value of the separate bins for every analyzed Figure 10. Averaged histograms of LBP with R = 1 and N = 8 of all images for all six microstructure image. classes. Averaging performed by calculating the mean value of the separate bins for every analyzed Theimage. R = 4.2 and N = 16 LBP gave the best results for the single-scale LBP with 83.60% accuracy. Figure 12 shows the averaged histograms for all six microstructure classes. Again, for an easier The R = 4.2 and N = 16 LBP gave the best results for the single-scale LBP with 83.60% accuracy. visualization, histograms are averaged by calculating the mean value of the separate bins for every Figure 12 shows the averaged histograms for all six microstructure classes. Again, for an easier analyzed image. Also, some histogram bins (bins 1, 7, 9, and 15) are separately shown in Figure 11b visualization, histograms are averaged by calculating the mean value of the separate bins for every for a closer look at differences. Although there is always overlap because of the scattering, a analyzed image. Also, some histogram bins (bins 1, 7, 9, and 15) are separately shown in Figure 11b tendency of clearer differences in the bainitic classes compared to the LBP 1/8 can be recognized, for a closer look at differences. Although there is always overlap because of the scattering, a explaining the better classification result with this LBP. tendency of clearer differences in the bainitic classes compared to the LBP 1/8 can be recognized, explaining the better classification result with this LBP.

Figure 11. Individual bins of the LBP 1/8 (a) and LBP4.2/16 (b) histograms for all six microstructure classes,Figure for11. Individual a better illustration bins of the of LBP diﬀ erences.1/8 (a) and Error LBP4.2/16 bars are (b) standardhistograms deviations for all six to microstructure indicate the scatteringclasses, for of alla better analyzed illustration images. of differences. Error bars are standard deviations to indicate the scattering of all analyzed images. The R = 4.2 and N = 16 LBP gave the best results for the single-scale LBP with 83.60% accuracy. Figure 12 shows the averaged histograms for all six microstructure classes. Again, for an easier visualization, histograms are averaged by calculating the mean value of the separate bins for every analyzed image. Also, some histogram bins (bins 1, 7, 9, and 15) are separately shown in Figure 11b for a closer look at diﬀerences. Although there is always overlap because of the scattering, a tendency of

Figure 12. Averaged histograms of LBP with R = 4.2 and N = 16 of all images for all six microstructure classes. Averaging performed by calculating the mean value of the separate bins for every analyzed image. Bins 0–16 are also shown enlarged.

Looking at all four single-scale LBP, the overall accuracies are mediocre to good (maximum accuracy of 83.60% for LBP 4.2/16) and there are usually strong variations between the recalls and precisions of the individual classes. This clearly shows that by considering only a single scale for the LBP, not all relevant features of the six different microstructures can be captured. By combining the four scales from the single-scale LBP into one multi-scale LBP the overall accuracy improves to a very good 88.70%. Accuracies for separate classes, assessed by the F1 score, are high for all classes. As LBP features capture small and fine details of an image and Haralick parameters recognize image features on a bit bigger scale, it seems reasonable to try to combine Haralick and LBP features. Furthermore, Haralick classification gives a higher accuracy for granular bainite than the LBP multi-scale classification, but is lower for other microstructure classes, so they could perhaps complement one another. Indeed, by combining those features, the overall classification accuracy gets improved to 90.60%. Because the combination of these parameters gives a big set of features (64), correlative features are removed for a better generalization. Only 31 features were kept, and by this, the classification result could again be slightly improved to 91.80%. Figure 13 gives some examples of misclassified microstructure images. With Picture IDs, textural parameters and the original microstructure images can be traced back from the classification

Metals 2020, 10, x FOR PEER REVIEW 15 of 19

Metals 2020, 10, 630 15 of 19 Figure 11. Individual bins of the LBP 1/8 (a) and LBP4.2/16 (b) histograms for all six microstructure clearerclasses, diﬀerences for a better in the illustration bainitic classes of differences. compared Erro tor thebars LBP are 1standard/8 can be deviations recognized, to indicate explaining the the scattering of all analyzed images. better classiﬁcation result with this LBP.

Figure 12. Averaged histograms of LBP with R = 4.2 and N = 16 of all images for all six microstructure Figure 12. Averaged histograms of LBP with R = 4.2 and N = 16 of all images for all six microstructure classes. Averaging performed by calculating the mean value of the separate bins for every analyzed classes. Averaging performed by calculating the mean value of the separate bins for every analyzed image. Bins 0–16 are also shown enlarged. image. Bins 0–16 are also shown enlarged. Looking at all four single-scale LBP, the overall accuracies are mediocre to good (maximum accuracyLooking of 83.60% at all forfour LBP single-scale 4.2/16) and LBP, there the are over usuallyall accuracies strong variations are mediocre between to good the recalls(maximum and precisionsaccuracy of of 83.60% the individual for LBP classes. 4.2/16) Thisand there clearly are shows usually that strong by considering variations only between a single the scale recalls for and the LBP,precisions not all of relevant the individual features classes. of the sixThis di clearlyfferent microstructuresshows that by considering can be captured. only a single By combining scale for thethe fourLBP, scales not all from relevant the single-scale features of LBPthe six into different one multi-scale microstructures LBP the overallcan be accuracycaptured. improves By combining to a very the goodfour scales 88.70%. from Accuracies the single-scale for separate LBP classes, into one assessed multi-scale by the LBP F1 score,the overall are high accuracy for all classes.improves to a veryAs good LBP 88.70%. features Accuracies capture small for separate and fine classes, details assessed of an image by the and F1 Haralick score, are parameters high for all recognize classes. imageAs features LBP features on a bit capture bigger scale,small itand seems fine reasonabledetails of an to image try to combineand Haralick Haralick parameters and LBP recognize features. Furthermore,image features Haralick on a bit classification bigger scale, gives it seems a higher reasonable accuracy to for try granular to combine bainite Haralick than the and LBP LBP multi-scale features. classification,Furthermore, butHaralick is lower classification for other microstructure gives a higher classes, accuracy so they for couldgranular perhaps bainite complement than the oneLBP another.multi-scale Indeed, classification, by combining but is those lower features, for other the overallmicrostructure classification classes, accuracy so they gets could improved perhaps to 90.60%.complement Because one the another. combination Indeed, of theseby combining parameters thos givese features, a big set ofthe features overall (64), classification correlative accuracy features aregets removed improved for to a 90.60%. better generalization. Because the combination Only 31 features of these were parameters kept, and give by this,s a big the set classification of features result(64), correlative could again features be slightly are removed improved for to a 91.80%. better generalization. Only 31 features were kept, and by this, Figurethe classification 13 gives some result examples could again of misclassified be slightly microstructureimproved to 91.80%. images. With Picture IDs, textural parametersFigure and13 thegives original some microstructureexamples of misclassified images can be microstructure traced back from images. the classification With Picture results. IDs, Thistextural allows parameters to check and which the feature original in microstructure the microstructure images image can could be trac haveed back caused from a misclassification.the classification Figure 13a shows a martensitic microstructure that was classified as pearlite. Comparing it with a correctly classified martensitic image (Figure 13b) the structure of the misclassified martensite is more ordered and less chaotic and thereby looking similar to a pearlitic microstructure, as shown in Figure 13c. In Figure 13d a lower bainite image that was misclassified as upper bainite is shown. The reason is probably that the lower bainite also exhibits some cementite precipitation on the lath boundaries in addition to the intra-lath cementite precipitation. Figure 13e shows upper bainite that was wrongly classified as degenerated upper bainite. This could be explained by the fact that not all precipitation on the lath boundaries are straight and slender and that there are also some cementite aggregates giving it a bit of a degenerated structure. These examples show the difficulty in bainite classification as one bainite subclass can also exhibit some features that are more associated with other subclasses, making the ground truth assignment challenging even though reference samples were used in this work. In principle, such images could be left out for the classification which would improve the Metals 2020, 10, x FOR PEER REVIEW 16 of 19 results. This allows to check which feature in the microstructure image could have caused a misclassification. Figure 13a shows a martensitic microstructure that was classified as pearlite. Comparing it with a correctly classified martensitic image (Figure 13b) the structure of the misclassified martensite is more ordered and less chaotic and thereby looking similar to a pearlitic microstructure, as shown in Figure 13c. In Figure 13d a lower bainite image that was misclassified as upper bainite is shown. The reason is probably that the lower bainite also exhibits some cementite precipitation on the lath boundaries in addition to the intra-lath cementite precipitation. Figure 13e shows upper bainite that was wrongly classified as degenerated upper bainite. This could be explained by the fact that not all precipitation on the lath boundaries are straight and slender and that there are also some cementite aggregates giving it a bit of a degenerated structure. These

Metalsexamples2020, 10show, 630 the difficulty in bainite classification as one bainite subclass can also exhibit16 some of 19 features that are more associated with other subclasses, making the ground truth assignment challenging even though reference samples were used in this work. In principle, such images could classiﬁcationbe left out for result. the classification However, itwhich is a goal would to cover improv alle ranges the classification of bainitic microstructures result. However, and it is because a goal multi-phaseto cover all ranges steels or of industrial bainitic microstructures samples will also and exhibit because mixed multi-phase structures steels it was or decided industrial to keep samples these imageswill also in exhibit the data mixed set. structures it was decided to keep these images in the data set.

Figure 13.13. ExamplesExamples of of misclassified misclassified pictures: pictures: (a) wrongly(a) wrongly martensite martensite showing showing a regular a lathregular structure lath comparedstructure compared to the more to irregularthe more lathsirregular of correctly laths of classified correctly martensite classified martensite (b) and similar (b) and to pearlite similar (cto), (pearlited) lower (c bainite), (d) lower with somebainite slender with some cementite slender precipitation cementite on precipitation the lath boundaries on the lath (red), boundaries similar to upper(red), bainite,similar andto upper (e) upper bainite, bainite and with (e some) upper cementite bainite aggregates, with some similar cementite to degenerated aggregates, upper similar bainite. to degenerated upper bainite. 4.2. Extensions to the Classification Approach 4.2. ExtensionsWhen looking to the atClassification the CM in Approach Table7, 9 of the 13 wrongly classified images come from mix-ups of lower,When upper, looking and at the degenerated CM in Table upper 7, 9 bainite.of the 13 Thesewrongly three classified bainite images classes come are thefrom most mix-ups similar of microstructureslower, upper, and because degenerated they all exhibitupper abainite. lath-like These structure. three Onebainit waye classes to improve are the the classificationmost similar resultmicrostructures could be to because combine they these all threeexhibit bainite a lath-like types structure. into one class: One lath-likeway to improve bainite. the After classification classifying pearlite,result could granular be to bainite,combine lath-like these three bainite, bainite and martensite,types into one morphological class: lath-like parameters bainite. After could classifying be used to furtherpearlite, separate granular the bainite, lath-like lath-like bainite bainite, into the and original martensite, lower, morphological upper, and degenerated parameters could upper be bainite used becauseto further all separate three types the lath-like have diff bainiteerent morphological into the original characteristics. lower, upper, Lower and degenerated bainite usually upper has bainite many smallbecause cementite all three precipitates types have different in the ferrite morphological laths, upper characteristics. bainite has longer Lower cementite bainite usually particles has whereas many thesmall carbon-rich cementite phaseprecipitates in degenerated in the ferrite upper laths, bainite upper is widerbainite than has thelonger cementite cementite of upper particles bainite. whereas the carbon-richThe images phase analyzed in degenerated in this work upper were squared bainite images,is wider cut than in the a uniform cementite way of from upper a bigger bainite. image. UsuallyThe regions images of analyzed a specific in microstructure this work were are squared not squared, images, but cut have in somea uniform arbitrary, way irregular from a bigger shape. Inimage. future Usually work, theregions suggested of a classificationspecific microstructu approachre could are not be appliedsquared, to realbut microstructurehave some arbitrary, objects insteadirregular of shape. squared In images. future work, the suggested classification approach could be applied to real microstructureThis work objects already instead covered of asquared wide variety images. of bainitic microstructures. However, for granular bainite,This more work variations already covered of the carbon-rich a wide variety second of phasebainitic exist. microstructures. In this work, thisHowever, carbon-rich for granular second phasebainite, was more mainly variations composed of the ofcarbon-rich MA particles second with phase some exist. debris In this of cementite. work, this Zajaccarbon-rich et al. [second7] also definephase was degenerated mainly composed pearlite or of incomplete MA particles transformation with some debris products of cementite. as carbon-rich Zajac second et al. [7] phases. also Indefine future degenerated work, these pearlite different or variations incomplete in transforma granular bainitetion products could also as be carbon-rich included in second the classification phases. In task.future Furthermore, work, these different self-tempered variatio martensitens in granular could bainite also becould added. also Thebe included production in the of classification isothermally transformedtask. Furthermore, samples self-tempered could also be martensite tried. If the could goal also would be beadded. a universal The production bainite classification of isothermally of all kindstransformed of bainitic samples steels, could more also reference be tried. samples If the couldgoal would be produced be a universal to build bainite a more classification robust model. of all kindsFrom of bainitic the machine steels, more learning reference perspective, samples the could SVM be could produced be further to build tuned a more and additionalrobust model. feature selection could be tried. Furthermore, by using data augmentation, especially for the microstructure classes that are less represented, in order to get better balanced classes, the model could perhaps be improved. Although using these reference samples gives more objectivity, there is still some subjectivity in assigning the classes for the ground truth. First, when using only SEM images for labeling some uncertainty will always remain. This could be improved by adding correlative data from TEM or EBSD analysis. However, the time required and the limited areas that can be measured restrict the practical use. Second, the choice of the classification scheme is already subjective. In this work, the scheme suggested by Zajac [7] was used, however there are many more schemes available in the literature, as discussed in the introduction. When choosing a different classification scheme for these reference samples, the classification result will also be different. To overcome this problem Metals 2020, 10, 630 17 of 19 and make the classification even more objective, future work will incorporate unsupervised learning techniques. Contrary to the supervised learning approach used in this work, no ground truth is given for unsupervised learning. These techniques can be used to cluster data or raw images by finding similarities. For data clustering, different algorithms like k-means, k-medoids, or hierarchical clustering are available. The raw images can also be clustered by using a pre-trained neural network as a feature extractor, followed by a clustering of these features with a clustering algorithm. Such an approach is presented by Kitahara et al. [33].

5. Conclusions Microstructures considered in the classification are pearlite, granular bainite, degenerated upper • bainite, upper bainite, lower bainite, and martensite, with bainite types according to the scheme suggested by Zajac [7]. An excellent classification accuracy of 91.80% is reached, showing the feasibility of using textural features to distinguish bainitic microstructures. The images used in this work come from specifically produced samples, by using a quenching • dilatometer, in order to get reference bainitic structures. This allows a more objective ground truth assignment for the classification which is, when dealing with complex microstructures like bainite, otherwise it is a challenging task with a big subjective component. In continuing this approach, these reference samples and the classification model based on them • can be used as a more objective ground truth assignment for other tasks of classifying complex steel microstructures, i.e., bainite, especially when analyzing multi-phase industrial steel grades. In future work, unsupervised learning techniques will be implemented to eliminate the subjective • component of choosing a classification scheme, which will make the ground truth assignment again more objective.

Supplementary Materials: The following are available online at http://www.mdpi.com/2075-4701/10/5/630/s1, Supplementary materials include representative SEM micrographs of each sample before cropping, to give an overview of the overall microstructures. Author Contributions: Conceptualization, M.M. and D.B.; Formal analysis, M.M. and L.U.; Investigation, M.M.; Methodology, M.M. and L.U.; Supervision, D.B., T.S., and F.M.; Visualization, M.M.; Writing—original draft, M.M.; Writing—review & editing, M.M., D.B., T.S., and F.M. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Acknowledgments: The authors thank steel manufacturer AG der Dillinger Hüttenwerke for providing the sample material and TA Instruments for the production of the reference samples by using DIL 805 A/D quenching dilatometer. Furthermore, the authors thank Johannes Webel who programmed a MATLAB script for automatic calculation of Haralick textural features. The authors acknowledge Saarland University within the funding program Open Access Publishing. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Bhadeshia, H.K.D.H. Bainite in Steels, 3rd ed.; Maney Publishing: Leeds, UK, 2015. 2. Ohmori, Y.; Ohtani, H.; Kunitake, T. The Bainite in Low Carbon Low Alloy High Strength Steels. Tetsu-to-Hagane 1971, 57, 1690–1705. [CrossRef] 3. Bramﬁtt, B.L.; Speer, J.G. A perspective on the morphology of bainite. Metall. Trans. A 1990, 21, 817–829. [CrossRef] 4. Lotter, U.; Hougardy, H.P. Kennzeichnung des Gefüges Bainit. Prakt. Metallogr. 1992, 29, 151–157. 5. Krauss, G.; Thompson, S.W. Ferritic Microstructures in Continuously Cooled Low- and Ultralow-carbon Steels. ISIJ Int. 1995, 35, 937–945. [CrossRef] 6. Fischer, M.D. Quantitative Analyse feinkörniger, komplexer und mehrphasiger Mikrostrukturen. Ph.D. Thesis, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany, 22 December 2015. Metals 2020, 10, 630 18 of 19

7. Zajac, S.; Schwinn, V.; Tacke, K.H. Characterisation and Quantification of Complex Bainitic Microstructures in High and Ultra-High Strength Linepipe Steels. Mater. Sci. Forum. 2005, 500–501, 387–394. [CrossRef] 8. Araki, T. Atlas for Bainitic Microstructures; ISIJ Bainite Committee: Tokyo, Japan, 1992. 9. Gerdemann, F.L.H.; Bleck, W. Bainite in medium carbon steels; Shaker Verlag GmbH: Düren, Germany, 2010. 10. Aarnts, M.P.;Rijkenberg, R.A.; Twisk, F.A.; Wilcox, D.; Zuijderwijk, M.J.; Arlazarov,A.; Barbier, D.; Germain, L.; Gouné, M.; Hazotte, A.; et al. Microstructural quantification of multi-phase steels (Micro-quant), European Commission. Res. Fund for Coal Steel 2011.[CrossRef] 11. Song, W. Characterization and Simulation of BAINITE Transformation in High Carbon Bearing Steel 100Cr6; RWTH Aachen Aachen: Aachen, Germany, 2014. 12. Gola, J.; Webel, J.; Britz, D.; Guitar, A.; Staudt, T.; Winter, M.; Mücklich, F. Objective microstructure classification by support vector machine (SVM) using a combination of morphological parameters and textural features for low carbon steels. Comput. Mater. Sci. 2019, 160, 186–196. [CrossRef] 13. Azimi, S.M.; Britz, D.; Engstler, M.; Fritz, M.; Mücklich, F. Advanced steel microstructural classification by deep learning methods. Sci. Rep. 2018, 8, 1–14. [CrossRef] 14. Zajac, S.; Komenda, J.; Morris, P.; Dierickx, P.; Matera, S.; Diaz, F.P. Quantitative Structure-Property Relationships for Complex Bainitic Microstructures. 2005. Available online: https://publications.europa.eu/ en/publication-detail/-/publication/22f902b8-37e3-4fa1-86fe-5876e4974329 (accessed on 29 April 2020). 15. Tsutsui, K.; Terasaki, H.; Maemura, T.; Hayashi, K.; Moriguchi, K.; Morito, S. Microstructural diagram for steel based on crystallography with machine learning. Comput. Mater. Sci. 2019, 159, 403–411. [CrossRef] 16. Komenda, J. Automatic recognition of complex microstructures using the Image Classifier. Mater. Charact. 2001, 46, 87–92. [CrossRef] 17. Miyama, E.; Voit, C.; Pohl, M. Zementitnachweis zur Unterscheidung von Bainitstufen in modernen, niedriglegierten Mehrphasenstählen. Prakt. Metallogr. 2011, 48, 261–272. [CrossRef] 18. Banerjee, S.; Datta, S.; Paul, B.; Saha, S.K. Segmentation of three phase micrograph: An automated approach. Proc. CUBE Int. Inf. Technol. Conf. ACM 2012, 1–4. [CrossRef] 19. Paul, A.; Gangopadhyay, A.; Chintha, A.R.; Mukherjee, D.P.; Das, P.; Kundu, S. Calculation of phase fraction in steel microstructure images using random forest classifier. IET Image Process. 2018, 12, 1370–1377. [CrossRef] 20. Shapiro, L. Computer Vision and Image Processing; Academic Press: San Diego, CA, USA, 1992. 21. Haralick, R.; Shanmugan, K.; Dinstein, I. Textural features for image classification. IEEE Trans. Syst. Man Cybern. 1973, 3, 610–621. [CrossRef] 22. Ojala, T.; Pietikäinen, M.; Mäenpää, T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 24, 971–987. [CrossRef] 23. Webel, J.; Gola, J.; Britz, D.; Mücklich, F. A new analysis approach based on Haralick texture features for the characterization of microstructure on the example of low-alloy steels. Mater. Charact. 2018, 144, 584–596. [CrossRef] 24. Gupta, S.; Sarkar, J.; Kundu, M.; Bandyopadhyay, N.R.; Ganguly, S. Automatic recognition of SEM microstructure and phases of steel using LBP and random decision forest operator. Measurement 2020, 151. [CrossRef] 25. Wibmer, A.; Hricak, H.; Gondo, T.; Matsumoto, K.; Veeraraghavan, H.; Fehr, D.; Zheng, J.; Goldman, D.; Moskowitz, C.; Fine, S.W.; et al. Haralick texture analysis of prostate MRI: Utility for differentiating non-cancerous prostate from prostate cancer and differentiating prostate cancers with different Gleason scores. Eur. Radiol. 2015, 25, 2840–2850. [CrossRef] 26. Mäenpää, T.; Pietikäinen, M. Multi-scale binary patterns for texture analysis. Lect. Notes Comput. Sci. 2003, 2749, 885–892. [CrossRef] 27. Gola, J.; Britz, D.; Staudt, T.; Winter, M.; Schneider, A.S.; Ludovici, M.; Mücklich, F. Advanced microstructure classification by data mining methods. Comput. Mater. Sci. 2018, 148, 324–335. [CrossRef] 28. Gupta, C.; Dey, G.K.; Chakravartty, J.K.; Srivastav, D.; Banerjee, S. A study of bainite transformation in a new CrMoV steel under continuous cooling conditions. Scr. Mater. 2005, 53, 559–564. [CrossRef] 29. Texture Analysis Using the Gray-Level Co-Occurrence Matrix (GLCM) - MATLAB & Simulink - MathWorks Deutschland, (n.d.). Available online: https://de.mathworks.com/help/images/texture-analysis-using-the- gray-level-co-occurrence-matrix-glcm.html (accessed on 13 March 2020). 30. Dutta, S.; Barat, K.; Das, A.; Das, S.K.; Shukla, A.K.; Roy, H. Characterization of micrographs and fractographs of Cu-strengthened HSLA steel using image texture analysis. Measurement 2014, 47, 130–144. [CrossRef] Metals 2020, 10, 630 19 of 19

31. Pietikäinen, M.; Zhao, G. Two decades of local binary patterns: A survey. In Advances in Independent Component Analysis and Learning Machines; Academic Press: Cambridge, MA, USA, 2015; pp. 175–210. [CrossRef] 32. Lahdenoja, O.; Poikonen, J.; Laiho, M. Towards Understanding the Formation of Uniform Local Binary Patterns. ISRN Mach. Vis. 2013, 2013, 1–20. [CrossRef] 33. Kitahara, A.R.; Holm, E.A. Microstructure Cluster Analysis with Transfer Learning and Unsupervised Learning. Integr. Mater. Manuf. Innov. 2018, 7, 148–156. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).