Predicting Hard Rock Pillar Stability Using GBDT, Xgboost, and Lightgbm Algorithms

mathematics

Article Predicting Hard Rock Pillar Stability Using GBDT, XGBoost, and LightGBM Algorithms

Weizhang Liang 1,2, Suizhi Luo 3, Guoyan Zhao 1 and Hao Wu 1,*

1 School of Resources and Safety Engineering, Central South University, Changsha 410083, China; [email protected] (W.L.); [email protected] (G.Z.) 2 The Robert M. Buchan Department of Mining, Queen’s University, Kingston, ON K7L 3N6, Canada 3 College of Systems Engineering, National University of Defense Technology, Changsha 410073, China; [email protected] * Correspondence: [email protected]

Received: 17 April 2020; Accepted: 8 May 2020; Published: 11 May 2020

Abstract: Predicting pillar stability is a vital task in hard rock mines as pillar instability can cause large-scale collapse hazards. However, it is challenging because the pillar stability is affected by many factors. With the accumulation of pillar stability cases, machine learning (ML) has shown great potential to predict pillar stability. This study aims to predict hard rock pillar stability using gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost), and light gradient boosting machine (LightGBM) algorithms. First, 236 cases with five indicators were collected from seven hard rock mines. Afterwards, the hyperparameters of each model were tuned using a five-fold cross validation (CV) approach. Based on the optimal hyperparameters configuration, prediction models were constructed using training set (70% of the data). Finally, the test set (30% of the data) was adopted to evaluate the performance of each model. The precision, recall, and F1 indexes were utilized to analyze prediction results of each level, and the accuracy and their macro average values were used to assess the overall prediction performance. Based on the sensitivity analysis of indicators, the relative importance of each indicator was obtained. In addition, the safety factor approach and other ML algorithms were adopted as comparisons. The results showed that GBDT, XGBoost, and LightGBM algorithms achieved a better comprehensive performance, and their prediction accuracies were 0.8310, 0.8310, and 0.8169, respectively. The average pillar stress and ratio of pillar width to pillar height had the most important influences on prediction results. The proposed methodology can provide a reliable reference for pillar design and stability risk management.

Keywords: pillar stability; hard rock; prediction; gradient boosting decision tree (GBDT); extreme gradient boosting (XGBoost); light gradient boosting machine (LightGBM)

1. Introduction Pillars are essential structural units to ensure mining safety in underground hard rock mines. Their functions are to provide safe access to working areas, and support the weight of overburden rocks for guaranteeing global stability [1,2]. Instable pillars can cause large-scale catastrophic collapse, and significantly increase safety hazards of workers [3]. If a pillar fails, the adjacent pillars have to bear a larger load. The increased load may exceed the strength of adjacent pillars, and lead to their failure. Like the domino effect, rapid and large-scale collapse can be induced [4]. In addition, with the increase of mining depth, the ground stress is larger and pillar instability accidents become more frequent [5–7]. Accordingly, determining pillar stability is essential for achieving safe and efficient mining. In general, there are three types of methods to determine pillar stability in hard rock mines. The first one is the safety factor approach. The safety factor indicates the ratio of pillar strength to

Mathematics 2020, 8, 765; doi:10.3390/math8050765 www.mdpi.com/journal/mathematics Mathematics 2020, 8, 765 2 of 17 stress [8]. Using this method, three parts, namely, pillar strength, pillar stress, and safety threshold, should be determined. For the calculation of pillar strength, many empirical methods have been proposed, such as the linear shape effect, power shape effect, size effect, and Hoek–Brown formulas [9]. For the determination of pillar stress, tributary area theory, and numerical modeling technology are the main approaches [10]. Based on the pillar strength and stress, the safety factor can be calculated. A larger safety factor means the pillar is more stable. Theoretically, the safety threshold is assigned as 1. If the safety factor is larger than 1, the pillar is stable, otherwise it is unstable [11]. Considering the possible deviations of this method, the safety threshold is generally larger than 1 to ensure safety in practice. Although the safety factor method is simple, the unified pillar strength formula and safety threshold are still not determined. The second one is the numerical simulation technique. Numerical simulation methods have been widely used because the complicated boundary conditions and rock mass properties can be considered. Mortazavi et al. [12] adopted fast Lagrangian analysis of continua (FLAC) approach to study the failure process and non-linear behavior of rock pillars; Shnorhokian et al. [13] used FLAC3D to assess the pillar stability based on different mining sequence scenarios; Elmo and Stead [14] integrated finite element method (FEM) and discrete element method (DEM) to investigate the failure characteristics of naturally fractured pillars; Li et al. [15] used rock failure process analysis (RFPA) to evaluate the pillar stability under coupled thermo-hydrologic-mechanical conditions; Jaiswal et al. [16] utilized boundary element method (BEM) to simulate asymmetry in the induced stresses over pillars; and Li et al. [17] introduced finite-discrete element method (FDEM) to analyze the mechanical behavior and failure mechanisms of the pillar. In addition, some scholars combined numerical simulation and other mathematical models to study pillar stability. Deng et al. [18] used FEM, neural networks, and reliability analysis to optimize pillar design; and Griffiths et al. [19] adopted random field theory, elastoplastic finite element algorithm and Monte-Carlo approach to analyze pillar failure probability. Using numerical simulation techniques, the complex failure behaviors of pillars can be modeled and the failure process and range can be obtained. However, because of the complicated nonlinear characteristics and anisotropic nature of rock mass, the model inputs and constitutive equations are not easy to be accurately determined [20]. Therefore, the reliability of results obtained by this method is limited. The third one is the machine learning (ML) algorithm. Big data has been proved to be useful for managing emergencies and disasters [21]. With the accumulation of pillar stability cases, it provides researchers the opportunity to develop advanced predictive models using ML algorithms. Tawadrous and Katsabanis [22] used the artificial neural networks (ANN) to analyze the stability of surface crown pillars; Wattimena [23] introduced the multinomial logistic regression (MLR) for pillar stability prediction; Ding et al. [24] adopted a stochastic gradient boosting (SGB) model to predict pillar stability, and found that this model achieved a better performance than the random forest (RF), support vector machine (SVM), and multilayer perceptron neural networks (MPNN); Ghasemi et al. [25] utilized J48 algorithm and SVM to develop two pillar stability graphs, and obtained acceptable prediction accuracy; and Zhou et al. [9] compared the prediction performance of pillar stability using six supervised learning methods, and revealed that SVM and RF performed better. Although these ML algorithms can solve pillar stability prediction issues to some extent, none can be applied to all engineering conditions. To date, there is not a uniform algorithm under the consensus of the mining industry. Compared with traditional safety factor and numerical simulation methods, ML approach can deeply find implicit relationships between variables, and well handle nonlinear problems. Therefore, it is a promising method for predicting pillar stability. In addition to these above ML algorithms, the gradient boosting decision tree (GBDT) method has shown great prediction performances in many fields [26–28]. As one of the ensemble learning algorithms, it integrates multiple decision trees (DTs) into a strong classifier using the boosting approach [29]. DTs refer to a ML method using tree-like structure, which can deal with various types of data and trace every path to the prediction results [30]. However, DTs are easy to overfit and sensitive to noises of the dataset. Through the integration of DTs, the overall prediction performances of GBDT become better because the errors of DTs are compensated Mathematics 2020, 8, 765 3 of 17 by each other. Under the framework of GBDT, extreme gradient boosting (XGBoost) [31] and light gradient boosting machine (LightGBM) [32] have been proposed recently. They have also received wide attentions because of their excellent performances. Particularly, these three algorithms work well with small datasets. Additionally, overfitting, which means the model fits the existing data too exactly but fails to predict the future data reliably, can be avoided to some extent [30]. However, to the best of our knowledge, GBDT, XGBoost, and LightGBM algorithms have not been used to predict pillar stability in hard rock mines. The objective of this study is to predict hard rock pillar stability using GBDT, XGBoost, and LightGBM algorithms. First, pillar stability cases are collected from seven hard rock mines. Next, these three algorithms are applied to predict pillar stability. Finally, their comprehensive performances are analyzed and compared with the safety factor approach and other ML algorithms.

2. Data Acquisition The existing hard rock pillar cases are obtained as supportive data for the establishment of prediction model. In this study, 236 samples were collected from seven underground hard rock mines [8,10,33]. These mines include the Elliot Lake uranium mine, Selebi-Phikwe mine, Open stope mine, Zinkgruvan mine, Westmin Resources Ltd.’s H-W mine, Marble mine, and Stone mine. The pillar stability data is shown in Table1 (the complete database is available in Table S1), where A1 indicates the pillar width, A2 indicates the pillar height, A3 indicates the ratio of pillar width to pillar height, A4 indicates the uniaxial compressive strength, and A5 indicates the average pillar stress.

Table 1. Pillar stability data.

Samples A1 (m) A2 (m) A3 A4 (MPa) A5 (MPa) Level 1 5.9 4 1.48 172 49.6 Stable 2 7.2 4 1.8 172 56.7 Stable 3 7.8 4 1.95 172 99.9 Unstable ...... 234 14.25 18 0.79 104 5.33 Unstable 235 13 17 0.76 104 16.67 Stable 236 7.75 11 0.70 104 4.57 Unstable

From Table1, it can be seen that each sample contains five indicators and the level of pillar stability. These five indicators have been widely used in a large amount of literature [9,10,24,33]. The advantages are two-fold: (1) They can reflect the necessary conditions of pillar stability, such as pillar size, strength, and load; (2) their values are relatively easy to be obtained. The descriptive statistics of each indicator are listed in Table2. Besides, according to the instability mechanism and failure process of pillars, the level of pillar stability is divided into three types: stable, instable, and failed. Among them, the stable level indicates that the pillar shows no sign of stress induced fracturing, or only has minor spallings which do not affect pillar stability; the instable level indicates that the pillar is partially failed, and has prominent spallings, but still possesses a certain supporting capacity; the failed level indicates that the pillar is crushed, and has pronounced openings of joints, which shows a great collapse risk. The distribution of these three pillar stability levels is described in Figure1.

Table 2. Descriptive statistics of each indicator.

Samples A1 (m) A2 (m) A3 A4 (MPa) A5 (MPa) Minimum 1.90 2.4 0.21 61 0.14 Maximum 45 61 4.50 316 127.60 Mean 11.51 12.58 1.17 141.06 41.50 Standard deviation 7.75 11.34 0.61 64.62 31.42 Mathematics 2020, 8, 765 4 of 17 Mathematics 2020, 8, x FOR PEER REVIEW 4 of 18 Mathematics 2020, 8, x FOR PEER REVIEW 4 of 18

InstableInstable FailedFailed 5353 casescases 8383 casescases

Stable 100 cases

FigureFigure 1.1.1. DistributionDistribution of pillar stability stability levels. levels.

TheThe box box plotplot plot ofof of eacheach each indicatorindicator indicator forforfor didifferentdifferentfferent pillarpillar stabilitystability levellevel is is shown shownshown in inin Figure FigureFigure 2.2 2.. According AccordingAccording tototo FigureFigure Figure2 2,, 2, some some some meaningful meaningful meaningful characteristics characteristicscharacteristics can can bebe found.found. First,First,First, allall indicatorsindicators have havehave several severalseveral outliers. outliers.outliers. Second, the pillar stability level is negatively correlated withAA1 and AA3(the Pearson correlation Second,Second, the the pillar pillar stability stability levellevel isis negativelynegatively correlated withwith A11 andand A33(the(the Pearson Pearson correlation correlation coecoefficientsfficients are are −0.3810.381 andand −0.266, respectively), while while is is positively positively correlated correlated with with AA5(the(the Pearson Pearson coefficients are −−0.381 and −0.266, respectively), while is positively correlated with A5 5 (the Pearson correlation coefficient is 0.433). However, there are no obvious correlations with A2 and A4 (the Pearson correlationcorrelation coefficient coefficient is is 0.0.433).433). However,However, there are no obviousobvious correlationscorrelations with with AA2 and and AA4 (the(the correlation coefficients are 0.121 and 0.079, respectively). Third, the distances between2 upper 4 and Pearson correlation coefficients− are −0.121 and 0.079, respectively). Third, the distances between lowerPearson quartiles correlation are di ffcoefficientserent for the are same −0.121 indicators and in0.079, various respectively). levels. Fourth, Third, there the are distances some overlapping between upper and lower quartiles are different for the same indicators in various levels. Fourth, there are partsupper for and the lower range quartiles of indicator are valuesdifferent in diforfferent the same levels. indicators Fifth, asthe in medianvarious islevels. not in Fourth, the center there of theare some overlapping parts for the range of indicator values in different levels. Fifth, as the median is not box,some the overlapping distribution parts of indicator for the range values of isindicator asymmetric. values All inthese different phenomena levels. Fifth, illustrate as the the median complexity is not in the center of the box, the distribution of indicator values is asymmetric. All these phenomena ofin pillarthe center stability of the prediction. box, the distribution of indicator values is asymmetric. All these phenomena illustrateillustrate the the complexity complexity of of pillar pillar stabilitystability prediction.prediction.

Figure 2. Cont.

Mathematics 2020, 8, 765 5 of 17 Mathematics 2020, 8, x FOR PEER REVIEW 5 of 18 Mathematics 2020, 8, x FOR PEER REVIEW 5 of 18

Figure 2. Distribution of pillar stability levels. FigureFigure 2.2. DistributionDistribution of pillar stability levels. levels.

3.3.3. Methodology Methodology

3.1.3.1.3.1. GBDT GBDT Algorithm Algorithm GBDTGBDTGBDT isis is an an an ensemble ensembleensemble ML MLML algorithm algorithm using using multiple multiplemultiple DTs DTsDTs as base asas base base learners. learners. learners. Every Every Every decision decision decision tree tree (DT)tree is(DT)(DT) not is independent, is not not independent, independent, because becausebecause a new addedaa new DTadded increases DTDT increasesincreases emphasis emphasisemphasis on the misclassiﬁedon on the the misclassified misclassified samples samples attainedsamples byattainedattained previous by by previous DTsprevious [34]. DTs TheDTs [34]. diagram[34]. TheThe ofdiagram GBDT of algorithm GBDT algorithmalgorithm is shown is is in shown shown Figure in in3 .Figure Figure It can 3. be 3. It noticedIt can can be be noticed that noticed the residualthatthat the the residual of residual former of of DTs formerformer is taken DTsDTs asisis taken the input as the for inin theputput next forfor thethe DT. nextnext Then, DT. DT. the Then, Then, added the the added DT added is usedDT DT is tois used used reduce to to residual,reducereduce residual, residual, so that theso so that lossthat the decreasesthe lossloss decreasesdecreases following following the negative thethe negativenegative gradient gradient gradient direction direction direction in each in iteration.in each each iteration. iteration. Finally, theFinally,Finally, prediction the the prediction prediction result is determinedresultresult isis determineddetermined based on based the sum onon the ofthe results sumsum of of from results results all from DTs.from all all DTs. DTs.

Training setset

Tree 1 T(x; α ) Tree 2 T(x; α ) … Tree nT(x; α ) Tree 1 T(x; α11) Tree 2 T(x; α22) … Tree nT(x; αn n) Residual ResidualResidual

Result 1 f1 (x) Result 2 f2 (x) … Result nfn (x) Result 1 f1 (x) Result 2 f2 (x) … Result nfn (x)

γ2 γ2 γ1 γn γ1 γn F(x) =sum(γ f (x)) F(x) =sum(γi fi (x)) i i Figure 3. Diagram of GBDT algorithm. FigureFigure 3.3. DiagramDiagram ofof GBDTGBDT algorithm.algorithm.

{} m m () Let x xyyii, m= indicates the sample data, wherex x=iiiri=,,,(xxx12x  xx )represent the indicators, Let {}ixy, ,i i=1i indicates1 the sample data, where i x =,,,()xx1i, 2i,  , xri represent the indicators, Let iii=1 indicates the sample data, where iiiri12··· represent the indicators, andandy i denotesy denotes the the label. label. The The speciﬁc specific steps steps of of GBDT GBDT are are indicated indicated as as follows follows [35 [35]:]: and y i denotes the label. The specific steps of GBDT are indicated as follows [35]: Stepi 1: The initial constant value γγis obtained as Step 1: The initial constant value γ is obtained as Step 1: The initial constant value is obtained asm m Fx()= argminXm Ly ( ,)γ 0 γ  i (1) FFx0(x()) ==argmin argmini=1 L Ly( (yi, ,)γγ) (1) 0 γγ  i (1) = where Ly(,)γ is the loss function. i=i 11 i γ where Ly(,)i is the loss function. whereStepL(yi ,2:γ )Theis the residual loss function. along the gradient direction is written as StepStep 2:2: The residualresidual alongalong thethe gradientgradient directiondirection isis writtenwritten asas ∂Ly(,()) Fx ˆ =− ii yi ∂ (2) " Ly(,())∂Fxii() Fx# yˆ =−∂L(yi, F(xii)) f ()=xf () x yˆ =i ∂ n−1 (2)(2) i Fx()i − ∂F(x ) f ()=xfn−1 () x i f (x)= fn 1(x) where n indicates the number of iterations, and nN=1, 2, , − . =  where n indicates the number of iterations, and nN1, 2, , .

Mathematics 2020, 8, 765 6 of 17 where n indicates the number of iterations, and n = 1, 2, , N. ··· Step 3: The initial model T(xi; αn) is obtained by ﬁtting sample data, and the parameter αn is calculated based on the least square method as

m X 2 αn = argmin (yˆi βT(xi; α)) (3) α,β − i=1

Step 4: By minimizing the loss function, the current model weight is expressed as

Xm γn = argmin L(yi, Fn 1(x) + γT(xi; αn)) (4) γ − i=1

Step 5: The model is updated as

Fn(x) = Fn 1(x) + γnT(xi; αn) (5) − This loop is performed until the speciﬁed iterations times or the convergence conditions are met.

3.2. XGBoost Algorithm XGBoost algorithm was proposed by Chen and Guestrin [31] based on the GBDT structure. It has received widespread attentions because of its excellent performances in Kaggle’s ML competitions [36]. Different from the GBDT, XGBoost introduces the regularization term in the objective function to prevent overfitting. The objective function is defined as

Xn Xt O = L(yi, F(xi)) + R( fk) + C (6) i=1 k=1 where R( fk) denotes the regularization term at the k time iteration, and C is a constant term, which can be selectively omitted. The regularization term R( fk) is expressed as

H 1 X R( f ) = αH + η w2 (7) k 2 j j=1 where α represents the complexity of leaves, H indicates the number of leaves, η denotes the penalty parameter, and wj is the output result of each leaf node. In particular, the leaves indicate the predicted categories following the classiﬁcation rules, and the leaf node indicates the node of the tree that cannot be split. Moreover, as opposed to use the ﬁrst-order derivative in GBDT, a second-order Taylor series of objective function is adopted in XGBoost. Suppose the mean square error is used as the loss function, then the objective function can be derived as

Xn XH 1 2 1 2 O = [piwq(x ) + (qiw )] + αH + η w (8) i 2 q(xi) 2 j i=1 j=1 where q(xi) indicates a function that assigns data points to the corresponding leaves, and gi and hi denote the ﬁrst and second derivative of loss function, respectively. The ﬁnal loss value is calculated based on the sum of all loss values. Because samples correspond to leaf nodes in the DT, the ultimate loss value can be determined by summing the loss values of leaf nodes. Therefore, the objective function is also expressed as Mathematics 2020, 8, 765 7 of 17

T X 1 O = [P w + (Q + η)w2] + αH (9) j j 2 j j j=1 P P where Pj = pi, Qj = qi, and Ij indicates all samples in leaf node j. i I i I ∈ j ∈ j To conclude, the optimization of objective function is converted to a problem of determining the minimum of a quadratic function. In addition, because of the introduction of regularization term, XGBoost has a better ability to against overfitting. Mathematics 2020, 8, x FOR PEER REVIEW 7 of 18 3.3. LightGBM Algorithm LightGBM was another algorithm designed by Microsoft Research Asia [32] using GBDT framework.LightGBM It aims was to anotherimprove algorithmthe computational designed efficiency, by Microsoft so that Research the prediction Asia problems [32] using with GBDT big framework.data can be solved It aims more to improve effectively. the computational In GBDT algorithm, efficiency, presorting so that the approach prediction is used problems to select with and big datasplit canindicators. be solved Although more eff ectively.this method In GBDT can precisel algorithm,y determine presorting the approach splitting is point, used toit selectrequires and more split indicators.time and memory. Although In Ligh this methodtGBM, the can histogram-based precisely determine algorithm the splitting and trees point, leaf-wise it requires growth more strategy time andwith memory. a maximum In LightGBM, depth limit the histogram-basedis adopted to incr algorithmease the and training trees leaf-wise speed growthand reduce strategy memory with a maximumconsumption. depth limit is adopted to increase the training speed and reduce memory consumption. The histogram-based DT algorithm is shown in FigureFigure4 4.. ItIt cancan bebe seenseen thatthat thethe successivesuccessive floating-pointfloating-point eigenvalueseigenvalues areare discretizeddiscretized intointo ss smallsmall bins.bins. After that, these bins are used to construct the histogram with width s. Once Once the the data data is is traversed traversed at at the the firs firstt time, the required statistics (sum of gradients and number of samples in each bin) are accumulated in the histogram. Based on the discrete value of histogram, the optimal segmentation point can be found [[37].37]. By using this method, the cost of storage and calculation can be reduced.

▪▪▪ ▪▪▪ ▪▪▪ ▪▪▪ ▪▪▪ Features

▪▪▪ Features ▪▪▪ Construct Decision Features Convert histogram tree Data Data Bins Float data Bin data Histogram ( ) = Histogram ( ) - Histogram ( ) Figure 4. Histogram-based decision tree algorithm.

The level-wiselevel-wise andand leaf-wiseleaf-wise growth growth strategies strategies are are shown shown in Figurein Figure5. According 5. According to the to level-wise the level- growthwise growth strategy, strategy, the leaves the leaves on the on same the same layer layer are simultaneously are simultaneously split. split. It is favorable It is favorable to optimize to optimize with multiplewith multiple threads, threads, and control and modelcontrol complexity. model complexity. However, leavesHowever, on same leaves layer on are same indiscriminately layer are treated,indiscriminately whereas treated, they have whereas different they information have different gain. information Information gain. gain Information indicates gain the expectedindicates reductionthe expected in entropy reduction caused in entropy by splitting caused the nodesby spli basedtting onthe attributes, nodes based which on canattributes, be determined which bycan [ 38be] determined by [38] X Bv IG(B, V) = En(B) | |En(Bv) (10) − BB =−v Values(V) v IG(,) B V En () B∈  En ( Bv ) (10) vValuesV∈ () B XD En(B) = D pd log2 pd (11) =−− (11) EnB()d=1 pdd log2 p d =1 where En(B) is the information entropy of the collection B, pd is the ratio of B pertaining to category d, where EnB() is the information entropy of the collection B, pd is the ratio of B pertaining to D is the number of categories, v is the value of attribute V, and Bv is the subset of B for which attribute category d , D is the number of categories, v is the value of attribute V , and B is the subset has value v. v of BActually, for which many attribute leaves has with value low informationv . gain are unnecessary to be searched and split, which increasesActually, a great many deal leaves of extra with memory low information consumption gain and are causes unnecessary this method to be tosearched be ineff ective.and split, In contrast,which increases leaf-wise a great growth deal strategy of extra is memory more effi consumptioncient because and it only causes split this the method leaf that to hasbe ineffective. the largest In contrast, leaf-wise growth strategy is more efficient because it only split the leaf that has the largest information gain on the same layer. Furthermore, considering this strategy may cause trees with high depth, resulting in overfitting, a maximum depth limitation is adopted during the growth of trees [39].

Mathematics 2020, 8, 765 8 of 17 information gain on the same layer. Furthermore, considering this strategy may cause trees with high Mathematics 2020, 8, x FOR PEER REVIEW 8 of 18 depth,Mathematics resulting 2020, 8 in, x overﬁtting,FOR PEER REVIEW a maximum depth limitation is adopted during the growth of trees8 of 18 [39 ].

… …

Level-wise tree growth Level-wise tree growth

… …

Leaf-wise tree growth Leaf-wise tree growth Figure 5. Level-wise and leaf-wise tree construction. FigureFigure 5. 5. Level-wiseLevel-wise and leaf-wise tree tree construction. construction. 4. Construction of Prediction Model 4.4. Construction Construction of of Prediction Prediction Model Model The construction process of prediction model is shown in Figure 6. First, approximately 70% and The construction process of prediction model is shown in Figure6. First, approximately 70% 30% Theof the construction original pillar process stability of prediction data are modelselected is asshown training in Figure and test 6. First, sets, approximately respectively. Second, 70% and a and 30% of the original pillar stability data are selected as training and test sets, respectively. 30%five-fold of the cross original validation pillar (CV)stability approach data are is utilizedselected to as tune training the model and test hyperparameters. sets, respectively. Third, Second, using a Second, a five-fold cross validation (CV) approach is utilized to tune the model hyperparameters. five-foldthe optimal cross hyperparameters validation (CV) approachconfiguration, is utilized the pred to tuneiction the model model is hyperparameters. fitted based on Third,training using set. Third, using the optimal hyperparameters configuration, the prediction model is fitted based on training theFourth, optimal the testhyperparameters set is adopted configuration,to evaluate model the predperformancesiction model according is fitted to basedthe overall on training prediction set. set.Fourth,results Fourth, andthe the testthe test setprediction set is isadopted adopted ability to toevaluate for evaluate each model stabi modellity performances performances level. Last, accordingby according comparing to tothe thethe overall overallcomprehensive prediction prediction resultsresultsperformance and and the the of prediction theseprediction models, ability ability the optimal forfor eacheach model stabilitystabi islity de level. termined. Last, Last, If by the by comparing prediction comparing performancethe the comprehensive comprehensive of this performanceperformancemodel is acceptable, of of these these models, models,then it the thecan optimal optimal be adopted modelmodel for is dedetermined.deployment.termined. If If theThe the prediction predictionentire calculation performance performance process of ofthis is this modelmodelperformed is acceptable,is acceptable, in Python then 3.7then it canbased it becan on adopted bethe adopted scikit-learn for deployment. for [40],deployment. XGBoost The entire [41],The calculation entireand LightGBM calculation process [42] isprocess libraries, performed is inperformedrespectively. Python 3.7 in based The Python hyperparameter on 3.7 the based scikit-learn on optimization the [scikit-learn40], XGBoost proc [40],ess [41 and XGBoost], and model LightGBM [41], evaluation and LightGBM [42 indexes] libraries, are [42] descripted respectively. libraries, Therespectively.as hyperparameterfollows. The hyperparameter optimization processoptimization and model process evaluation and model indexes evaluation are indexes descripted are descripted as follows. as follows.

Pillar stability database Pillar stability database Stable Instable Failed Stable Instable Failed

Training set Test set Training set Test set GBDT, XGBoost and LightGBMGBDT, XGBoost algorithms and LightGBM algorithms Five-fold CV Five-fold CV Optimal prediction models Optimal prediction models Performance evaluation Indicators importance Performance evaluation Indicators importance

FigureFigure 6. 6.Construction Construction process of of prediction prediction model. model. Figure 6. Construction process of prediction model. 4.1. Hyperparameter Optimization 4.1. Hyperparameter Optimization 4.1. Hyperparameter Optimization MostMost of of ML ML algorithms algorithms contain contain hyperparametershyperparameters that need need to to be be tuned. tuned. These These hyperparameters hyperparameters shouldshouldMost be adjustedbe of MLadjusted algorithms based based on dataset contain on rather datasethyperparameters than rather specifying thanthat need manually.specifying to be tuned. Generally, manually. These the hyperparametersGenerally, hyperparameters the searchshouldhyperparameters methods be adjusted include search based gridmethods search,on includedataset Bayesian grid rather search, optimization, than Bayesian specifying optimization, heuristic manually. search heuristic andGenerally, search randomized andthe searchhyperparametersrandomized [43]. Because search search [43]. the Because randomizedmethods the include randomized search grid method search, search Bayesian ismethod more is eoptimization,ﬃ morecient efficient for simultaneously heuristic for simultaneously search tuningand randomizedtuning multiple search hyperparameters, [43]. Because the it is randomized used to reco seargnizech themethod best set is moreof hyperparameters efficient for simultaneously in this study. tuning multiple hyperparameters, it is used to recognize the best set of hyperparameters in this study.

Mathematics 2020, 8, 765 9 of 17

multipleMathematics hyperparameters, 2020, 8, x FOR PEER it REVIEW is used to recognize the best set of hyperparameters in this9 of 18 study. In general, K-fold CV approach is utilized for hyperparameters configuration [44]. In our work, five-foldIn general, CV method K-fold is CV used, approach which is is utilized illustrated for hy inperparameters Figure7. The configuration original training [44]. setIn our is randomly work, five- split fold CV method is used, which is illustrated in Figure 7. The original training set is randomly split into five equal size subsamples. Among them, a singular subsample is adopted as the validation set, into five equal size subsamples. Among them, a singular subsample is adopted as the validation set, and the other four subsamples are used as the training sub-set. This procedure is repeated five times and the other four subsamples are used as the training sub-set. This procedure is repeated five times until eachuntil subsampleeach subsample is selected is selected as a as validation a validation set se once.t once. Thereafter, Thereafter, the the average average accuracy accuracy ofof thesethese five validationfive validation sets is used sets to is determineused to dete thermine optimal the optimal hyperparameter hyperparameter values. values.

Training set 1st iteration Accuracy 1 2nd iteration Accuracy 2 3rd iteration Accuracy 3 4th iteration Accuracy 4 5th iteration Accuracy 5

Training sub-set Validation set Figure 7. Flowchart of five-fold CV. Figure 7. Flowchart of five-fold CV. In this study, some critical hyperparameters in GBDT, XGBoost, and LightGBM algorithms are In this study, some critical hyperparameters in GBDT, XGBoost, and LightGBM algorithms are tuned, as shown in Table 3. The specific meanings of these hyperparameters are also explained in tuned, as shown in Table3. The specific meanings of these hyperparameters are also explained Table 3. First, the search range of different hyperparameters values is specified. In particular, for in Tabledifferent3. First, algorithms, the search the search range range of di offferent the same hyperparameters hyperparameters values is kept is consistent. specified. Further In particular, on, for diffaccordingerent algorithms, to the maximum the search average range accuracy, of the the same optimal hyperparameters values for each is set kept of hyperparameters consistent. Further are on, accordingobtained, to the which maximum are indicted average in Table accuracy, 3. the optimal values for each set of hyperparameters are obtained, which are indicted in Table3. Table 3. Hyperparameters optimization results. Table 3. Hyperparameters optimization results. Search Optimal Algorithm Hyperparameters Meanings Algorithm Hyperparameters Meanings SearchRanges Ranges OptimalValues Values GBDC n_estimators Number of trees (1000, 2000) 1200 GBDC n_estimators Number of trees (1000, 2000) 1200 learning_rate Shrinkage coefficient of each tree (0.01, 0.2) 0.17 learning_rate Shrinkage coefficient of each tree (0.01, 0.2) 0.17 max_depth max_depth Maximum Maximum depth depth of a treeof a tree (10, (10, 100) 100) 20 20 MinimumMinimum number number of samplesof samples for min_samples_leafmin_samples_leaf (2, 11)(2, 11) 7 7 for leafleaf nodes nodes MinimumMinimum number number of samplesof samples for min_samples_splitmin_samples_split (2, 11)(2, 11) 7 7 for nodesnodes split split XGBoost n_estimators Number of trees (1000, 2000) 1000 XGBoost n_estimators Number of trees (1000, 2000) 1000 learning_rate Shrinkage coefficient of each tree (0.01, 0.2) 0.2 max_depth learning_rate Shrinkage Maximum coefficient depth of a treeof each tree (10, (0.01, 100) 0.2) 0.2 50 max_depth Subsample Maximum ratio of depth columns of a for tree (10, 100) 50 colsample_bytree (0.1, 1) 1 Subsampletree construction ratio of columns for colsample_bytree (0.1, 1) 1 Subsample ratio of training subsample tree construction (0.1, 1) 0.3 Subsamplesamples ratio of training LightGBM n_estimatorssubsample Number of trees (1000,(0.1, 2000) 1) 0.31900 samples learning_rate Shrinkage coefficient of each tree (0.01, 0.2) 0.2 LightGBM max_depth n_estimators Maximum Number depth ofof atrees tree (10, (1000, 100) 2000) 1900 80 num_leaves learning_rate Number Shrinkage of leaves coefficient for each oftree each tree (2, (0.01, 100) 0.2) 0.2 11 max_depth Maximum depth of a tree (10, 100) 80 num_leaves Number of leaves for each tree (2, 100) 11 4.2. Model Evaluation Indexes 4.2. Model Evaluation Indexes The accuracy, precision, recall and F1 measures have been widely used for the performance evaluationThe of MLaccuracy, algorithms precision, [43 ].recall Among and them,F1 measures accuracy have indicates been widely the proportion used for the of performance samples that is correctlyevaluation predicted; of ML precision algorithms is the [43]. ability Among to accuratelythem, accuracy predict indicates samples; the proportion recall denotes of samples the capability that is of correctlycorrectly predicting predicted; as many precision samples is the as ability possible to accurately in actual predict samples; samples; and F recall1 represents denotes athe comprehensive capability

Mathematics 2020, 8, 765 10 of 17 metric that measures the performance of both precision and recall. Accordingly, these indexes are adopted to evaluate model performances in this study. Assume the confusion matrix is expressed as    g11 g12 g1E   ···   g g g   21 22 2E  G =  . . ··· .  (12)  . . .. .   . . . .    g gE gEE E1 2 ··· where E is the number of pillar stability levels, gaa represents the number of samples correctly predicted for level a, and gab denotes the number of samples of level a that are classiﬁed to level b. Based on the confusion matrix, the precision, recall and F1 measure for each pillar stability level are respectively determined by gaa Pr = (13) PE gab a=1 gaa Re = (14) PE gab b=1 2 Pr Re F = × × (15) 1 Pr + Re To further reﬂect the overall prediction performance, the accuracy and macro average of precision, recall and F (expressed as macro Pr, macro Re and macro F , respectively) are calculated as 1 − − − 1 E 1 X Accuracy = gaa (16) E E P P a=1 gab a=1 b=1

E X gaa macro Pr = ( )/E (17) − E b=1 P gab a=1 E X gaa macro Re = ( )/E (18) − E a=1 P gab b=1 2 macro Pr macro Re macro F = × − × − (19) − 1 macro Pr + macro Re − − 5. Results and Analysis

5.1. Overall Prediction Results The prediction results of GBDT, XGBoost, and LightGBM algorithms were obtained on the test set. Subsequently, the confusion matrix of each algorithm was determined, as shown in Table4. The values on the main diagonal indicated the number of samples correctly predicted. It can be seen that most samples were accurately classiﬁed using these three algorithms. Based on Table4, the accuracy, macro Pr, macro Re, and macro F were calculated based on Equations (16)–(19), − − − 1 which were listed in Table5. According to the results, all these three algorithms had good performances in predicting the pillar stability of hard rock. Furthermore, the accuracy degrees of GBDT and XGBoost were largest and up to 0.8310, followed by LightGBM with an accuracy of 0.8169. Although GBDT and XGBoost possessed the same accuracy, the prediction performances were diﬀerent. Comparing their Mathematics 2020, 8, x FOR PEER REVIEW 11 of 18 predicting the pillar stability of hard rock. Furthermore, the accuracy degrees of GBDT and XGBoost were largest and up to 0.8310, followed by LightGBM with an accuracy of 0.8169. Although GBDT andMathematics XGBoost2020 ,possessed8, 765 the same accuracy, the prediction performances were different. Comparing11 of 17 − − − their values of macro Pr , macro Re , and macro F1 , GBDT performed better than XGBoost. valuesTherefore, of macro based Pron, macrotheir overallRe, and predictionmacro F ,performances, GBDT performed the betterrank was than GBDT XGBoost. > XGBoost Therefore, > − − − 1 LightGBM.based on their overall prediction performances, the rank was GBDT > XGBoost > LightGBM.

Table 4. Prediction results of each algorithm.

PredictedPredicted GBDT XGBoost LightGBM GBDT XGBoost LightGBM 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 25 4 1 26 2 2 26 3 1 True 2 2 112 25 42 1 26 4 210 2 262 3 1 4 11 1 True 2 2 12 2 4 10 2 4 11 1 3 1 32 1 222 22 11 1 23 1 223 2 21 2 2 21

Table 5. Overall prediction results of each algorithm. Table 5. Overall prediction results of each algorithm. − − − Accuracy macro Pr macro Re macro F1 Accuracy macro Pr macro Re macro F1 GBDT 0.8310 0.8132− 0.8211− 0.8160− GBDT 0.8310 0.8132 0.8211 0.8160 XGBoostXGBoost 0.8310 0.8310 0.8199 0.8199 0.8039 0.8039 0.8089 0.8089 LightGBMLightGBM 0.8169 0.8169 0.8043 0.8043 0.7981 0.7981 0.8004 0.8004

5.2. Prediction Prediction Results of Each Stability To analyze the prediction performance of algorithms for each stability level, the precision, recall and F1 indexes were calculated based on Equations (13)–(15), which were shown in Figures 8–10, and F1 indexes were calculated based on Equations (13)–(15), which were shown in Figures8–10, respectively. It It can be seen that the prediction pe performancerformance of each algorithm for different diﬀerent stability level was not the same. For For the the stable stable level, GBDT achieved the highest pr precisionecision value (0.8929) and the highest F1 value (0.8621); and XGBoost achieved the highest recall value (0.8667). For the instable the highest F1 value (0.8621); and XGBoost achieved the highest recall value (0.8667). For the instable level, XGBoost possessed the highest precision valu valuee (0.7692); and GBDT posse possessedssed the highest recall value (0.7500) and the highest F1 value (0.7059). For the failed level, LightGBM obtained the highest value (0.7500) and the highest F1 value (0.7059). For the failed level, LightGBM obtained the highest precision value (0.9130); and XGBoost obtained the highest recall value (0.9200) and the highest F1 precision value (0.9130); and XGBoost obtained the highest recall value (0.9200) and the highest F1 value (0.8846). Moreover, it can be observed that the prediction performance of these algorithms for failed level was the best, followed by stable level. However, However, the prediction performance for instable level was the worst.

0.95 Stable Instable Failed

0.90

0.85

0.80

0.75

Values of precision of Values 0.70

0.65

GBDT XGBoost LightGBM Algorithms Figure 8. Precision values of algorithms for each stability level. Figure 8. Precision values of algorithms for each stability level.

Mathematics 2020, 8, 765x FOR PEER REVIEW 1212 of of 18 17 Mathematics 2020, 8, x FOR PEER REVIEW 12 of 18

0.95 Stable Instable Failed 0.95 Stable Instable Failed 0.90 0.90 0.85 0.85 0.80 0.80 0.75 0.75

Values of recall of Values 0.70 Values of recall of Values 0.70 0.65 0.65 0.60 0.60 GBDT XGBoost LightGBM GBDT XGBoost LightGBM Algorithms Algorithms Figure 9. Recall values of algorithms for each stability level. Figure 9.9. Recall values of algorithms for each stability level.

0.95 0.95 Stable Instable Failed Stable Instable Failed 0.90 0.90

0.85 0.85 1 1 0.80 0.80

0.75 Values of F of Values 0.75 Values of F of Values 0.70 0.70

0.65 0.65 GBDT XGBoost LightGBM GBDT XGBoost LightGBM Algorithms Algorithms

Figure 10. F1 valuesvalues of of each each algorithm for each stability level. Figure 10. F1 values of each algorithm for each stability level. 5.3. Relative Importance of Indicators 5.3. Relative Importance of Indicators 5.3. Relative Importance of Indicators The relative importance of indicators is a valuable reference for taking reasonable measures The relative importance of indicators is a valuable reference for taking reasonable measures to to preventThe relative pillar importance instability. Inof indica this study,tors is the a valuable importance reference degree for of taking each indicatorreasonable was measures obtained to prevent pillar instability. In this study, the importance degree of each indicator was obtained using usingprevent GBDT, pillar XGBoost, instability. and In LightGBM this study, algorithms, the importance which degree was indicated of each indicator in Figure was11. Based obtained on GBDTusing GBDT, XGBoost, and LightGBM algorithms, which was indicated in Figure 11. Based on GBDT algorithm,GBDT, XGBoost, the rank and of LightGBM importance algorithms, degree was whichA5 > wasA>>>>3 > indicatedA1 > A4 in> FigureA2. According 11. Based to on XGBoost GBDT algorithm, the rank of importance degree was A53142>>>>AAAA . According to XGBoost algorithm, thethe rank rank of of importance importance degree degree was wasA5 > AA531423 > AAAAA1 > A4 > A2. Using . According LightGBM to algorithm, XGBoost algorithm, the rank of importance degree was A >>>>AAAA. Using LightGBM algorithm, thealgorithm, rank of importancethe rank of degreeimportance was A degree3 > A5 >wasA1 >A53142A>>>>2 >AAAAA4. Therefore, there. Using were LightGBM two most algorithm, important >>>>53142 indicators:the rank of importanceA5 (average degree pillar stress)was A and35124>>>>AAAAA3 (ratio of pillar. widthTherefore, to pillar there height). were two most important the rank of importance degree was A35124AAAA. Therefore, there were two most important indicators: A5 (average pillar stress) and A3 (ratio of pillar width to pillar height). indicators: A5 (average pillar stress) and A3 (ratio of pillar width to pillar height).

Mathematics 2020, 8, 765 13 of 17

Mathematics 2020, 8, x FOR PEER REVIEW 13 of 18

GBDT XGBoost LightGBM

0.4

0.3

0.2

Importancedegree 0.1

0.0 A A A A A 1 2 3 4 5 FigureFigure 11. 11.Importance Importance degree of of each each indicator. indicator.

6. Discussions6. Discussions AlthoughAlthough the the pillar pillar stability stability can becan well be predicted well predicted using GBDT,using XGBoost,GBDT, XGBoost, and LightGBM and LightGBM algorithms, thealgorithms, prediction the performance prediction forperforma differentnce instablefor different level instable was not level the was same. not The the reasonsame. The may reason be two-fold. may Onebe istwo-fold. that the One number is that of samplesthe number for instableof samples level for isinstable smaller level than is other smaller two than levels. other Another two levels. is that theAnother discrimination is that the boundary discrimination of instable boundary level is of more instable uncertain, level whichis more would uncertain, influence which the would quality of data.influence As data-driventhe quality of approaches, data. As data-driven the prediction approaches, performances the prediction of GBDT, performances XGBoost, and of LightGBM GBDT, algorithmsXGBoost, areand greatly LightGBM affected algorithms by the numberare greatly and affe qualitycted by of the supportive number and data. quality Therefore, of supportive compared withdata. other Therefore, two levels, compared the prediction with other performance two levels, forthe instableprediction level performance was worse. for instable level was worse.According to the analysis of indicator importance, A5 and A3 were the most important indicator. The reasonAccording may be to thatthe theanalysis pillar of stability indicator is greatlyimportance, affected A5 by and the A external3 were the stress most conditions important and inherentindicator. dimension The reason characteristics. may be that According the pillar tostab theility field is experience,greatly affected the spallingby the external phenomenon stress of pillarconditions was more and common inherent in dimension deep mines characteristics. because of high According stress, and to the the field pillar experience, with small theA3 valuespalling was morephenomenon vulnerable of to pillar be damaged. was more Based common on thein deep importance mines because degrees of of high indicators, stress, and some the measures pillar with can besmall adopted A3 to value improve was more pillar vulnerable stability fromto be damaged. two directions. Based Oneon the is importance reducing pillar degrees stress, of indicators, such as the adjustmentsome measures of excavation can be adopted sequence to andimprove relief pillar of pressure stability [32 from]. Another two directions. is optimizing One is pillar reducing parameters, pillar suchstress, as the such increase as the of adjustment pillar width of and excavation ratio of pillar sequence width and to pillarrelief height. of pressure [32]. Another is optimizingTo further pillar illustrate parameters, the eff ectivenesssuch as the ofincrease GBDT, of XGBoost, pillar width and and LightGBM ratio of pillar algorithms, width to the pillar safety factorheight. approach and other ML algorithms were adopted as comparisons. First,To further the safety illustrate factor the approach effectiveness was used of GBDT to determine, XGBoost, pillar andstability. LightGBM According algorithms, to thethe researchsafety of Esterhuizenfactor approach et al. and [33 other], the ML pillar algorithms strength were can beadopted calculated as comparisons. by First, the safety factor approach was used to determine pillar stability. According to the research 0.30 of Esterhuizen et al. [33], the pillar strength can be calculatedA1 by S = 0.65 A 0.30 (20) 4 A 0.59 =××× × A12 SA0.65 4 0.59 (20) A2 Based on Equation (20) and the given pillar stress, the safety factor can be determined by Based on Equation (20) and the given pillar stress, the safety factor can be determined by their their quotient. From the work of Lunder and Pakalnis [45], when η > 1.4, the pillar was stable; quotient. From the work of Lunder and Pakalnis [45], when η >1.4 , the pillar was stable; when when 1 < η < 1.4, the pillar was unstable; and when η < 1, the pillar was failed. According to this 11.4<<η , the pillar was unstable; and when η <1 , the pillar was failed. According to this discrimination method, the prediction accuracy was 0.6610. From the work of González-Nicieza et al. [8], discrimination method, the prediction accuracy was 0.6610. From the work of González-Nicieza et al. when η > 1.25, the pillar was stable; when 0.90 < η < 1.25, the pillar was unstable; and when η < 0.90, [8], when η >1.25 , the pillar was stable; when 0.90<<η 1.25 , the pillar was unstable; and when the pillar was failed. Base on this classification method, the prediction accuracy was 0.6398. Therefore, η < 0.90 the prediction, the accuracypillar was of failed. safety Base factor on this approach classification was lower method, than the that prediction of the proposed accuracy methodology.was 0.6398. It canTherefore, be inferred the prediction that this empirical accuracy methodof safety has fact diorffi approachculty in obtaining was lower satisfactory than that of prediction the proposed results onmethodology. these pillar samples. It can be inferred that this empirical method has difficulty in obtaining satisfactory prediction results on these pillar samples. On the other hand, some ML algorithms, such as RF, adaptive boosting (AdaBoost), MLR, Gaussian On the other hand, some ML algorithms, such as RF, adaptive boosting (AdaBoost), MLR, naive Bayes (GNB), Gaussian processes (GP), DT, MLPNN, SVM, and k-nearest neighbor (KNN), Gaussian naive Bayes (GNB), Gaussian processes (GP), DT, MLPNN, SVM, and k-nearest neighbor were adopted to predict pillar stability. First, some key hyperparameters in these algorithms were

Mathematics 2020, 8, 765 14 of 17 tuned, and the optimization results were listed in Table6. Afterwards, these algorithms with optimal hyperparameters were used to predict pillar stability based on the same dataset. The accuracy of each algorithm was shown in Figure 12. It can be seen that the accuracies of these algorithms were all lower than those of GBDT, XGBoost, and LightGBM algorithms. The accuracies of most algorithms (except for GP) were lower than 0.8, whereas the accuracies of GBDT, XGBoost, and LightGBM algorithms were all higher than 0.8. It demonstrated that compared with other ML algorithms, GBDT, XGBoost, and LightGBM algorithms were better for predicting pillar stability in hard rock mines.

Table 6. Hyperparameters optimization of comparison algorithms.

Algorithm Hyperparameters Meanings Search Range Optimal Values RF n_estimators Number of trees (1000, 2000) 1100 Minimum number of min_samples_split (2, 11) 4 samples for nodes split max_depth Maximum depth of a tree (10, 100) 20 Minimum number of min_samples_leaf (2, 11) 2 samples for leaf nodes AdaBoost n_estimators Number of trees (1000, 2000) 1200 Shrinkage coeﬃcient of each learning_rate (0.01, 0.2) 0.01 tree Norm used in the LR penalty {l1, l2} l2 penalization Inverse of regularization C (0.1, 10) 1.0 1 strength GNB Parameter of radial basis GP a (0.1, 2.0) 0.9 function (a RBF(b)) ∗ Parameter of radial basis b (0.1, 2.0) 0.3 function (a RBF(b)) ∗ Minimum number of DT min_samples_split (2, 11) 4 samples for nodes split max_depth Maximum depth of a tree (10, 100) 80 Minimum number of min_samples_leaf (2, 11) 4 samples for leaf nodes Number of neurons in the MLPNN hidden_layer_sizes (10, 100) 87 hidden layer Initial learning rate for learning_rate_init (0.0001,0.002) 0.0019 weight updates SVM C2 Regularization parameter (0.1, 10) 1.2 KNN n_neighbors Number of neighbors (1, 20) 3 Mathematics 2020, 8, x FOR PEER REVIEW 15 of 18

0.8028 0.8 0.7887 0.7465 0.7465 0.7 0.67610.6761 0.6901 0.6479 0.6 0.5915

0.5

0.4 Accuracy 0.3

0.2

0.1

0.0 RF AdaBoost LR GNB GP DT MLPNN SVM KN Algorithm Figure 12. Accuracy of each comparison algorithm. Figure 12. Accuracy of each comparison algorithm.

Although the proposed approach obtains desirable prediction results, some limitations should be addressed in the future. (1) The dataset is relatively small and unbalanced. The prediction performance of ML algorithms is heavily affected by the number and quality of dataset. Generally, if the dataset is small, the generalization and reliability of model would be influenced. Although GBDT, XGBoost, and LightGBM algorithms work well with small datasets, the prediction performances could be better on a larger dataset. In addition, the dataset is unbalanced, particularly for samples with instable level. Compared with other levels, the prediction performance for the instable level is not good. This illustrates the adverse effects of imbalanced data on results. Therefore, it is meaningful to establish a larger and more balanced pillar stability database. (2) Other indicators may also have influences on the prediction results. Pillar stability is affected by numerous factors, including the inherent characteristics and external environments. Although the five indictors adopted in this study can describe the necessary conditions of pillar stability to some extent, some other indicators may also have effects on pillar stability, such as joints, underground water, and blasting disturbance. Theoretically, the joints and underground water can affect the pillar strength, and blasting disturbance can be deemed as a dynamic stress on the pillar. Accordingly, it is significant to analyze the influences of these indicators on the prediction results.

7. Conclusions To ensure mining safety, pillar stability prediction is a crucial task in underground hard rock mines. This study investigated the performance of GBDT, XGBoost, and LightGBM algorithms for pillar stability prediction. The prediction models were constructed based on training set (165 cases) after their hyperparameters were tuned using the five-fold CV method. The test set (71 cases) were adopted to validate the feasibility of trained models. Overall, the performances of GBDT, XGBoost, and LightGBM were acceptable, and their prediction accuracies were 0.8310, 0.8310, and 0.8169, respectively. By comprehensively analyzing the accuracy and macro average of precision, recall and F1, the rank of overall prediction performance was GBDT > XGBoost > LightGBM. According to the precision, recall and F1 of each stability level, the prediction performance for stable and failed levels was better than that for instable level. Based on the importance scores of indicators from these three algorithms, the average pillar stress and ratio of pillar width to pillar height were the most influential indicators on the prediction results. Compared with the safety factor approach and other ML algorithms (RF, AdaBoost, LR, GNB, GP, DT, MLPNN, SVM, and KNN), the performances of GBDT, XGBoost, and LightGBM were better, which further verified that they were reliable and effective for the pillar stability prediction.

Mathematics 2020, 8, 765 15 of 17

Although the proposed approach obtains desirable prediction results, some limitations should be addressed in the future.

(1) The dataset is relatively small and unbalanced. The prediction performance of ML algorithms is heavily affected by the number and quality of dataset. Generally, if the dataset is small, the generalization and reliability of model would be influenced. Although GBDT, XGBoost, and LightGBM algorithms work well with small datasets, the prediction performances could be better on a larger dataset. In addition, the dataset is unbalanced, particularly for samples with instable level. Compared with other levels, the prediction performance for the instable level is not good. This illustrates the adverse effects of imbalanced data on results. Therefore, it is meaningful to establish a larger and more balanced pillar stability database. (2) Other indicators may also have influences on the prediction results. Pillar stability is affected by numerous factors, including the inherent characteristics and external environments. Although the five indictors adopted in this study can describe the necessary conditions of pillar stability to some extent, some other indicators may also have effects on pillar stability, such as joints, underground water, and blasting disturbance. Theoretically, the joints and underground water can affect the pillar strength, and blasting disturbance can be deemed as a dynamic stress on the pillar. Accordingly, it is significant to analyze the influences of these indicators on the prediction results.

7. Conclusions To ensure mining safety, pillar stability prediction is a crucial task in underground hard rock mines. This study investigated the performance of GBDT, XGBoost, and LightGBM algorithms for pillar stability prediction. The prediction models were constructed based on training set (165 cases) after their hyperparameters were tuned using the five-fold CV method. The test set (71 cases) were adopted to validate the feasibility of trained models. Overall, the performances of GBDT, XGBoost, and LightGBM were acceptable, and their prediction accuracies were 0.8310, 0.8310, and 0.8169, respectively. By comprehensively analyzing the accuracy and macro average of precision, recall and F1, the rank of overall prediction performance was GBDT > XGBoost > LightGBM. According to the precision, recall and F1 of each stability level, the prediction performance for stable and failed levels was better than that for instable level. Based on the importance scores of indicators from these three algorithms, the average pillar stress and ratio of pillar width to pillar height were the most influential indicators on the prediction results. Compared with the safety factor approach and other ML algorithms (RF, AdaBoost, LR, GNB, GP, DT, MLPNN, SVM, and KNN), the performances of GBDT, XGBoost, and LightGBM were better, which further verified that they were reliable and effective for the pillar stability prediction. In the future, a larger and more balanced pillar stability database can be established to further illustrate the adequacy of these algorithms for the prediction of stable, instable, and failed pillar levels. The influences of other indicators on the prediction results are essential to be analyzed. The methodology can also be applied in other fields, such as the risk prediction of landslide, debris flow, and rockburst.

Supplementary Materials: The following are available online at http://www.mdpi.com/2227-7390/8/5/765/s1, Table S1: Pillar stability cases database. Author Contributions: This research was jointly performed by W.L., S.L., G.Z., and H.W. Conceptualization, W.L. and S.L.; Methodology, S.L.; Software, G.Z. and H.W.; Validation, G.Z. and H.W.; Formal analysis, H.W.; Investigation, W.L.; Resources, S.L.; Data curation, G.Z. and H.W.; Writing—original draft preparation, W.L.; Writing—review and editing, W.L., S.L., G.Z., and H.W.; Funding acquisition, G.Z. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the National Natural Science Foundation of China (no. 51774321) and National Key Research and Development Program of China (no. 2018YFC0604606). Conﬂicts of Interest: The authors declare no conﬂict of interest. Mathematics 2020, 8, 765 16 of 17

References

1. Ghasemi, E.; Ataei, M.; Shahriar, K. Prediction of global stability in room and pillar coal mines. Nat. Hazards 2014, 72, 405–422. [CrossRef] 2. Liang, W.Z.; Dai, B.; Zhao, G.Y.; Wu, H. Assessing the Performance of Green Mines via a Hesitant Fuzzy ORESTE–QUALIFLEX Method. Mathematics 2019, 7, 788. [CrossRef] 3. Wang, J.A.; Shang, X.C.; Ma, H.T. Investigation of catastrophic ground collapse in Xingtai gypsum mines in China. Int. J. Rock Mech. Min. 2008, 45, 1480–1499. [CrossRef] 4. Zhou, Y.J.; Li, M.; Xu, X.D.; Li, X.T.; Ma, Y.D.; Ma, Z. Research on catastrophic pillar instability in room and pillar gypsum mining. Sustainability 2018, 10, 3773. [CrossRef] 5. Liang, W.Z.; Zhao, G.Y.; Wu, H.; Chen, Y. Assessing the risk degree of goafs by employing hybrid TODIM method under uncertainty. B. Eng. Geol. Environ. 2019, 78, 3767–3782. [CrossRef] 6. Peng, K.; Liu„ Z.P.; Zou, Q.L.; Zhang, Z.Y.; Zhou, J.Q. Static and dynamic mechanical properties of granite from various burial depths. Rock Mech. Rock Eng. 2019, 52, 3545–3566. 7. Luo, S.Z.; Liang, W.Z.; Xing, L.N. Selection of mine development scheme based on similarity measure under fuzzy environment. Neural Comput. Appl. 2019, 32, 5255–5266. [CrossRef] 8. González-Nicieza, C.; Álvarez-Fernández, M.I.; Menéndez-Díaz, A.; Alvarez-Vigil, A.E. A comparative analysis of pillar design methods and its application to marble mines. Rock Mech. Rock Eng. 2006, 39, 421–444. [CrossRef] 9. Zhou, J.; Li, X.; Mitri, H.S. Comparative performance of six supervised learning methods for the development of models of hard rock pillar stability prediction. Nat. Hazards 2015, 79, 291–316. [CrossRef] 10. Lunder, P.J. Hard Rock Pillar Strength Estimation: An Applied Empirical Approach. Master’s Thesis, University of British Columbia, Vancouver, BC, Canada, 1994. 11. Cauvin, M.; Verdel, T.; Salmon, R. Modeling uncertainties in mining pillar stability analysis. Risk Anal. 2009, 29, 1371–1380. [CrossRef] 12. Mortazavi, A.; Hassani, F.P.; Shabani, M. A numerical investigation of rock pillar failure mechanism in underground openings. Comput. Geotech. 2009, 36, 691–697. [CrossRef] 13. Shnorhokian, S.; Mitri, H.S.; Moreau-Verlaan, L. Stability assessment of stope sequence scenarios in a diminishing ore pillar. Int. J. Rock Mech. Min. 2015, 74, 103–118. [CrossRef] 14. Elmo, D.; Stead, D. An integrated numerical modelling–discrete fracture network approach applied to the characterisation of rock mass strength of naturally fractured pillars. Rock Mech. Rock Eng. 2010, 43, 3–19. [CrossRef] 15. Li, L.C.; Tang, C.A.; Wang, S.Y.; Yu, J. A coupled thermo-hydrologic-mechanical damage model and associated application in a stability analysis on a rock pillar. Tunn. Undergr. Space Technol. 2013, 34, 38–53. [CrossRef] 16. Jaiswal, A.; Sharma, S.K.; Shrivastva, B.K. Numerical modeling study of asymmetry in the induced stresses over coal mine pillars with advancement of the goaf line. Int. J. Rock Mech. Min. 2004, 41, 859–864. [CrossRef] 17. Li, X.Y.; Kim, E.; Walton, G. A study of rock pillar behaviors in laboratory and in-situ scales using combined finite-discrete element method models. Int. J. Rock Mech. Min. 2019, 118, 21–32. [CrossRef] 18. Liang, W.Z.; Zhao, G.Y.; Wang, X.; Zhao, J.; Ma, C.D. Assessing the rockburst risk for deep shafts via distance-based multi-criteria decision making approaches with hesitant fuzzy information. Eng. Geol. 2019, 260, 105211. [CrossRef] 19. Deng, J.; Yue, Z.Q.; Tham, L.G.; Zhu, H.H. Pillar design by combining finite element methods, neural networks and reliability: A case study of the Feng Huangshan copper mine, China. Int. J. Rock Mech. Min. 2003, 40, 585–599. [CrossRef] 20. Griffiths, D.V.; Fenton, G.A.; Lemons, C.B. Probabilistic analysis of underground pillar stability. Int. J. Numer. Anal. Met. 2002, 26, 775–791. [CrossRef] 21. Amato, F.; Moscato, V.; Picariello, A.; Sperli’ì, G. Extreme events management using multimedia social networks. Future Gener. Comput. Syst. 2019, 94, 444–452. [CrossRef] 22. Tawadrous, A.S.; Katsabanis, P.D. Prediction of surface crown pillar stability using artificial neural networks. Int. J. Numer. Anal. Met. 2007, 31, 917–931. [CrossRef] 23. Wattimena, R.K. Predicting the stability of hard rock pillars using multinomial logistic regression. Int. J. Rock Mech. Min. 2014, 71, 33–40. [CrossRef] Mathematics 2020, 8, 765 17 of 17

24. Ding, H.X.; Li, G.H.; Dong, X.; Lin, Y. Prediction of pillar stability for underground mines using the stochastic gradient boosting technique. IEEE Access 2018, 6, 69253–69264. 25. Ghasemi, E.; Kalhori, H.; Bagherpour, R. Stability assessment of hard rock pillars using two intelligent classification techniques: A comparative study. Tunn. Undergr. Space Technol 2017, 68, 32–37. [CrossRef] 26. Sun, R.; Wang, G.Y.; Zhang, W.Y.; Hsu, L.T.; Ochieng, W.Y. A gradient boosting decision tree based GPS signal reception classification algorithm. Appl. Soft Comput. 2020, 86, 105942. [CrossRef] 27. Lombardo, L.; Cama, M.; Conoscenti, C.; Märker, M.; Rotigliano, E.J.N.H. Binary logistic regression versus stochastic gradient boosted decision trees in assessing landslide susceptibility for multiple-occurring landslide events: Application to the 2009 storm event in Messina (Sicily, southern Italy). Nat. Hazards 2015, 79, 1621–1648. [CrossRef] 28. Tama, B.A.; Rhee, K.H. An in-depth experimental study of anomaly detection using gradient boosted machine. Neural Comput. Appl. 2019, 31, 955–965. [CrossRef] 29. Sachdeva, S.; Bhatia, T.; Verma, A.K. GIS-based evolutionary optimized gradient boosted decision trees for forest fire susceptibility mapping. Nat. Hazards 2018, 92, 1399–1418. [CrossRef] 30. Kotsiantis, S.B. Decision trees: A recent overview. Artif. Intell. Rev. 2013, 39, 261–283. [CrossRef] 31. Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. 32. Ke, G.L.; Meng, Q.; Finley, T.; Wang, T.F.; Chen, W.; Ma, W.D.; Ye, Q.W.; Liu, T.Y. LightGBM: A highly efficient gradient boosting decision tree. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 3146–3154. 33. Esterhuizen, G.S.; Dolinar, D.R.; Ellenberger, J.L. Pillar strength in underground stone mines in the United States. Int. J. Rock Mech. Min. 2011, 48, 42–50. [CrossRef] 34. Friedman, J.H. Greedy function approximation: A gradient boosting machine. Ann. Stat. 2001, 29, 1189–1232. [CrossRef] 35. Rao, H.D.; Shi, X.Z.; Rodrigue, A.K.; Feng, J.J.; Xia, Y.C.; Elhoseny, M.; Yuan, X.H.; Gu, L.C. Feature selection based on artificial bee colony and gradient boosting decision tree. Appl. Soft Comput. 2019, 74, 634–642. [CrossRef] 36. Website of the Kaggle. Available online: https://www.kaggle.com/ (accessed on 3 May 2020). 37. Zeng, H.; Yang, C.; Zhang, H.; Wu, Z.H.; Zhang, J.M.; Dai, G.J.; Babiloni, F.; Kong, W.Z. A lightGBM-based EEG analysis method for driver mental states classification. Comput. Intel. Neurosc. 2019, 2019, 3761203. [CrossRef][PubMed] 38. Kodaz, H.; Öz¸sen, S.; Arslan, A.; Güne¸s,S. Medical application of information gain based artificial immune recognition system (AIRS): Diagnosis of thyroid disease. Expert Syst. Appl. 2009, 36, 3086–3092. [CrossRef] 39. Weng, T.Y.; Liu, W.Y.; Xiao, J. Supply chain sales forecasting based on lightGBM and LSTM combination model. Ind. Manage. Data Syst. 2019, 120, 265–279. [CrossRef] 40. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. 41. Website of the XGBoost Library. Available online: https://xgboost.readthedocs.io/en/latest/ (accessed on 3 May 2020). 42. Website of the LightGBM Library. Available online: https://lightgbm.readthedocs.io/en/latest/ (accessed on 3 May 2020). 43. Kumar, P. Machine Learning Quick Reference; Packt Publishing Ltd.: Birmingham, UK, 2019. 44. Jung, Y. Multiple predicting K-fold cross-validation for model selection. J. Nonparametr. Stat. 2018, 30, 197–215. [CrossRef] 45. Lunder, P.J.; Pakalnis, R.C. Determination of the strength of hard-rock mine pillars. CIM Bull. 1997, 90, 51–55.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).