2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)

Research on ensemble model of based on

Yaning Han Yunyun Ma University of Chinese Academy of Technology and Engineering Center for Sciences Space Utilization, Chinese Academy of Technology and Engineering Center for Sciences Space Utilization, Chinese Academy of Beijing, China Sciences [email protected] Beijing, China [email protected] l Jinbo Wang Jianmin Wang Technology and Engineering Center for Technology and Engineering Center for Space Utilization, Chinese Academy of Space Utilization, Chinese Academy of Sciences Sciences Beijing, China Beijing, China [email protected] [email protected]

Abstract—In the fields of technology such as aerospace, machine (SVM) is a common classification anomaly detection is critical to the overall system. With the large model [18], however the performance of support vector increase in data volume and dimensions, the traditional machine is greatly affected by hyperparameters. For detection methods have great limitations, and thus anomaly inexperienced scholars, the hyperparameters cannot be detection algorithms based on have received reasonably combined and the results cannot be guaranteed. widespread attention. In this paper, based on autoencoder: standard autoencoder, denoising autoencoder, and sparse Neighbor-based methods assume that normal data has autoencoder, an ensemble detection model that can extract more relatively more neighbors than the outlier data. In [19], feature information is proposed. To make more use of these Breunig adopted a density-based (LOF) to feature information, inspired by the idea of pooling layer of the address this issue. In [20], Kriegel proposed the local outlier CNN, two feature fusion methods are proposed. Finally, the probabilities to detect outliers. However, the search for the experiment verifies that the result of this model is better than nearest neighbors prohibits such methods to be applied to the single autoencoder model. high-dimensional data due to the curse of dimensionality. High dimensional data will fool the algorithm to locate the Keywords—anomaly detection, autoencoder, ensemble improper neighbors, which will decrease the detection accuracy. I. INTRODUCTION The Internet and information technology are developing In practical applications, massive data and ultra-high rapidly. In the face of complex network environments, the dimensions have become important challenges for anomaly amount of data is growing at an extremely fast rate, and data detection. The most common method for working with data processing technology is also developing rapidly, which has dimensions is . The principal also caused more and more network security exceptions. component analysis (PCA) method is representative of the People build network security precautions from various levels, dimensionality reduction method. However, in the face of such as data packet encryption, access authentication, and ultra-high feature space, calculation of covariance matrix anomaly detection. Data packet encryption and access requires a lot of computational cost and time cost[5]. With the authentication are passive defenses, which can detect and development of neural networks, deep learning technology block most intrusions, but cannot detect and block intrusions has been received widespread attention. The accuracy of the inside the network system. Anomaly detection can analyze the neural network detection is relatively high, and accordingly information of the network system and can detect the intrusion the quality of the training data is higher. In [6], Li proposed a behavior inside the system. Therefore, compared to data deep learning approach for intrusion detection using a multi- packet encryption and access authentication, anomaly convolutional neural network (multi-CNN) fusion method. detection can more effectively ensure network security. According to the correlation, the feature data are divided into four parts, and then the one-dimensional feature data are Currently, scholars and engineers need to develop some converted into a grayscale graph. methods to find valuable information in massive data and hidden safety hazards in the network. has gained Although these traditional intelligent diagnosis methods the attention. It is based on statistical methods and uses such as Neural Network (NN) and Support vector machine machine learning, and other technologies (SVM) can obtain accurate diagnostic results, they have two to analyze massive data and extract useful information. inherent drawbacks: 1) They cannot generate features automatically, and fault features need to be designed by II. RELATED WORK experts manually. 2) In practical applications, since the signal So far, many experts and scholars have done a lot of has non-linear, non-Gaussian or other characteristics, a lot of research on anomaly detection problems and improved a lot pre-analysis and comparison processes are required[7]. With of anomaly detection methods. Common methods of anomaly the boosting of artificial intelligence, autoencoder provides an detection are clustering and support vector machine in data effective way to learn representative features, which mining. The clustering algorithm has fast detection speed, but overcomes the above drawbacks of manual feature extraction. the false detection rate is relatively high [17]. Support vector In [9], Sakurada proposed an Autoencoder-based outlier

978-1-7281-8913-0/20/$31.00 ©2020 IEEE 414 DOI 10.1109/QRS51102.2020.00060 detection method. As autoencoder can capture the nonlinear different from ordinary PCA. With the new feature h, the input correlations as well as the linear correlations, this method has x can be reconstructed, that is, the decoding process: better detection performance than PCA-based. However, when applying the autoencoder to image outlier detection, the  xgWhbc    detection performance is not always very palatable. This is dd because the single autoencoder fails to fully capture the correlations among features, especially in the high- We want the reconstructed xc and x to be as same as dimensional datasets, and resulting in poor detection accuracy. possible, and this model can be trained using a loss function Therefore, the autoencoder based on the ensemble came into that minimizes negative log-likelihood: being. In [10], Chen proposed a novel image outlier detection method by combining autoencoder with Adaboost (ADAE). By ensembling many weak , the method can  Loss log P x | xc    better capture the statistical correlations among the features of normal data than the single autoencoder. Therefore, the P is the loss function of x and xc . Sometimes we add proposed ADAE is able to determine the outliers efficiently. more constraints or penalties to autoencoder, such as This paper is based on ensemble multiple autoencoders for denoising autoencoder and sparse autoencoder. Because it is feature extraction. not meaningful to simply reconstruct the original input most of the time, we hope that the autoencoder can capture more III. PROPOSED MODEL valuable information of the original input in the case of approximate reconstruction of the original input. A. Standard Autoencoder (AE) Autoencoder is an unsupervised artificial neural network B. Denoising Autoencoder (DAE) that is trained for automatic feature extraction Through Due to the existence of noise and outliers in the actual data, training, the weight and bias parameters of each hidden layer it is still difficult to learn the robust sample features using the can be adjusted under the optimal output to obtain different above method to improve the applicability of the autoencoder. representations of the input (each layer represents a In order to force the hidden layer to acquire more robust representation ), So-called higher-order features. Studies show features, a method of introducing noise to reconstruct the that automatic learning methods can greatly improve the original input signal has been used to train the autoencoder. accuracy, and thus achieve better classification results than that is the denoising autoencoder(DAE). other traditional classification algorithms. This method is called an autoencoder. The structure of the autoencoder is x x h y shown in Fig. 1.

x x h xc x 1 1 f g

h x2 1 x2

... x3 x3

Fig. 2. The structure of DAE ... h ... Vhidd As shown in Fig. 2, after introducing random noise, the

x xqxxx ~(|)~(|(|) x x input signal follows the distribution . the Vin Vin signal that reaches the hidden layer(4). Fig. 1. The structure of AE The autoencoder encodes the input x to obtain a new  hfxfWxb ()x) ffWx Wx  b    feature h. Assume that the original input x can be ee reconstructed from the new feature h. The encoding process is as follows:  yg ()hh g Wdd b  

 hfWxb ee  then (5) reconstructs the input data, and finally trains the parameters to minimize the error by (6). We is a weight vector and be is a constant. Like the neural network structure, its encoding is a linear combination 2 followed by a non-linear activation function. If there is no  LxyH (, ) yx    non-linear activation function, then the autoencoder is no

415 so the model output y gradually approaches the real EAE integrates three types of autoencoders above: AE, DAE and SAE in Fig. 3. Each autoencoder has the functions inSXW x  of dimensionality reduction and automatic feature extraction. For CNN, the convolutional layer is mainly used to extract C. Sparse Autoencoder (SAE) features. This paper uses autoencoder to extract features. The SAE is also based on AE. Make some restrictions on the size and quality of the features by different convolution hidden layer, such as reducing the number of neurons, the kernels are different, so the pooling layer is introduced. neural network will compress the data and extract features. Essentially, the pooling layer is a feature fusion between When the penalty term is added to make the neuron inactive multiple channels. Through the pooling layer, not only can the in most cases, the network can learn very interesting edge feature map be reduced, but also the amount of calculation can features. be reduced by reducing network parameters, and overfitting can be controlled to a certain extent. Similar to the pooling Firstly, the objective function of the SAE is given as (7). layer of a convolutional neural network, after the feature is extracted by the autoencoder, we also propose and define three 11m 2 feature fusion methods: feature non-fusion or single feature  §·()ii ()    JWbsparse (,) ¦¨¸ hxyW,1 b  PP2(SF), max feature (MF) and average feature (AF) in Fig. 4. m i 1 ©¹2 f1 f2 f3

L1 ssll1  O ()l 2    P1 ¦¦¦ 4ji 2 lij 11 1

s2  ˆ    PKL2 EUU¦ ý j j 1 F = {f1,f2,f3} (9) is a new penalty for the hidden layer. s2 represents the F number of neurons in the hidden layer. Uˆ j represents the average activation degree of hidden layer neuron j for all training data.  Fig. 4. Feature fusion m  Uˆ ªºax(2) (i )    As shown in Fig. 4, assuming three encoders get three jj¦ ¬¼ i 1 feature maps:f1, f2 and f3, all of which are nm* . For each feature map, ij,(0dd i n  1,0 d j d m  1) SAE can restrict the activation degree of each neuron node by adding sparseness restrictions, making the fault represents row and column respectively. The feature fusion features extracted from the original input data more robust methods are as follows: and less affected by interference factors such as noise, so to a SF: No processing the feature map as (11). certain extent Improve accuracy.  F(,)ij f(,) ij or f(,) ij or f(,) ij    D. The proposed Ensemble Autoencoder(EAE) for anomaly 123 detection MF: Take the largest feature at the same location as input (12).

feature extraction  F(,)ij max(f(,),123 ij f(,), ij f(,)) ij    AE DAE SAE AF: Mean of the same location feature as (13). m is the dimension of these feature maps. feature fusion f(,)+f(,)ij ij f(,) ij  F(ij , ) 12 3   classification m

Finally get F() and input to the classifier. The overall output  experimental scheme is shown in Fig. 5. (SVM: Support Vector Machine, LR: Logistics Regression, KNN: k-Nearest Fig. 3. The flowchart of EAE Neighbor) This section introduces the proposed EAE method for anomaly detection in detail, and proposes the feature fusion method by analogy to the pooling layer of convolutional neural network(CNN).

416 ACKNOWLEDGMENT dataset The relevant research done in this paper are supported by the Equipment Pre-Research Field Fund, China feature map AE DAE SAE (No.61400020401).

REFERENCES feature fusion SF1 SF2 SF3 MF AF [1] Yang, Q., et al, "Prediction of aptamer-protein interacting pairs based on sparse autoencoder feature extraction and an ensemble classifier," Mathematical Biosciences 311: 103-108, 2019. classifier SVM LR KNN [2] Shi, ., et al, "Boosting sparsity-induced autoencoder: A novel sparse feature ensemble learning for image classification," International Journal of Advanced Robotic Systems 16(3), 2019. output [3] Principi, E., et al, "Unsupervised Electric Motor Fault Detection by Using Deep Autoencoders," Ieee-Caa Journal of Automatica Sinica 6(2): 441-451, 2019濁 Fig. 5. Experimental schema [4] Li, Y., et al, "Robust detection for network intrusion of industrial IoT based on multi-CNN fusion." Measurement 154, 2020. IV. EVALUATIONS AND RESULTS [5] Zhang, B., et al, "Network Intrusion Detection Based on Stacked The proposed model is validated on some credit card Sparse Autoencoder and Binary Tree Ensemble Method," 2018 Ieee detection dataset with three autoencoders and three classifiers. International Conference on Communications Workshops, 2018. In order to balance the positive and negative proportions, the [6] Li, Y., et al, "Robust detection for network intrusion of industrial IoT based on multi-CNN fusion," Measurement 154, 2020. positive samples are down-sampled and the negative samples [7] Zhang, Y., et al, "Intelligent fault diagnosis of rotating machinery using are up-sampled. The data set is randomly divided into the train a new ensemble deep auto-encoder method," Measurement 151, 2020. set and the test set according. normalize the train data and [8] K. Tidriri, N. Chatti, S. Verron, T. Tiplica, "Bridging data-driven and adjust network parameters to make the autoencoder reach the model-based approaches for process fault diagnosis and health optimal state. The results are shown in TABLE I. monitoring: A review of researches and future challenges," Annual Reviews in Control, 42 (2016) 63-81. TABLE I. RESULTS [9] M. Sakurada, T. Yairi, "Anomaly detection using autoencoders with non-linear dimensionality reduction," Proceedings of the MLSDA Accuracy 2014 2ndWorkshop on Machine Learning for Sensory Data Analysis,

SVM LR KNN ACM, 2014. [10] Chen, Z., et al, "Evolutionary multi-objective optimization based SF1 0.784 0.775 0.79 ensemble autoencoders for image outlier detection," Neurocomputing SF2 0.796 0.817 0.809 309: 192-200, 2018. [11] Chen, L., et al, "Multiperiod-Ahead Wind Speed Forecasting Using SF3 0.811 0.803 0.812 Deep Neural Architecture and Ensemble Learning," Mathematical Problems in Engineering, 2019. MF 0.84 0.848 0.853 [12] Hu, J., et al, "An Efficient and Robust Unsupervised Anomaly AF 0.845 0.854 0.837 Detection Method Using Ensemble Random Projection in Surveillance Videos," Sensors 19(19), 2019.

[13] Ibrahim, A. K., et al, "Classification of red hind grouper call types As TABLE I, the accuracy of MF and AF are better than using random ensemble of stacked autoencoders," Journal of the that of SF. It shows that the ensemble model with feature Acoustical Society of America 146(4): 2155-2162, 2019. [14] Khan, S. S. and B. Taati, "Detecting unseen falls from wearable devices fusion extracts and uses more feature information than the using channel-wise ensemble of autoencoders," Expert Systems with single model (SF can be regarded as the single model). The Applications 87: 280-290, 2017. generalization performance is better. In addition, the results of [15] Wen, L., et al, "A New Snapshot Ensemble Convolutional Neural SF2 and SF3 are better than SF1, with less loss of feature Network for Fault Diagnosis," Ieee Access 7: 32037-32047, 2019. information. This proves that the addition of penalty terms and [16] Xu, F., et al, "Roller bearing fault diagnosis using stacked denoising denoising operation has helped improve the results. Because autoencoder in deep learning and Gath-Geva clustering algorithm all the autoencoders use a single hidden layer, the accuracy without principal component analysis and data label," Applied Soft results are not very high. If multiple hidden layers are used, Computing 73: 898-913, 2018. the result may be higher. [17] I. Chairunnisa, Lukas, and H. D. Widiputra, "Clustering base intrusion detection for network profiling using k-means, ECM and k-nearest neighbor algorithms," Konferensi Nasional Sistem dan Informatika V. CONCLUSION 2009, nov 2009. In anomaly detection, this paper proposes an ensemble [18] Y. Bengio, "Learning deep architectures for AI," Foundations and model based on autoencoder, and uses some data to conduct Trends in Machine Learning, 2009,2(1): 1–127. experiments. The verification results show that the results of [19] M. M. Breunig, H.-P. Kriegel, R. T. Ng, J. Sander, "Lof: identifying the ensemble model are better than the single model, and more density-based local outliers," ACM Sigmod Record, Vol. 29, ACM, feature information is used. Autoencoder has better automatic 2000, pp.93–104. feature extraction capability and can be widely used in more [20] H.-P. Kriegel, P. Kr¨ oger, E. Schubert, A. Zimek, "LoOP: local outlier prob-abilities," Proceedings of the 18th ACM Conference on fields. Information and Knowledge Management, 2009, pp. 1649–1652.

417