Journal of Information Processing Vol.27 335–339 (Apr. 2019) [DOI: 10.2197/ipsjjip.27.335] Technical Note Low-cost Unsupervised Outlier Detection by Autoencoders with Robust Estimation Yoshinao Ishii1,a) Masaki Takanashi1,b) Received: October 18, 2018, Accepted: December 4, 2018 Abstract: Recently, an unsupervised outlier detection method based on the reconstruction errors of an autoencoder (AE), which achieves high detection accuracy, was proposed. This method, however, requires a high calculation cost because of its ensemble scheme. Therefore, in this paper, we propose a novel AE-based unsupervised method that can achieve high detection performance at a low calculation cost. Our method introduces the concept of robust estima- tion to appropriately restrict reconstruction capability and ensure robustness. Experimental results on several public benchmark datasets show that our method outperforms well-known outlier detection methods and at a low calculation cost. Keywords: outlier detection, unsupervised learning, autoencoder, robust estimation As a generalization of linear methods, an unsupervised out- 1. Introduction lier detection method based on the reconstruction errors of an In real datasets, it often happens that some samples have dif- autoencoder (AE) [5], which is a special type of multi-layer neu- ferent values or features from that of the majority (inliers). Such ral network has been proposed. We refer to this unsupervised samples are called outliers, and detecting outliers from the given method as the AE-based method. In the AE-based method, an AE data is generally called outlier detection. Since outlier detection is trained by constraining its reconstruction capability in order plays an important role in detecting anomalies in target systems, to prevent identity mapping, and the reconstruction errors of the in creating normal models from real datasets, and so on, outlier samples are taken as their outlier scores. An encoder is a nonlin- detection methods with high accuracy are required. In particu- ear mapping (regression) from an original data space to a feature lar, unsupervised outlier detection methods are important because space, a decoder is a nonlinear mapping from the feature space to real datasets are not often labeled. Some common unsupervised the original data space, and the reconstruction errors correspond methods include distance-based methods [1], [2], density-based to the residuals in the linear methods. Therefore, the AE-based methods [3], and linear methods [4]. These methods basically cal- method is regarded as a generalization of linear methods. The culate outlier scores that indicate the outlierness of each sample, AE-based method is capable of achieving high detection accu- and detect the outliers using their outlier scores. racy even for data having high dimensionality and nonlinear fea- Distance-based methods and density-based methods derive tures. However, the reconstruction errors of the outliers as well outlier scores using the distance and density ratio between sam- as the inliers are small if its reconstruction capability is prop- ples, respectively. Although both methods can obtain outlier erly restricted. This leads to a low detection accuracy or namely scores by considering the nonlinear features in the data, it is dif- overfitting. Due to this drawback, most of the AE’s reconstruc- ficult to achieve highly accurate detection in the case of high- tion error-based methods have been explored with regard to semi- dimensional data. In linear methods, we regress the high- supervised outlier detection requiring normal labels, and almost dimensional data to a low-dimensional linear model, and calcu- no unsupervised method has been proposed [6]. late the outlier scores from the residuals of the samples. These Recently, RandNet [6], an AE-based method that overcomes methods utilize the property that the squared residuals of outliers the aforementioned drawback and achieves high detection accu- tend to be large because the obtained regression model fits well racy was proposed. In that study, an ensemble scheme was in- to the inliers (majority) rather than the outliers. We could detect troduced and various randomly connected AEs (100 AEs were outliers with high accuracy from high-dimensional data. How- used in the experiment) with different structures and connection ever, the nonlinear data detection accuracy might be low in some densities were prepared. In RandNet, each AE is trained inde- cases because these methods cannot extract the nonlinear charac- pendently, and then the median of the reconstruction errors of teristics in the data. all the trained AEs are taken as outlier scores of the samples. This ensemble scheme constrains the reconstruction capability 1 Toyota Central R&D Laboratories, Inc., Nagakute, Aichi 480–1118, and improves robustness; therefore, RandNet achieves a high de- Japan a) [email protected] tection accuracy. However, its calculation cost is huge because b) [email protected] it is necessary to train a large number of AEs and pre-train each c 2019 Information Processing Society of Japan 335 Journal of Information Processing Vol.27 335–339 (Apr. 2019) AE layer-wise. In particular, when parallel computational envi- the LTS estimator avoids the adverse effects of outliers by not us- ronment is not available, the high computational cost needed to ing samples with higher squared residuals in the regression for independently train a large number of AEs becomes a significant parameter estimation. problem. 3. Proposed Method Robust Deep Autoencoders (RDAs) [7] have also been pro- posed in recent years as an AE-based method with a similar pur- Our proposed method uses a loss function that incorporates pose. Since RDA learns not to reconstruct samples considered as concepts of the LTS estimator in order to restrict the reconstruc- outliers it is more robust than a normal AE. RDA decomposes tion capability and ensure robustness of AEs. Specifically, our a data matrix X nonlinearly into a matrix LD whichiseasyto method utilizes a mini-batch learning approach to minimize the reconstruct and a sparse error matrix S which is difficult to re- loss function Lprop defined by the following equation. = + construct by an AE, subject to X LD S . Then, S can be B 1 identified with outliers by outlier detection. In order to perform L = w · e , (3) prop B i i this decomposition, RDA optimizes parameters of AE and S in a i=1 alternating manner. However, there is a drawback that the opti- = − 2 w where B denotes the mini-batch size, ei xi x¯i 2 holds, and i mization phase of S does not always optimize the whole objec- satisfies the following equation. ⎧ tive function. Therefore, it could take long time to converge or in ⎪ ⎨ 1(ei ≤ c) other words impose a high computational cost. Additionally, the wi = ⎪ (4) ⎩ 0(e > c). experimental results in Ref. [7] show that RDA highly depends i on its parameter choice. This is another drawback. Owing to Here, c denotes the αp-th percentile value of e = {e1,...,eB}, αp these drawbacks, RDA is hard to use as an unsupervised anomaly is a parameter, and wi is updated in every mini-batch learning. detection method for real datasets. Namely, in our proposed method, samples with higher recon- Therefore, in this paper, we propose a novel AE-based method struction error are not used for updating the parameters during that can achieve high detection performance at a low calculation batch learning since their losses are forced to 0. The final out- cost. Our method introduces the concept of robust estimation [8] lier scores of the samples in our method are ei obtained from the to restrict the reconstruction capability and ensure robustness. An trained model. The additional calculation cost of our proposed outline of the AEs and robust estimation is provided in Section 2. method compared to normal AE, due to this update process, is Section 3 discusses our proposed method, and Section 4 discusses only a derivation of the αp percentile value. the experimental results using real datasets. The use of the proposed loss function results in the following effects. First, the reconstruction errors of the inliers with w = 0 2. Autoencoders and Robust Estimation i tend to be small since the inliers with wi = 1 are trained to be An AE is a special type of multi-layer neural network in which reconstructed. This is because the inliers exist close to each other the number of nodes in the output layer is the same as that in the and make up a majority regardless of the values of wi = 0and input layer. Generally, the model parameters are trained to mini- wi = 1. Second, even if several outliers with wi = 1 are obtained mize the reconstruction error (loss function) L, which is defined in a training step, such outliers are less likely to be reconstructed by the following equation. than the inliers. This is because, in general, there are few similar w N samples around the outliers. The i of outliers is set to 0 in the 1 L = x − x¯ 2. (1) successive steps. As a result, the outliers are not reconstructed as N i i 2 i=1 the training progresses, and finally only the reconstruction errors In this Eq. (1), {xi}(i = 1,...,N), {x¯i}(i = 1,...,N), and N re- (i.e. outlier scores) of the outliers will be large. spectively denote the training data, the outputs from the AE cor- 4. Evaluations responding to each training data, and the number of samples. Robust estimation is a technique for regressing inliers to the 4.1 Experimental Settings model, avoiding the strong adverse effect of outliers present in In this paper, we utilize 15 types of datasets published in Out- the data. Robust estimation has been studied for a long time, and lier Detection DataSets (ODDS) [11], which is usually used as it is capable of overcoming the low robustness against outliers of the benchmark for outlier detection methods.