DOCSLIB.ORG

A Review of Local Outlier Factor Algorithms for Outlier Detection in Big Data Streams

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

big data and cognitive computing Review A Review of Local Outlier Factor Algorithms for Outlier Detection in Big Data Streams Omar Alghushairy 1,2,* , Raed Alsini 1,3 , Terence Soule 1 and Xiaogang Ma 1,* 1 Department of Computer Science, University of Idaho, Moscow, ID 83844, USA; [email protected] (R.A.); [email protected] (T.S.) 2 College of Computer Science and Engineering, University of Jeddah, Jeddah 23890, Saudi Arabia 3 Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia * Correspondence: [email protected] (O.A.); [email protected] (X.M.) Abstract: Outlier detection is a statistical procedure that aims to find suspicious events or items that are different from the normal form of a dataset. It has drawn considerable interest in the field of data mining and machine learning. Outlier detection is important in many applications, including fraud detection in credit card transactions and network intrusion detection. There are two general types of outlier detection: global and local. Global outliers fall outside the normal range for an entire dataset, whereas local outliers may fall within the normal range for the entire dataset, but outside the normal range for the surrounding data points. This paper addresses local outlier detection. The best-known technique for local outlier detection is the Local Outlier Factor (LOF), a density-based technique. There are many LOF algorithms for a static data environment; however, these algorithms cannot be applied directly to data streams, which are an important type of big data. In general, local outlier detection algorithms for data streams are still deficient and better algorithms need to be developed that can effectively analyze the high velocity of data streams to detect local outliers. This paper presents a literature review of local outlier detection algorithms in static and stream environments, with an emphasis on LOF algorithms. It collects and categorizes existing local outlier detection algorithms and analyzes their characteristics. Furthermore, the paper discusses the advantages and Citation: Alghushairy, O.; Alsini, R.; limitations of those algorithms and proposes several promising directions for developing improved Soule, T.; Ma, X. A Review of Local local outlier detection methods for data streams. Outlier Factor Algorithms for Outlier Detection in Big Data Streams. Big Keywords: outlier detection; data science; local outlier factor; genetic algorithm; stream data mining Data Cogn. Comput. 2021, 5, 1. https://doi.org/10.3390/bdcc 5010001 1. Introduction Received: 5 November 2020 Accepted: 23 December 2020 Outlier detection is the term used for anomaly detection, fraud detection, and novelty Published: 29 December 2020 detection. The purpose of outlier detection is to detect rare events or unusual activities that differ from the majority of data points in a dataset [1]. Recently, outlier detection has Publisher’s Note: MDPI stays neu- become an important problem in many applications such as in health care, fraud transaction tral with regard to jurisdictional clai- detection for credit cards, and intrusion detection in computer networks. Therefore, many ms in published maps and institutio- algorithms have been proposed and developed to detect outliers. However, most of these nal affiliations. algorithms are designed for a static data environment and are difficult to apply to streams of data, which represent a significant number of important application areas. Outlier detection can be viewed as a specific application of generalized data mining. The process of data mining involves two steps: data processing and data mining. The aim Copyright: © 2020 by the authors. Li- censee MDPI, Basel, Switzerland. of data processing is to create data in good form, i.e., the outliers are removed. Data mining This article is an open access article is the next step, which involves the discovery and understanding of the behavior of the distributed under the terms and con- data and extracting this information by machine learning [2]. Outlier detection (or data ditions of the Creative Commons At- cleaning) is an important step in data processing because, if an outlier data point is used tribution (CC BY) license (https:// during data mining, it is likely to lead to inaccurate outputs [3]. For example, a traffic creativecommons.org/licenses/by/ pattern of outliers in the network might mean the data has been sent out by a hacked 4.0/). computer to an unauthorized device [4]; if the outliers cannot be detected and removed, Big Data Cogn. Comput. 2021, 5, 1. https://doi.org/10.3390/bdcc5010001 https://www.mdpi.com/journal/bdcc Big Data Cogn. Comput. 2021, 5, x FOR PEER REVIEW 2 of 25 Big Data Cogn. Comput. 2021, 5, 1 2 of 24 computer to an unauthorized device [4]; if the outliers cannot be detected and removed, a machine learning technique applied to the data is likely to be misled. Additionally, in manya machine cases, learning the goal technique of data mining applied is to find the data the outliers. is likely For to be example, misled. an Additionally, outlier image in inmany magnetic cases, resonance the goal of imaging data mining could is mean to find the the presence outliers. of For malignant example, tumors an outlier [5]. image An out- in liermagnetic reading resonance from an imagingaircraft sensor could may mean indicate the presence a fault of in malignant some components tumors [ 5of]. the An aircraft outlier [6].reading from an aircraft sensor may indicate a fault in some components of the aircraft [6]. Outlier detection (also known as anomaly detection) is split into twotwo types,types, globalglobal and local detection. For a global outlier, outlieroutlier detection considers all data points, and the data point pt is consideredconsidered an outlieroutlier if itit isis farfar awayaway fromfrom allall otherother datadata pointspoints [[7].7]. TheThe local outlier detection covers a small subset ofof datadata pointspoints atat aa timetime (Figure(Figure1 ).1). AA local local outlier is based on the probability of data point pt being an outlier as compared to its local neighborhood, which is measured by the kk-Nearest-Nearest NeighborsNeighbors ((kkNN)NN) algorithmalgorithm [[8].8]. Figure 1. The types of outliers, where grey points are global outliers, and the red point is a local outlier. Figure 1. The types of outliers, where grey points are global outliers, and the red point is a local outlier. Recently, many studies have been published in the area of outlier detection, and thereRecently, is a need many for these studies studies have to been be reviewedpublished to in understand the area of outlier the depth detection, of the and field there and isthe a need most for promising these studies areas to for be additional reviewed to research. understand This the review depth is of focused the field on and local the outlier most promisingdetection algorithmsareas for additional in data stream research. environments. This review is This focused review on islocal distinguished outlier detection from previousalgorithms reviews in data as stream it specifically environments. covers This the Localreview Outlier is distinguished Factor (LOF) from algorithms previous re- in viewsdata stream as it specifically environments covers and reviewsthe Local the Outlier most recent,Factor state-of-the-art(LOF) algorithms techniques. in data stream In this paper,environments different and methods reviews and the algorithms most recent, for stat locale-of-the-art outlier detection techniques. are summarized.In this paper, Thedif- ferentlimitations methods and and challenges algorithms of the for LOFlocal andoutlier other detection local outlier are summarized. detection algorithms The limitations in a andstream challenges environment of the are LOF analyzed. and other Moreover, local outlier new detection methods algorithms for the detection in a stream of an envi- LOF ronmentscore in a are data analyzed. stream are Moreover, proposed. new This methods literature for reviewthe detection will greatly of an LOF benefit score researchers in a data streamin the field are proposed. of local outlier This detectionliterature becausereview wi itll provides greatly benefit a comprehensive researchers explanation in the field ofof localthe features outlier ofdetection each algorithm because and it provides shows their a comprehensive strengths and explanation weaknesses, of particularly the features in of a eachdata streamalgorithm environment. and shows The their paper strengths has five and remaining weaknesses, parts: particularly the second sectionin a data is Relatedstream Worksenvironment. and Background; The paper has the five third remaining section is part Literatures: the second Review section Methodology; is Related Works the fourth and section is Literature Review Results; the fifth section is Analysis and Discussion; and the sixth section is the Conclusion. Big Data Cogn. Comput. 2021, 5, 1 3 of 24 2. Related Works and Background Many methods and algorithms have been developed to detect outliers in datasets. Most of these methods and algorithms have been developed for specific applications and domains. Outlier detection has been the focus of many review and survey papers, as well as several books. Patcha et al. and Snyder [9,10] published a survey of outlier detection techniques for intrusion detection. Markou and Singh [11,12] provided a review of outlier detection methods that use statistical approaches and neural networks. A comprehensive
Recommended publications
  • A Review on Outlier/Anomaly Detection in Time Series Data

    A Review on Outlier/Anomaly Detection in Time Series Data

    A review on outlier/anomaly detection in time series data ANE BLÁZQUEZ-GARCÍA and ANGEL CONDE, Ikerlan Technology Research Centre, Basque Research and Technology Alliance (BRTA), Spain USUE MORI, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain JOSE A. LOZANO, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain and Basque Center for Applied Mathematics (BCAM), Spain Recent advances in technology have brought major breakthroughs in data collection, enabling a large amount of data to be gathered over time and thus generating time series. Mining this data has become an important task for researchers and practitioners in the past few years, including the detection of outliers or anomalies that may represent errors or events of interest. This review aims to provide a structured and comprehensive state-of-the-art on outlier detection techniques in the context of time series. To this end, a taxonomy is presented based on the main aspects that characterize an outlier detection technique. Additional Key Words and Phrases: Outlier detection, anomaly detection, time series, data mining, taxonomy, software 1 INTRODUCTION Recent advances in technology allow us to collect a large amount of data over time in diverse research areas. Observations that have been recorded in an orderly fashion and which are correlated in time constitute a time series. Time series data mining aims to extract all meaningful knowledge from this data, and several mining tasks (e.g., classification, clustering, forecasting, and outlier detection) have been considered in the literature [Esling and Agon 2012; Fu 2011; Ratanamahatana et al.
  • Outlier Identification.Pdf

    Outlier Identification.Pdf

    STATGRAPHICS – Rev. 7/6/2009 Outlier Identification Summary The Outlier Identification procedure is designed to help determine whether or not a sample of n numeric observations contains outliers. By “outlier”, we mean an observation that does not come from the same distribution as the rest of the sample. Both graphical methods and formal statistical tests are included. The procedure will also save a column back to the datasheet identifying the outliers in a form that can be used in the Select field on other data input dialog boxes. Sample StatFolio: outlier.sgp Sample Data: The file bodytemp.sgd file contains data describing the body temperature of a sample of n = 130 people. It was obtained from the Journal of Statistical Education Data Archive (www.amstat.org/publications/jse/jse_data_archive.html) and originally appeared in the Journal of the American Medical Association. The first 20 rows of the file are shown below. Temperature Gender Heart Rate 98.4 Male 84 98.4 Male 82 98.2 Female 65 97.8 Female 71 98 Male 78 97.9 Male 72 99 Female 79 98.5 Male 68 98.8 Female 64 98 Male 67 97.4 Male 78 98.8 Male 78 99.5 Male 75 98 Female 73 100.8 Female 77 97.1 Male 75 98 Male 71 98.7 Female 72 98.9 Male 80 99 Male 75 2009 by StatPoint Technologies, Inc. Outlier Identification - 1 STATGRAPHICS – Rev. 7/6/2009 Data Input The data to be analyzed consist of a single numeric column containing n = 2 or more observations.
  • A Machine Learning Approach to Outlier Detection and Imputation of Missing Data1

    A Machine Learning Approach to Outlier Detection and Imputation of Missing Data1

    Ninth IFC Conference on “Are post-crisis statistical initiatives completed?” Basel, 30-31 August 2018 A machine learning approach to outlier detection and imputation of missing data1 Nicola Benatti, European Central Bank 1 This paper was prepared for the meeting. The views expressed are those of the authors and do not necessarily reflect the views of the BIS, the IFC or the central banks and other institutions represented at the meeting. A machine learning approach to outlier detection and imputation of missing data Nicola Benatti In the era of ready-to-go analysis of high-dimensional datasets, data quality is essential for economists to guarantee robust results. Traditional techniques for outlier detection tend to exclude the tails of distributions and ignore the data generation processes of specific datasets. At the same time, multiple imputation of missing values is traditionally an iterative process based on linear estimations, implying the use of simplified data generation models. In this paper I propose the use of common machine learning algorithms (i.e. boosted trees, cross validation and cluster analysis) to determine the data generation models of a firm-level dataset in order to detect outliers and impute missing values. Keywords: machine learning, outlier detection, imputation, firm data JEL classification: C81, C55, C53, D22 Contents A machine learning approach to outlier detection and imputation of missing data ... 1 Introduction ..............................................................................................................................................
  • Fastlof: an Expectation-Maximization Based Local Outlier Detection Algorithm

    Fastlof: an Expectation-Maximization Based Local Outlier Detection Algorithm

    FastLOF: An Expectation-Maximization based Local Outlier Detection Algorithm Markus Goldstein German Research Center for Artificial Intelligence (DFKI), Kaiserslautern www.dfki.de [email protected] Introduction Performance Improvement Attempts Anomaly detection finds outliers in data sets which Space partitioning algorithms (e.g. search trees): Require time to build the tree • • – only occur very rarely in the data and structure and can be slow when having many dimensions – their features significantly deviate from the normal data Locality Sensitive Hashing (LSH): Approximates neighbors well in dense areas but • Three different anomaly detection setups exist [4]: performs poor for outliers • 1. Supervised anomaly detection (labeled training and test set) FastLOF Idea: Estimate the nearest neighbors for dense areas approximately and compute • 2. Semi-supervised anomaly detection exact neighbors for sparse areas (training with normal data only and labeled test set) Expectation step: Find some (approximately correct) neighbors and estimate • LRD/LOF based on them Maximization step: For promising candidates (LOF > θ ), find better neighbors • 3. Unsupervised anomaly detection (one data set without any labels) Algorithm 1 The FastLOF algorithm 1: Input 2: D = d1,...,dn: data set with N instances 3: c: chunk size (e.g. √N) 4: θ: threshold for LOF In this work, we present an unsupervised algorithm which scores instances in a 5: k: number of nearest neighbors • Output given data set according to their outlierliness 6: 7: LOF = lof1,...,lofn:
  • Incremental Local Outlier Detection for Data Streams

    Incremental Local Outlier Detection for Data Streams

    IEEE Symposium on Computational Intelligence and Data Mining (CIDM), April 2007 Incremental Local Outlier Detection for Data Streams Dragoljub Pokrajac Aleksandar Lazarevic Longin Jan Latecki CIS Dept. and AMRC United Tech. Research Center CIS Department. Delaware State University 411 Silver Lane, MS 129-15 Temple University Dover DE 19901 East Hartford, CT 06108, USA Philadelphia, PA 19122 Abstract. Outlier detection has recently become an important have labeled data, which can be extremely time consuming for problem in many industrial and financial applications. This real life applications, and (2) inability to detect new types of problem is further complicated by the fact that in many cases, rare events. In contrast, unsupervised learning methods outliers have to be detected from data streams that arrive at an typically do not require labeled data and detect outliers as data enormous pace. In this paper, an incremental LOF (Local Outlier points that are very different from the normal (majority) data Factor) algorithm, appropriate for detecting outliers in data streams, is proposed. The proposed incremental LOF algorithm based on some measure [3]. These methods are typically provides equivalent detection performance as the iterated static called outlier/anomaly detection techniques, and their success LOF algorithm (applied after insertion of each data record), depends on the choice of similarity measures, feature selection while requiring significantly less computational time. In addition, and weighting, etc. They have the advantage of detecting new the incremental LOF algorithm also dynamically updates the types of rare events as deviations from normal behavior, but profiles of data points. This is a very important property, since on the other hand they suffer from a possible high rate of false data profiles may change over time.
  • Image Anomaly Detection Using Normal Data Only by Latent Space Resampling

    Image Anomaly Detection Using Normal Data Only by Latent Space Resampling

    applied sciences Article Image Anomaly Detection Using Normal Data Only by Latent Space Resampling Lu Wang 1 , Dongkai Zhang 1 , Jiahao Guo 1 and Yuexing Han 1,2,* 1 School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China; [email protected] (L.W.); [email protected] (D.Z.); [email protected] (J.G.) 2 Shanghai Institute for Advanced Communication and Data Science, Shanghai University, 99 Shangda Road, Shanghai 200444, China * Correspondence: [email protected] Received: 29 September 2020; Accepted: 30 November 2020; Published: 3 December 2020 Abstract: Detecting image anomalies automatically in industrial scenarios can improve economic efficiency, but the scarcity of anomalous samples increases the challenge of the task. Recently, autoencoder has been widely used in image anomaly detection without using anomalous images during training. However, it is hard to determine the proper dimensionality of the latent space, and it often leads to unwanted reconstructions of the anomalous parts. To solve this problem, we propose a novel method based on the autoencoder. In this method, the latent space of the autoencoder is estimated using a discrete probability model. With the estimated probability model, the anomalous components in the latent space can be well excluded and undesirable reconstruction of the anomalous parts can be avoided. Specifically, we first adopt VQ-VAE as the reconstruction model to get a discrete latent space of normal samples. Then, PixelSail, a deep autoregressive model, is used to estimate the probability model of the discrete latent space. In the detection stage, the autoregressive model will determine the parts that deviate from the normal distribution in the input latent space.
  • Unsupervised Network Anomaly Detection Johan Mazel

    Unsupervised Network Anomaly Detection Johan Mazel

    Unsupervised network anomaly detection Johan Mazel To cite this version: Johan Mazel. Unsupervised network anomaly detection. Networking and Internet Architecture [cs.NI]. INSA de Toulouse, 2011. English. tel-00667654 HAL Id: tel-00667654 https://tel.archives-ouvertes.fr/tel-00667654 Submitted on 8 Feb 2012 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. 5)µ4& &OWVFEFMPCUFOUJPOEV %0$503"5%&-6/*7&34*5²%&506-064& %ÏMJWSÏQBS Institut National des Sciences Appliquées de Toulouse (INSA de Toulouse) $PUVUFMMFJOUFSOBUJPOBMFBWFD 1SÏTFOUÏFFUTPVUFOVFQBS Mazel Johan -F lundi 19 décembre 2011 5J tre : Unsupervised network anomaly detection ED MITT : Domaine STIC : Réseaux, Télécoms, Systèmes et Architecture 6OJUÏEFSFDIFSDIF LAAS-CNRS %JSFDUFVS T EFʾÒTF Owezarski Philippe Labit Yann 3BQQPSUFVST Festor Olivier Leduc Guy Fukuda Kensuke "VUSF T NFNCSF T EVKVSZ Chassot Christophe Vaton Sandrine Acknowledgements I would like to first thank my PhD advisors for their help, support and for letting me lead my research toward my own direction. Their inputs and comments along the stages of this thesis have been highly valuable. I want to especially thank them for having been able to dive into my work without drowning, and then, provide me useful remarks.
  • Effects of Outliers with an Outlier

    Effects of Outliers with an Outlier

    • The mean is a good measure to use to describe data that are close in value. • The median more accurately describes data Effects of Outliers with an outlier. • The mode is a good measure to use when you have categorical data; for example, if each student records his or her favorite color, the color (a category) listed most often is the mode of the data. In the data set below, the value 12 is much less than Additional Example 2: Exploring the Effects of the other values in the set. An extreme value such as Outliers on Measures of Central Tendency this is called an outlier. The data shows Sara’s scores for the last 5 35, 38, 27, 12, 30, 41, 31, 35 math tests: 88, 90, 55, 94, and 89. Identify the outlier in the data set. Then determine how the outlier affects the mean, median, and x mode of the data. x x xx x x x 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 55, 88, 89, 90, 94 outlier 55 1 Additional Example 2 Continued Additional Example 2 Continued With the Outlier Without the Outlier 55, 88, 89, 90, 94 55, 88, 89, 90, 94 outlier 55 mean: median: mode: mean: median: mode: 55+88+89+90+94 = 416 55, 88, 89, 90, 94 88+89+90+94 = 361 88, 89,+ 90, 94 416 5 = 83.2 361 4 = 90.25 2 = 89.5 The mean is 83.2. The median is 89. There is no mode.
  • How Significant Is a Boxplot Outlier?

    How Significant Is a Boxplot Outlier?

    Journal of Statistics Education, Volume 19, Number 2(2011) How Significant Is A Boxplot Outlier? Robert Dawson Saint Mary’s University Journal of Statistics Education Volume 19, Number 2(2011), www.amstat.org/publications/jse/v19n2/dawson.pdf Copyright © 2011 by Robert Dawson all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the author and advance notification of the editor. Key Words: Boxplot; Outlier; Significance; Spreadsheet; Simulation. Abstract It is common to consider Tukey’s schematic (“full”) boxplot as an informal test for the existence of outliers. While the procedure is useful, it should be used with caution, as at least 30% of samples from a normally-distributed population of any size will be flagged as containing an outlier, while for small samples (N<10) even extreme outliers indicate little. This fact is most easily seen using a simulation, which ideally students should perform for themselves. The majority of students who learn about boxplots are not familiar with the tools (such as R) that upper-level students might use for such a simulation. This article shows how, with appropriate guidance, a class can use a spreadsheet such as Excel to explore this issue. 1. Introduction Exploratory data analysis suddenly got a lot cooler on February 4, 2009, when a boxplot was featured in Randall Munroe's popular Webcomic xkcd. 1 Journal of Statistics Education, Volume 19, Number 2(2011) Figure 1 “Boyfriend” (Feb. 4 2009, http://xkcd.com/539/) Used by permission of Randall Munroe But is Megan justified in claiming “statistical significance”? We shall explore this intriguing question using Microsoft Excel.
  • Lecture 14 Testing for Kurtosis

    Lecture 14 Testing for Kurtosis

    9/8/2016 CHE384, From Data to Decisions: Measurement, Kurtosis Uncertainty, Analysis, and Modeling • For any distribution, the kurtosis (sometimes Lecture 14 called the excess kurtosis) is defined as Testing for Kurtosis 3 (old notation = ) • For a unimodal, symmetric distribution, Chris A. Mack – a positive kurtosis means “heavy tails” and a more Adjunct Associate Professor peaked center compared to a normal distribution – a negative kurtosis means “light tails” and a more spread center compared to a normal distribution http://www.lithoguru.com/scientist/statistics/ © Chris Mack, 2016Data to Decisions 1 © Chris Mack, 2016Data to Decisions 2 Kurtosis Examples One Impact of Excess Kurtosis • For the Student’s t • For a normal distribution, the sample distribution, the variance will have an expected value of s2, excess kurtosis is and a variance of 6 2 4 1 for DF > 4 ( for DF ≤ 4 the kurtosis is infinite) • For a distribution with excess kurtosis • For a uniform 2 1 1 distribution, 1 2 © Chris Mack, 2016Data to Decisions 3 © Chris Mack, 2016Data to Decisions 4 Sample Kurtosis Sample Kurtosis • For a sample of size n, the sample kurtosis is • An unbiased estimator of the sample excess 1 kurtosis is ∑ ̅ 1 3 3 1 6 1 2 3 ∑ ̅ Standard Error: • For large n, the sampling distribution of 1 24 2 1 approaches Normal with mean 0 and variance 2 1 of 24/n 3 5 • For small samples, this estimator is biased D. N. Joanes and C. A. Gill, “Comparing Measures of Sample Skewness and Kurtosis”, The Statistician, 47(1),183–189 (1998).
  • Machine Learning and Extremes for Anomaly Detection — Apprentissage Automatique Et Extrêmes Pour La Détection D’Anomalies

    Machine Learning and Extremes for Anomaly Detection — Apprentissage Automatique Et Extrêmes Pour La Détection D’Anomalies

    École Doctorale ED130 “Informatique, télécommunications et électronique de Paris” Machine Learning and Extremes for Anomaly Detection — Apprentissage Automatique et Extrêmes pour la Détection d’Anomalies Thèse pour obtenir le grade de docteur délivré par TELECOM PARISTECH Spécialité “Signal et Images” présentée et soutenue publiquement par Nicolas GOIX le 28 Novembre 2016 LTCI, CNRS, Télécom ParisTech, Université Paris-Saclay, 75013, Paris, France Jury : Gérard Biau Professeur, Université Pierre et Marie Curie Examinateur Stéphane Boucheron Professeur, Université Paris Diderot Rapporteur Stéphan Clémençon Professeur, Télécom ParisTech Directeur Stéphane Girard Directeur de Recherche, Inria Grenoble Rhône-Alpes Rapporteur Alexandre Gramfort Maitre de Conférence, Télécom ParisTech Examinateur Anne Sabourin Maitre de Conférence, Télécom ParisTech Co-directeur Jean-Philippe Vert Directeur de Recherche, Mines ParisTech Examinateur List of Contributions Journal Sparse Representation of Multivariate Extremes with Applications to Anomaly Detection. (Un- • der review for Journal of Multivariate Analysis). Authors: Goix, Sabourin, and Clémençon. Conferences On Anomaly Ranking and Excess-Mass Curves. (AISTATS 2015). • Authors: Goix, Sabourin, and Clémençon. Learning the dependence structure of rare events: a non-asymptotic study. (COLT 2015). • Authors: Goix, Sabourin, and Clémençon. Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking. (AIS- • TATS 2016). Authors: Goix, Sabourin, and Clémençon. How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? (to be submit- • ted). Authors: Goix and Thomas. One-Class Splitting Criteria for Random Forests with Application to Anomaly Detection. (to be • submitted). Authors: Goix, Brault, Drougard and Chiapino. Workshops Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking. (NIPS • 2015 Workshop on Nonparametric Methods for Large Scale Representation Learning). Authors: Goix, Sabourin, and Clémençon.
  • Machine Learning for Anomaly Detection and Categorization In

    Machine Learning for Anomaly Detection and Categorization In

    Machine Learning for Anomaly Detection and Categorization in Multi-cloud Environments Tara Salman Deval Bhamare Aiman Erbad Raj Jain Mohammed SamaKa Washington University in St. Louis, Qatar University, Qatar University, Washington University in St. Louis, Qatar University, St. Louis, USA Doha, Qatar Doha, Qatar St. Louis, USA Doha, Qatar [email protected] [email protected] [email protected] [email protected] [email protected] Abstract— Cloud computing has been widely adopted by is getting popular within organizations, clients, and service application service providers (ASPs) and enterprises to reduce providers [2]. Despite this fact, data security is a major concern both capital expenditures (CAPEX) and operational expenditures for the end-users in multi-cloud environments. Protecting such (OPEX). Applications and services previously running on private environments against attacks and intrusions is a major concern data centers are now being migrated to private or public clouds. in both research and industry [3]. Since most of the ASPs and enterprises have globally distributed user bases, their services need to be distributed across multiple Firewalls and other rule-based security approaches have clouds, spread across the globe which can achieve better been used extensively to provide protection against attacks in performance in terms of latency, scalability and load balancing. the data centers and contemporary networks. However, large The shift has eventually led the research community to study distributed multi-cloud environments would require a multi-cloud environments. However, the widespread acceptance significantly large number of complicated rules to be of such environments has been hampered by major security configured, which could be costly, time-consuming and error concerns.