DOCSLIB.ORG

Unsupervised Network Anomaly Detection Johan Mazel

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Unsupervised Network Anomaly Detection Johan Mazel Unsupervised network anomaly detection Johan Mazel To cite this version: Johan Mazel. Unsupervised network anomaly detection. Networking and Internet Architecture [cs.NI]. INSA de Toulouse, 2011. English. tel-00667654 HAL Id: tel-00667654 https://tel.archives-ouvertes.fr/tel-00667654 Submitted on 8 Feb 2012 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. 5)µ4& &OWVFEFMPCUFOUJPOEV %0$503"5%&-6/*7&34*5²%&506-064& %ÏMJWSÏQBS Institut National des Sciences Appliquées de Toulouse (INSA de Toulouse) $PUVUFMMFJOUFSOBUJPOBMFBWFD 1SÏTFOUÏFFUTPVUFOVFQBS Mazel Johan -F lundi 19 décembre 2011 5J tre : Unsupervised network anomaly detection ED MITT : Domaine STIC : Réseaux, Télécoms, Systèmes et Architecture 6OJUÏEFSFDIFSDIF LAAS-CNRS %JSFDUFVS T EFʾÒTF Owezarski Philippe Labit Yann 3BQQPSUFVST Festor Olivier Leduc Guy Fukuda Kensuke "VUSF T NFNCSF T EVKVSZ Chassot Christophe Vaton Sandrine Acknowledgements I would like to first thank my PhD advisors for their help, support and for letting me lead my research toward my own direction. Their inputs and comments along the stages of this thesis have been highly valuable. I want to especially thank them for having been able to dive into my work without drowning, and then, provide me useful remarks. I want to make a special thank to Pedro Casas-Hernandez who worked with me inside the ECODE project at LAAS-CNRS. Working with him during one year and a half has been a real pleasure. I think that the extent of this thesis contributions would not have been the same without Pedro’s work. I extend a special thank to Ion Alberdi. He introduced me to functionnal programming through the Ocaml language and pointed me toward new and exciting territories in terms of programming and computer science in general. Our discussions have been on of the most valuable discussions I ever had. I want to also thank all the people working at LAAS-CNRS for the help they provided me. I here want to especially thank David Gauchard and Christian Berty for their exceptionnal availability and reactivity. I finally want to thank my family, friends and colleagues at LAAS-CNRS. I won’t quote any names but be assured that I am truly grateful for your support during these three and half years. 1 Abstract Anomaly detection has become a vital component of any network in today’s Inter- net. Ranging from non-malicious unexpected events such as flash-crowds and failures, to network attacks such as denials-of-service and network scans, network traffic anomalies can have serious detrimental effects on the performance and integrity of the network. The continuous arising of new anomalies and attacks create a continuous challenge to cope with events that put the network integrity at risk. Moreover, the inner polymorphic nature of traffic caused, among other things, by a highly changing protocol landscape, complicates anomaly detection system’s task. In fact, most network anomaly detection systems proposed so far employ knowledge-dependent techniques, using either misuse detection signature-based detection methods or anomaly detection relying on supervised- learning techniques. However, both approaches present major limitations: the former fails to detect and characterize unknown anomalies (letting the network unprotected for long periods) and the latter requires training over labeled normal traffic, which is a dif- ficult and expensive stage that need to be updated on a regular basis to follow network traffic evolution. Such limitations impose a serious bottleneck to the previously presented problem. We introduce an unsupervised approach to detect and characterize network anoma- lies, without relying on signatures, statistical training, or labeled traffic, which represents a significant step towards the autonomy of networks. Unsupervised detection is accom- plished by means of robust data-clustering techniques, combining Sub-Space clustering with Evidence Accumulation or Inter-Clustering Results Association, to blindly identify anomalies in traffic flows. Correlating the results of several unsupervised detections is also performed to improve detection robustness. The correlation results are further used along other anomaly characteristics to build an anomaly hierarchy in terms of danger- ousness. Characterization is then achieved by building efficient filtering rules to describe a detected anomaly. The detection and characterization performances and sensitivities to parameters are evaluated over a substantial subset of the MAWI repository which contains real network traffic traces. Our work shows that unsupervised learning techniques allow anomaly detection sys- tems to isolate anomalous traffic without any previous knowledge. We think that this contribution constitutes a great step towards autonomous network anomaly detection. This PhD thesis has been funded through the ECODE project by the European Com- mission under the Framework Programme 7. The goal of this project is to develop, implement, and validate experimentally a cognitive routing system that meet the chal- lenges experienced by the Internet in terms of manageability and security, availability and accountability, as well as routing system scalability and quality. The concerned use case inside the ECODE project is network anomaly detection. Keywords: anomaly detection, network traffic, unsupervised learning R´esum´e La d´etection d’anomalies est une tˆache critique de l’administration des r´eseaux. L’ap- parition continue de nouvelles anomalies et la nature changeante du trafic r´eseau com- pliquent de fait la d´etection d’anomalies. Les m´ethodes existantes de d´etection d’anoma- lies s’appuient sur une connaissance pr´ealable du trafic : soit via des signatures cr´e´ees `apartir d’anomalies connues, soit via un profil de normalit´e. Ces deux approches sont limit´ees : la premi`ere ne peut d´etecter les nouvelles anomalies et la seconde requiert une constante mise `ajour de son profil de normalit´e. Ces deux aspects limitent de fa¸con importante l’efficacit´edes m´ethodes de d´etection existantes. Nous pr´esentons une approche non-supervis´ee qui permet de d´etecter et caract´eri- ser les anomalies r´eseaux de fa¸con autonome. Notre approche utilise des techniques de partitionnement afin d’identifier les flux anormaux. Nous proposons ´egalement plusieurs techniques qui permettent de traiter les anomalies extraites pour faciliter la tˆache des op´erateurs. Nous ´evaluons les performances de notre syst`eme sur des traces de trafic r´eel issues de la base de trace MAWI. Les r´esultats obtenus mettent en ´evidence la possibilit´e de mettre en place des syst`emes de d´etection d’anomalies autonomes et fonctionnant sans connaissance pr´ealable. Mots-cl´es : d´etection d’anomalies, trafic r´eseau, apprentissage non-supervis´e Contents 1 Introduction 5 1.1 Motivatingproblems ............................. 6 1.1.1 Changing traffic in an evolving Internet . 6 1.1.1.1 Changingtraffic ...................... 6 1.1.1.2 Mutating anomalies . 10 1.1.2 Weaknesses of current network anomaly detection approa-ches .. 11 1.1.3 Theneedforanautonomousanomalydetection . 12 1.1.4 Summary ............................... 13 1.2 Contributions ................................. 13 1.2.1 Unsupervisedanomalydetection. 13 1.2.2 Anomalypost-processing. 14 1.2.3 Summary ............................... 14 1.3 Summary ................................... 14 1.4 ContextinsidetheECODEproject . 15 1.4.1 Overview ............................... 15 1.4.2 Usecases ............................... 15 1.4.3 Proposedarchitecture . .. .. .. .. .. 15 1.4.4 Summary ............................... 16 1.5 Dissertationoutline.............................. 16 2 Related work 17 2.1 Unsupervisedmachinelearning . 17 2.1.1 Blind signal separation . 17 2.1.2 Neuralnetworks............................ 18 2.1.3 Clustering algorithms . 18 2.1.3.1 General purpose clustering algorithms . 18 2.1.3.2 Clustering data in high-dimensional space . 19 2.1.3.3 Clustering evaluations . 25 2.2 Network anomaly detection and intrusion detection . .. 26 2.2.1 Classic network anomaly and intrusion detection . 26 2.2.1.1 Misuse detection-based intrusion and network anomaly detection .......................... 26 2.2.1.2 Anomaly detection-based network anomaly detection . 27 2.2.2 Machine learning applied to network anomaly detection and intru- siondetection ............................. 30 1 2.2.2.1 Misuse and anomaly-based supervised network anomaly detection .......................... 31 2.2.2.2 Supervised intrusion detection . 32 2.2.2.3 Unsupervised intrusion detection . 32 2.3 Summary ................................... 33 3 Unsupervised network anomaly detection 34 3.1 Change-Detection, Multi-resolution Flow Aggregation & Attribute com- putation.................................... 35 3.1.1 Changedetection ........................... 35 3.1.2 Multi-resolution Flow Aggregation . 36 3.1.3 Attribute building . 36 3.1.3.1 Metric definition . 36 3.1.3.2 Metric building . 36 3.1.3.3 Attributes generation over packet sets . 37 3.1.4 Summary ............................... 38 3.2 Finding anomalies with unsupervised
Load more

Recommended publications
  • A Review on Outlier/Anomaly Detection in Time Series Data
    A review on outlier/anomaly detection in time series data ANE BLÁZQUEZ-GARCÍA and ANGEL CONDE, Ikerlan Technology Research Centre, Basque Research and Technology Alliance (BRTA), Spain USUE MORI, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain JOSE A. LOZANO, Intelligent Systems Group (ISG), Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain and Basque Center for Applied Mathematics (BCAM), Spain Recent advances in technology have brought major breakthroughs in data collection, enabling a large amount of data to be gathered over time and thus generating time series. Mining this data has become an important task for researchers and practitioners in the past few years, including the detection of outliers or anomalies that may represent errors or events of interest. This review aims to provide a structured and comprehensive state-of-the-art on outlier detection techniques in the context of time series. To this end, a taxonomy is presented based on the main aspects that characterize an outlier detection technique. Additional Key Words and Phrases: Outlier detection, anomaly detection, time series, data mining, taxonomy, software 1 INTRODUCTION Recent advances in technology allow us to collect a large amount of data over time in diverse research areas. Observations that have been recorded in an orderly fashion and which are correlated in time constitute a time series. Time series data mining aims to extract all meaningful knowledge from this data, and several mining tasks (e.g., classification, clustering, forecasting, and outlier detection) have been considered in the literature [Esling and Agon 2012; Fu 2011; Ratanamahatana et al.
    [Show full text]
  • Image Anomaly Detection Using Normal Data Only by Latent Space Resampling
    applied sciences Article Image Anomaly Detection Using Normal Data Only by Latent Space Resampling Lu Wang 1 , Dongkai Zhang 1 , Jiahao Guo 1 and Yuexing Han 1,2,* 1 School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China; [email protected] (L.W.); [email protected] (D.Z.); [email protected] (J.G.) 2 Shanghai Institute for Advanced Communication and Data Science, Shanghai University, 99 Shangda Road, Shanghai 200444, China * Correspondence: [email protected] Received: 29 September 2020; Accepted: 30 November 2020; Published: 3 December 2020 Abstract: Detecting image anomalies automatically in industrial scenarios can improve economic efficiency, but the scarcity of anomalous samples increases the challenge of the task. Recently, autoencoder has been widely used in image anomaly detection without using anomalous images during training. However, it is hard to determine the proper dimensionality of the latent space, and it often leads to unwanted reconstructions of the anomalous parts. To solve this problem, we propose a novel method based on the autoencoder. In this method, the latent space of the autoencoder is estimated using a discrete probability model. With the estimated probability model, the anomalous components in the latent space can be well excluded and undesirable reconstruction of the anomalous parts can be avoided. Specifically, we first adopt VQ-VAE as the reconstruction model to get a discrete latent space of normal samples. Then, PixelSail, a deep autoregressive model, is used to estimate the probability model of the discrete latent space. In the detection stage, the autoregressive model will determine the parts that deviate from the normal distribution in the input latent space.
    [Show full text]
  • Quickest Change Detection
    1 Quickest Change Detection Venugopal V. Veeravalli and Taposh Banerjee ECE Department and Coordinated Science Laboratory 1308 West Main Street, Urbana, IL 61801 Email: vvv, [email protected]. Tel: +1(217) 333-0144, Fax: +1(217) 244-1642. Abstract The problem of detecting changes in the statistical properties of a stochastic system and time series arises in various branches of science and engineering. It has a wide spectrum of important applications ranging from machine monitoring to biomedical signal processing. In all of these applications the observations being monitored undergo a change in distribution in response to a change or anomaly in the environment, and the goal is to detect the change as quickly as possibly, subject to false alarm constraints. In this chapter, two formulations of the quickest change detection problem, Bayesian and minimax, are introduced, and optimal or asymptotically optimal solutions to these formulations are discussed. Then some generalizations and extensions of the quickest change detection problem are described. The chapter is concluded with a discussion of applications and open issues. I. INTRODUCTION The problem of quickest change detection comprises three entities: a stochastic process under obser- vation, a change point at which the statistical properties of the process undergo a change, and a decision maker that observes the stochastic process and aims to detect this change in the statistical properties of arXiv:1210.5552v1 [math.ST] 19 Oct 2012 the process. A false alarm event happens when the change is declared by the decision maker before the change actually occurs. The general objective of the theory of quickest change detection is to design algorithms that can be used to detect the change as soon as possible, subject to false alarm constraints.
    [Show full text]
  • Statistical Methods for Change Detection - Michèle Basseville
    CONTROL SYSTEMS, ROBOTICS AND AUTOMATION – Vol. XVI - Statistical Methods for Change Detection - Michèle Basseville STATISTICAL METHODS FOR CHANGE DETECTION Michèle Basseville Institut de Recherche en Informatique et Systèmes Aléatoires, Rennes, France Keywords: change detection, statistical methods, early detection, small deviations, fault detection, fault isolation, component faults, statistical local approach, vibration monitoring. Contents 1. Introduction 1.1. Motivations for Change Detection 1.2. Motivations for Statistical Methods 1.3. Three Types of Change Detection Problems 2. Foundations-Detection 2.1. Likelihood Ratio and CUSUM Tests 2.1.1. Hypotheses Testing 2.1.2. On-line Change Detection 2.2. Efficient Score for Small Deviations 2.3. Other Estimating Functions for Small Deviations 3. Foundations-Isolation 3.1. Isolation as Nuisance Elimination 3.2. Isolation as Multiple Hypotheses Testing 3.3. On-line Isolation 4. Case Studies-Vibrations Glossary Bibliography Biographical Sketch Summary Handling parameterized (or parametric) models for monitoring industrial processes is a natural approach to fault detection and isolation. A key feature of the statistical approach is its ability to handle noises and uncertainties. Modeling faults such as deviations in the parameter vector with respect to (w.r.t.) its nominal value calls for the use of statisticalUNESCO change detection and isolation – EOLSS methods. The purpose of this article is to introduce key concepts for the design of such algorithms. SAMPLE CHAPTERS 1. Introduction 1.1 Motivations for Change Detection Many monitoring problems can be stated as the problem of detecting and isolating a change in the parameters of a static or dynamic stochastic system. The use of a model of the monitored system is reasonable, since many industrial processes rely on physical principles, which write in terms of (differential) equations, providing us with (dynamical) models.
    [Show full text]
  • Machine Learning and Extremes for Anomaly Detection — Apprentissage Automatique Et Extrêmes Pour La Détection D’Anomalies
    École Doctorale ED130 “Informatique, télécommunications et électronique de Paris” Machine Learning and Extremes for Anomaly Detection — Apprentissage Automatique et Extrêmes pour la Détection d’Anomalies Thèse pour obtenir le grade de docteur délivré par TELECOM PARISTECH Spécialité “Signal et Images” présentée et soutenue publiquement par Nicolas GOIX le 28 Novembre 2016 LTCI, CNRS, Télécom ParisTech, Université Paris-Saclay, 75013, Paris, France Jury : Gérard Biau Professeur, Université Pierre et Marie Curie Examinateur Stéphane Boucheron Professeur, Université Paris Diderot Rapporteur Stéphan Clémençon Professeur, Télécom ParisTech Directeur Stéphane Girard Directeur de Recherche, Inria Grenoble Rhône-Alpes Rapporteur Alexandre Gramfort Maitre de Conférence, Télécom ParisTech Examinateur Anne Sabourin Maitre de Conférence, Télécom ParisTech Co-directeur Jean-Philippe Vert Directeur de Recherche, Mines ParisTech Examinateur List of Contributions Journal Sparse Representation of Multivariate Extremes with Applications to Anomaly Detection. (Un- • der review for Journal of Multivariate Analysis). Authors: Goix, Sabourin, and Clémençon. Conferences On Anomaly Ranking and Excess-Mass Curves. (AISTATS 2015). • Authors: Goix, Sabourin, and Clémençon. Learning the dependence structure of rare events: a non-asymptotic study. (COLT 2015). • Authors: Goix, Sabourin, and Clémençon. Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking. (AIS- • TATS 2016). Authors: Goix, Sabourin, and Clémençon. How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? (to be submit- • ted). Authors: Goix and Thomas. One-Class Splitting Criteria for Random Forests with Application to Anomaly Detection. (to be • submitted). Authors: Goix, Brault, Drougard and Chiapino. Workshops Sparse Representation of Multivariate Extremes with Applications to Anomaly Ranking. (NIPS • 2015 Workshop on Nonparametric Methods for Large Scale Representation Learning). Authors: Goix, Sabourin, and Clémençon.
    [Show full text]
  • Machine Learning for Anomaly Detection and Categorization In
    Machine Learning for Anomaly Detection and Categorization in Multi-cloud Environments Tara Salman Deval Bhamare Aiman Erbad Raj Jain Mohammed SamaKa Washington University in St. Louis, Qatar University, Qatar University, Washington University in St. Louis, Qatar University, St. Louis, USA Doha, Qatar Doha, Qatar St. Louis, USA Doha, Qatar [email protected] [email protected] [email protected] [email protected] [email protected] Abstract— Cloud computing has been widely adopted by is getting popular within organizations, clients, and service application service providers (ASPs) and enterprises to reduce providers [2]. Despite this fact, data security is a major concern both capital expenditures (CAPEX) and operational expenditures for the end-users in multi-cloud environments. Protecting such (OPEX). Applications and services previously running on private environments against attacks and intrusions is a major concern data centers are now being migrated to private or public clouds. in both research and industry [3]. Since most of the ASPs and enterprises have globally distributed user bases, their services need to be distributed across multiple Firewalls and other rule-based security approaches have clouds, spread across the globe which can achieve better been used extensively to provide protection against attacks in performance in terms of latency, scalability and load balancing. the data centers and contemporary networks. However, large The shift has eventually led the research community to study distributed multi-cloud environments would require a multi-cloud environments. However, the widespread acceptance significantly large number of complicated rules to be of such environments has been hampered by major security configured, which could be costly, time-consuming and error concerns.
    [Show full text]
  • CSE601 Anomaly Detection
    Anomaly Detection Jing Gao SUNY Buffalo 1 Anomaly Detection • Anomalies – the set of objects are considerably dissimilar from the remainder of the data – occur relatively infrequently – when they do occur, their consequences can be quite dramatic and quite often in a negative sense “Mining needle in a haystack. So much hay and so little time” 2 Definition of Anomalies • Anomaly is a pattern in the data that does not conform to the expected behavior • Also referred to as outliers, exceptions, peculiarities, surprise, etc. • Anomalies translate to significant (often critical) real life entities – Cyber intrusions – Credit card fraud 3 Real World Anomalies • Credit Card Fraud – An abnormally high purchase made on a credit card • Cyber Intrusions – Computer virus spread over Internet 4 Simple Example Y • N1 and N2 are N regions of normal 1 o1 behavior O3 • Points o1 and o2 are anomalies o2 • Points in region O 3 N2 are anomalies X 5 Related problems • Rare Class Mining • Chance discovery • Novelty Detection • Exception Mining • Noise Removal 6 Key Challenges • Defining a representative normal region is challenging • The boundary between normal and outlying behavior is often not precise • The exact notion of an outlier is different for different application domains • Limited availability of labeled data for training/validation • Malicious adversaries • Data might contain noise • Normal behaviour keeps evolving 7 Aspects of Anomaly Detection Problem • Nature of input data • Availability of supervision • Type of anomaly: point, contextual, structural
    [Show full text]
  • A Survey of Change Detection Methods Based on Remote Sensing Images for Multi-Source and Multi-Objective Scenarios
    remote sensing Article A Survey of Change Detection Methods Based on Remote Sensing Images for Multi-Source and Multi-Objective Scenarios Yanan You , Jingyi Cao * and Wenli Zhou School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China; [email protected] (Y.Y.); [email protected] (W.Z.) * Correspondence: [email protected] Received: 11 June 2020; Accepted: 27 July 2020; Published: 31 July 2020 Abstract: Quantities of multi-temporal remote sensing (RS) images create favorable conditions for exploring the urban change in the long term. However, diverse multi-source features and change patterns bring challenges to the change detection in urban cases. In order to sort out the development venation of urban change detection, we make an observation of the literatures on change detection in the last five years, which focuses on the disparate multi-source RS images and multi-objective scenarios determined according to scene category. Based on the survey, a general change detection framework, including change information extraction, data fusion, and analysis of multi-objective scenarios modules, is summarized. Owing to the attributes of input RS images affect the technical selection of each module, data characteristics and application domains across different categories of RS images are discussed firstly. On this basis, not only the evolution process and relationship of the representative solutions are elaborated in the module description, through emphasizing the feasibility of fusing diverse data and the manifold application scenarios, we also advocate a complete change detection pipeline. At the end of the paper, we conclude the current development situation and put forward possible research direction of urban change detection, in the hope of providing insights to the following research.
    [Show full text]
  • Change Detection Algorithms
    25 2 Change Detection Algorithms In this chapter, we describe the simplest change detection algorithms. We consider a sequence of indepen- y p y k dent random variables k with a probability density depending upon only one scalar parameter. t 0 1 0 Before the unknown change time 0 , the parameter is equal to , and after the change it is equal to . The problems are then to detect and estimate this change in the parameter. The main goal of this chapter is to introduce the reader to the design of on-line change detection al- gorithms, basically assuming that the parameter 0 before change is known. We start from elementary algorithms originally derived using an intuitive point of view, and continue with conceptually more involved but practically not more complex algorithms. In some cases, we give several possible derivations of the same algorithm. But the key point is that we introduce these algorithms within a general statistical framework, based upon likelihood techniques, which will be used throughout the book. Our conviction is that the early introduction of such a general approach in a simple case will help the reader to draw up a unified mental picture of change detection algorithms in more complex cases. In the present chapter, using this general approach and for this simplest case, we describe several on-line algorithms of increasing complexity. We also discuss the off-line point of view more briefly. The main example, which is carried through this chapter, is concerned with the detection of a change in the mean of an independent Gaussian sequence.
    [Show full text]
  • Anomaly Detection in Data Mining: a Review Jagruti D
    Volume 7, Issue 4, April 2017 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Anomaly Detection in Data Mining: A Review Jagruti D. Parmar Prof. Jalpa T. Patel M.E IT, SVMIT, Bharuch, Gujarat, CSE & IT, SVMIT Bharuch, Gujarat, India India Abstract— Anomaly detection is the new research topic to this new generation researcher in present time. Anomaly detection is a domain i.e., the key for the upcoming data mining. The term ‘data mining’ is referred for methods and algorithms that allow extracting and analyzing data so that find rules and patterns describing the characteristic properties of the information. Techniques of data mining can be applied to any type of data to learn more about hidden structures and connections. In the present world, vast amounts of data are kept and transported from one location to another. The data when transported or kept is informed exposed to attack. Though many techniques or applications are available to secure data, ambiguities exist. As a result to analyze data and to determine different type of attack data mining techniques have occurred to make it less open to attack. Anomaly detection is used the techniques of data mining to detect the surprising or unexpected behaviour hidden within data growing the chances of being intruded or attacked. This paper work focuses on Anomaly Detection in Data mining. The main goal is to detect the anomaly in time series data using machine learning techniques. Keywords— Anomaly Detection; Data Mining; Time Series Data; Machine Learning Techniques.
    [Show full text]
  • Anomaly Detection in Raw Audio Using Deep Autoregressive Networks
    ANOMALY DETECTION IN RAW AUDIO USING DEEP AUTOREGRESSIVE NETWORKS Ellen Rushe, Brian Mac Namee Insight Centre for Data Analytics, University College Dublin ABSTRACT difficulty to parallelize backpropagation though time, which Anomaly detection involves the recognition of patterns out- can slow training, especially over very long sequences. This side of what is considered normal, given a certain set of input drawback has given rise to convolutional autoregressive ar- data. This presents a unique set of challenges for machine chitectures [24]. These models are highly parallelizable in learning, particularly if we assume a semi-supervised sce- the training phase, meaning that larger receptive fields can nario in which anomalous patterns are unavailable at training be utilised and computation made more tractable due to ef- time meaning algorithms must rely on non-anomalous data fective resource utilization. In this paper we adapt WaveNet alone. Anomaly detection in time series adds an additional [24], a robust convolutional autoregressive model originally level of complexity given the contextual nature of anomalies. created for raw audio generation, for anomaly detection in For time series modelling, autoregressive deep learning archi- audio. In experiments using multiple datasets we find that we tectures such as WaveNet have proven to be powerful gener- obtain significant performance gains over deep convolutional ative models, specifically in the field of speech synthesis. In autoencoders. this paper, we propose to extend the use of this type of ar- The remainder of this paper proceeds as follows: Sec- chitecture to anomaly detection in raw audio. In experiments tion 2 surveys recent related work on the use of deep neu- using multiple audio datasets we compare the performance of ral networks for anomaly detection; Section 3 describes the this approach to a baseline autoencoder model and show su- WaveNet architecture and how it has been re-purposed for perior performance in almost all cases.
    [Show full text]
  • Detecting Anomalies in System Log Files Using Machine Learning Techniques
    Institute of Software Technology University of Stuttgart Universitätsstraße 38 D–70569 Stuttgart Bachelor’s Thesis 148 Detecting Anomalies in System Log Files using Machine Learning Techniques Tim Zwietasch Course of Study: Computer Science Examiner: Prof. Dr. Lars Grunske Supervisor: M. Sc. Teerat Pitakrat Commenced: 2014/05/28 Completed: 2014/10/02 CR-Classification: B.8.1 Abstract Log files, which are produced in almost all larger computer systems today, contain highly valuable information about the health and behavior of the system and thus they are consulted very often in order to analyze behavioral aspects of the system. Because of the very high number of log entries produced in some systems, it is however extremely difficult to find relevant information in these files. Computer-based log analysis techniques are therefore indispensable for the process of finding relevant data in log files. However, a big problem in finding important events in log files is, that one single event without any context does not always provide enough information to detect the cause of the error, nor enough information to be detected by simple algorithms like the search with regular expressions. In this work, three different data representations for textual information are developed and evaluated, which focus on the contextual relationship between the data in the input. A new position-based anomaly detection algorithm is implemented and compared to various existing algorithms based on the three new representations. The algorithms are executed on a semantically filtered set of a labeled BlueGene/L log file and evaluated by analyzing the correlation between the labels contained in the log file and the anomalous events created by the algorithms.
    [Show full text]