Network Security Monitoring and Analysis Based on Big Data Technologies

University of Nevada, Reno

Network Security Monitoring and Analysis based on Big Data Technologies

A dissertation submitted in partial fulﬁllment of the requirements for the degree of Doctor of Philosophy in Computer Science and Engineering

Bingdong Li

Dr. Mehmet Hadi Gunes / Dissertation Co-Advisor Dr. George Bebis / Dissertation Co-Advisor

December, 2013 c Copyright by Bingdong Li 2013

THE GRADUATE SCHOOL

We recommend that the dissertation prepared under our supervision by

BINGDONG LI

entitled

Network Security Monitoring and Analysis based on Bigig DataData TechnologiesTechnologies

be accepted in partial fulfillment of the requirements for the degree of

DOCTOR OF PHILOSOPHY

Mehmet Hadi Gunes , Ph.D. , Co-Advisor

George Bebis , Ph.D., Co -Advisor

Murat Yuksel , Ph.D. ,, CommitteeCommittee MemberMember

Minggen Lu , Ph.D. ,, CommitteeCommittee MemberMember

Dong Yu, Ph.D. ,, CommitteeCommittee MemberMember

Yantao Shen , Ph.D. ,, GraduateGraduate SchoolSchool RepresentativeRepresentative

Marsha H. Read, Ph. D., Dean, Graduate School

December, 2013

Abstract

Network ﬂow data provide valuable information to understand the network state and to be

aware of the network security threats. However, processing the large amount of data

collected from the network and providing real time information remain as big challenges.

Big data technologies provide new approaches to collect, store, measure, and analyze the

large amount of data. This dissertation aims to provide a system of network security

monitoring and analysis based on the big data technologies.

First, I present an extensive survey of the network ﬂow applications that covers past

research perspectives, methodologies, and a discussion of challenges and future works.

Then, I present system design of the network security monitoring and analysis platform

based on the Big Data technologies. Components of this system include Flume and Kafka for real time distributed data collection; Storm for real time streaming distributed data processing; Cassandra for NoSQL data storage, data processing, and user interfaces. The

system supports real time continuous network monitoring, interactive visualization,

network measurement, and modeling to classify host roles based on host behaviors and to

identify a particular user among the other users.

It is critical to continuously monitor the network status and network security threats in real

time, but it is a challenge to process the large amount of data in real time. I demonstrate

how the big data security system designed in this dissertation supports such features.

Another usage of the network ﬂow data is to measure the contents of the network. I

demonstrate how this big data system provides understanding of the usage of anonymity ii

technologies on the campus Internet. Then, I present methods and the results of

classification and identification of network objects based on the big data system designed in this dissertation. Finally, I use Decision Tree and Support Vector Machine to model host role behaviors and user behaviors. Sample results indicate very high accuracy of host role classification and user identification. iii

Dedication

This work is dedicated to my family; to my parent who loved me unconditionally but I could not be with them at their last minutes which I regret the most in my life, and to my wife Zheng Chen and my son Daniel who sweet my heart day and night. iv

Acknowledgements

I would like to take this opportunity to thank my advisors Dr. Mehmet Gunes and Dr.

George Bebis for their advices and encouragements. This work would not have been successful without their guidance.

Special thanks to my manager Jeff Springer, who supported me in many ways; encouraging me pursue PhD degree and providing the research environment. This work would not have been possible without his support.

Thanks to Dr. Murat Yuskel, Dr. Yantao Shen, Dr. Minggen Lu, and Dr. Dong Yu for accepting to serve on my dissertation committee.

I would also like to thank all people who have inspired me getting this step.

Finally, thanks to my family whom my ﬂesh and soul depend on. My parent taught me the fundamental and most important things in my life; to be a strong, honest, and independent person, to work hard, and to keep tackling difﬁcult things. My wife has been taking care of our home so that I can spend time on my research. My son has been such a great kid and always reminds me to rest at the right time.

During my Ph.D. study, I have went through up and down. There were many times I thought to give up. It is my faith that led me to the ﬁnal line. v

Table of Contents

Abstract ...... i Dedication ...... iii Acknowledgements ...... iv Table of Contents ...... v List of Figures ...... viii List of Tables ...... x Chapter 1 Introduction ...... 1 1.1 Motivations...... 2 1.2 Objectives ...... 4 1.3 Contributions...... 4 Chapter 2 Background ...... 6 2.1 NetworkFlow ...... 6 2.1.1 NetFlow...... 7 2.1.2 sFlow ...... 8 2.1.3 IPFIX ...... 9 2.1.4 Network Flow Analysis ...... 10 2.2 BigDataandRelatedTechnologies ...... 10 2.2.1 FileSystem ...... 12 2.2.2 Distributed, Parallel and Concurrent Computing ...... 12 2.2.3 DataCollection ...... 14 2.2.4 DataStorage ...... 16 2.3 MachineLearning ...... 16 2.4 WebTechnologies ...... 18 2.4.1 AJAX ...... 18 2.4.2 HTML5...... 19 vi

2.4.3 Nodejs...... 19 2.4.4 Data Visualization on the Web ...... 19 Chapter 3 A Survey of Network Flow Applications ...... 21 3.1 Perspectives ...... 21 3.1.1 Network Monitoring, Measurement and Analysis ...... 22 3.1.2 Network Application Classiﬁcation ...... 25 3.1.3 User Identity Inferring ...... 27 3.1.4 Security Awareness and Intrusion Detection ...... 27 3.1.5 IssuesofDataError...... 30 3.2 Methodologies ...... 32 3.2.1 Statistics...... 32 3.2.2 MachineLearning...... 33 3.2.3 Proﬁling...... 36 3.2.4 Behavior-based Approaches ...... 38 3.2.5 Visualization ...... 40 3.2.6 Anonymization ...... 40 3.2.7 AnalysisSystems ...... 42 3.3 Discussion ...... 45 3.3.1 Datasets...... 45 3.3.2 ResearchPerspectives...... 45 3.3.3 Methodologies ...... 46 3.3.4 Challenges ...... 46 3.3.5 Future Directions ...... 47 Chapter 4 The Big Data Security System Design ...... 49 4.1 Approach...... 49 4.2 Components of the Security Analysis System ...... 50 4.2.1 DataCollection ...... 52 4.2.2 DataStorage ...... 52 vii

4.2.3 SecurityGateway ...... 53 4.2.4 DataProcessing...... 54 4.2.5 UserInterfaces ...... 54 4.3 Features...... 54 4.3.1 Real Time Continuous Network Security Monitoring and Interac- tiveVisualization ...... 54 4.3.2 NetworkMeasurement ...... 55 4.3.3 Advanced Network Modeling ...... 55 4.4 Discussion ...... 55 Chapter 5 Real Time Continuous Network Monitoring and Interactive Visualization ...... 57 5.1 Real Time Network Host Querying ...... 58 5.2 Real Time Continuous Network Monitoring ...... 59 5.2.1 Network Flow Status ...... 60 5.2.2 Top N Conversations ...... 61 5.3 Interactive Network Security Awareness Visualization ...... 61 5.4 Discussion ...... 63 Chapter 6 A Case Study of Network Flow Measurement ...... 64 6.1 Usage of Anonymity Network ...... 64 6.1.1 Campus Network Traffic Flows ...... 65 6.2 RelatedWork...... 67 Chapter 7 Classification and Identification of Network Objects ...... 71 7.1 Methods ...... 73 7.1.1 DataSet...... 73 7.1.2 Algorithm...... 73 7.1.3 Modelling...... 74 7.1.4 GroundTruth ...... 74 7.2 HostsRoleClassification...... 74 viii

7.2.1 ClassificationFeatures ...... 75 7.2.2 Classification of Client versus Server ...... 77 7.2.3 Classification of Web Email Server versus Web Non-email Server 78 7.2.4 Classification of Hosts from Personal Office versus Public Place . 79 7.2.5 Classification of Hosts from Two Different Colleges ...... 80 7.2.6 Feature Contributions ...... 81 7.3 UserIdentification ...... 83 7.3.1 IdentificationFeatures ...... 84 7.3.2 User Identification Results ...... 86 7.3.3 Feature Contributions ...... 87 7.4 Discussion ...... 89 7.4.1 Classification of Network Applications ...... 89 7.4.2 Profiling the Host and Network ...... 90 7.4.3 Machine Learning Approaches ...... 90 7.4.4 Classifying and Clustering Host Roles ...... 91 7.4.5 Identifying the User among Others based on Network Flow.... 92 Chapter 8 Conclusion ...... 93 Chapter 9 Future Work ...... 95 9.1 Improvement to the Current Work ...... 95 9.2 Extensions to the Current Work ...... 95 9.2.1 BackgroundTraffic ...... 95 9.2.2 Detection of Operating System Fingerprinting ...... 96 9.2.3 Identity Anonymity ...... 96 9.2.4 Fusion with Other Network Security Data ...... 96 9.3 Vision...... 97 Bibliography ...... 98 ix

List of Figures

Figure2.1 NetFlowProcess ...... 8 Figure2.2 sFlowProcess ...... 9 Figure2.3 MapReduceWordCountProcess ...... 13 Figure 2.4 A Simple Storm Topology with Word Count ...... 13 Figure 2.5 Flume Agent Data Flow Model ...... 14 Figure 2.6 Kafka Architecture ...... 15

Figure3.1 PublicationsbyYear ...... 22 Figure 3.2 Analyzed Perspectives by Years ...... 23 Figure3.3 MethodsbyYears ...... 33 Figure 3.4 Bipartite Graph (left) and One-mode Projection (right) ...... 38

Figure 4.1 The Big Data Security System Architecture Design ...... 51

Figure 5.1 Real Time Host Trafﬁc Query ...... 60 Figure 5.2 Analyzing Network Targets ...... 61 Figure 5.3 Real Time Network Flow Status Monitor ...... 62 Figure 5.4 Real Time Network Flow Top N Status ...... 63

Figure 6.1 Geo-Location of Anonymity Usage on Campus (sFlow Data) .... 65

Figure 7.1 Average Accuracy of Classifying Client versus Server ...... 78 Figure 7.2 Average Accuracy of Classifying Regular Web Email Server versus WebNon-emailServer...... 79 Figure 7.3 Average Accuracy of Classifying Host from Personal Ofﬁce Client versusPublicPlaceClient ...... 80 Figure 7.4 Average Accuracy of Classifying Host from Two Different Colleges . 81 Figure 7.5 C5.0 Feature Contributions in Classifying Host Roles ...... 82 x

Figure 7.6 Average Accuracy of Identifying the User ...... 87 Figure 7.7 C5.0 Feature Contribution in Identifying the User ...... 88 xi

List of Tables

Table 3.1 Summary of Security Awareness and Intrusion Detection ...... 31 Table 3.2 Summary of Machine Learning Approaches of Network Application Classiﬁcation...... 36 Table 3.3 Summary of Machine Learning Approaches of Anomaly Detection . 36 Table 3.4 Summary of NetFlow Visualization Applications ...... 41

Table 6.1 Analyzed Anonymity Systems ...... 64 Table 6.2 Usage of Anonymity Systems based on sFlow Data ...... 65 Table 6.3 Application Usage of Anonymity Systems based on sFlow Data . . . 70

Table 7.1 Formal Description of Network Objects ...... 73 Table7.2 AnalyzedSystems ...... 74 1

Chapter 1. Introduction

Understanding what information is ﬂowing in the computer network is valuable for network

administrators, accounting, network planning, network security, forensics, and counter- terrorism. Network ﬂow data are supported by a wide range of products including Cisco

Netflow [91], Juniper “cflowd”, NetStream by 3COM and Huawei, and sFlow. Network flow 1 records high-level descriptions, i.e., meta-data, of Internet connections but not the actual data transferred. In general, collection and analysis of network flow information are more efficient than deep packet inspection. It protects the privacy of the users and works even for encrypted traffic. The gathered information helps to uncover both external activities as well as internal activities such as network misconfiguration and policy violation. Since the Cisco’s NetFlow patent in 1996, extensive research of network flow has been conducted and many applications have been developed [112]. We have seen a rise of research interest and application development for network flow monitoring and analysis in recent years [112]. Traditionally, network flow analysis has focused on the network monitoring, planning, and billing. Recent research, however, has focused more on the network security analysis. Behavior-based approaches, which are based on normal network host and user behaviors rather than static assumptions, have become the mainstream [69]. Advanced techniques such as machine learning have been employed as well. The analysis systems are moving toward distributed system to provide more scalability, robustness, and computational power for real-time in depth analysis. However, many challenges still remain including a huge volume of network flow data, a small amount of relevant information in the network flow data, and real time requirements for network security analysis. Additionally,

1Network ﬂow is used to refer these data in this dissertation to avoid these trademarks. 2

network security is becoming more challenging as attacks are getting more sophisticated, organized, targeted, persistent, and dynamic. Attacks are not only from external sources but also internal sources.

1.1 Motivations

Traditional network security systems such as firewall and anti-virus software have been developed and deployed for a long time. These traditional security systems assume a static system. There have been many network flow monitoring and analysis systems that have been developed and deployed. However, few of them have made the network security as the primary goal, and none can meet the real time performance requirement of network security monitoring. These systems store data in traditional databases (e.g., Oracle, Microsoft SQL, or MySQL) or flat files, which can not handle real time performance requirements.

Computer hardware and software have been constantly improving, and so have hackers’ abilities. Hence, new intelligence-driven real time security systems that are more agile, dynamic, risk-aware, and contextual are in need. Big data [5], which is deﬁned as volume, velocity, variety and veracity, has become a new trend in dealing with large amount of data in real time. These new technologies provide innovative approaches to collect, store, real time measure, and analyze a large amount of data. Among these technologies, Apache Hadoop [3] related technologies are open source and provide a valuable platform for public and researchers. The Apache Hadoop project includes a collection of tools for reliable, scalable, and distributed data collection, storage, and processing. Big data technologies have played an important role in global information processing, and have showed considerable promise for improving network security. There are a large amount of data that are valuable to network security including network ﬂow data, network 3

logs, application access records, and firewall logs. However, these data have not been used effectively because the data are too large and variable making it challenging, not just to store the data but also to analyze it in meaningful ways. These data are growing hourly as the network is ever-expanding, connecting, and mobilizing. In particular, the big data technologies may provide valuable information to network security operation in a timely manner. Big data technologies may transform the network security into a new stage where these additional data are utilized [6]. Additionally, analysis by adding context of role-based behavior provides a new direction for accurate cyber security monitoring. Behavior-based user identification provides information for user identification and forensic investigation, and opens a possible area of network user identification based on the user’s network flow data.

Behavior-based approaches and machine learning have been valuable for network flow analysis. Machine learning represents a collection of methods for discovering knowledge by searching for patterns, and has been very successful in computer vision and speech recognition. Applying machine learning methods to role-based behavior approach provides overall dynamic and contextual abilities to the network security analysis system. Ad- ditionally, behavior-based approaches first learn normal behaviors to build models and then make decisions based on the deviation from the normal behavior. They overcome the static assumptions made by traditional security systems. Different behavior based studies have focused on application classification, anomaly detection, visualization of host behaviors, and host clustering. Most of the studies use basic statistics of network flow data, and focus on network and known attack patterns. Few studies use machine learning approaches, and none, as far as we are aware of, has applied machine learning to classify and identify network objects based on roles. 4

1.2 Objectives

Given the challenges network security is facing and new opportunities that Big data technologies provide, this dissertation proposes a network security monitoring and analysis system based on big data technologies to collect, store, access, measure, and analyze network flow data. This system makes the network security a primary goal, measures the campus network, provides real time continuous network monitoring and interactive visualization, and enables intelligent network objects, i.e., network hosts and users, classification and identification based on role behavior as context. Compared with most network flow analysis approaches of modeling abstract and unknown objects such as virus, botnet or anomaly with static signatures, in this dissertation, I model concrete and specific network objects such as hosts and users based on their profile and behaviors. The big data network security system is low cost and built on open source projects. By building the system to collect online streaming data, to real time continuously monitor, and to analyze the role-based behavior, the network security system provides real time continuous monitoring, interactive visualization, intelligent analysis, and is low false rate of classification and identification of security threats.

1.3 Contributions

The contributions of this work are relevant within the network security monitoring, measurement, classification, identification, and intrusion detection. The experiment of processing network flow based on the big data technologies in this dissertation provides an example and insights for the use of big data technologies in cyber security. The big data technologies have opened new opportunities to design a better system for handling security related information flows and providing agile response. The platform designed in this 5

system can be used for security as a service for customers. In addition, the distributed network ﬂow monitoring and analysis system has been implemented and is being used by

the network security administrators of our campus. The highlights of my contributions can be summarized as below.

• A distributed system based on big data technologies for network ﬂow real time monitoring, network measurement, and analysis based on the open source big data tech-

nologies.

• Use of the big data technologies to achieve real time continuous network monitoring and interactive visualization.

• A case study of network measurement by measuring usage of anonymity technologies on campus.

• Models of classiﬁcation of clients versus servers, clients from public places versus personal ofﬁces, servers of web email servers versus web servers, and clients from different colleges using sFlow data, and identifying a particular user among the other

users using NetFlow data. 6

Chapter 2. Background

This chapter provides background information related to technologies used in this disserta-

tion. These technologies are summarized in four categories:

• Network Flow: NetFlow, sFlow, IP Flow Information Export protocol (IPFIX), and Network Flow Analysis

• Big data and Related Technologies: Apache Hadoop open source distributed platform for collecting, storing, and analyzing large amount of data

• Machine Learning

• Web Development Technologies

2.1 Network Flow

A network flow is defined as an unidirectional or bidirectional sequence of packets between two endpoints (from server to client or from the client to server) with some attributes in common. Most important key fields includes: source/destination IP address, source/destination port number, protocol type, type of services, bytes transferred, and input logical interface. Other fields may be included that depend on the NetFlow version or configuration for the export. These fields provide a rich set of traffic statistics including user, protocol, port, and type of service that can be used for a wide variety of purposes that include network security, network monitoring, traffic analysis, capacity planning, traffic classification, accounting, and billing. The general process of working with NetFlow includes capturing, sampling, generating, exporting, collecting, analyzing, and visualization. 7

There are various systems that capture network ﬂow data from network links or devices such as Ntop [46], NG-MON [77], NetFlow, NetFlow-lite, sFlow, cﬂowd, and NetStream.

NetFlow and sFlow are widely used systems, and IPFIX [14] is the new deﬁned standard for IP ﬂow format. In this section, I give a brief introduction to NetFlow, sFlow, IPFIX,

and trafﬁc analysis.

2.1.1 NetFlow

NetFlow is a trafﬁc monitoring technology developed by Darren and Barry Bruins in 1996

at Cisco [91]. It deﬁnes how a router exports information and statistics of routed sockets. As a de facto industry standard, it is a built-in feature of most routers and switches produced

by Cisco, Juniper, and other vendors. Network devices look at the packets arriving on the interfaces, and capture traffic statistics per flow based on configuration for sampling or

filtering, then they create a flow cache, aggregate and export the data through UDP or SCTP. NetFlow cache entry is created by the first packet of a flow, maintained for similar flow characteristics, and exported to collectors periodically based on flow timers or flow cache

management. The export format was ﬁxed before version 8. After version 9, extensibility and ﬂexibility were added to integrate with Multiprotocol Label Switching (MPLS), IPv6,

Border Gateway Protocol (BGP), and user deﬁned records. NetFlow version 5 and 9 are the most popular versions. Sampled NetFlow is a variant originally introduced by Cisco to reduce computational

burden by reducing the number of NetFlows. Sampling rate can be conﬁgured as prede- termined nth packet or randomly selected interval. Figure 2.1 presents the basic process of

NetFlow formation, exportation, storage and analysis. Due to the large volume of network trafﬁc and limited computational resources (ie., memory, CPU and bandwidth), caching, sampling and UDP exportation are used. These can cause quality issues for the collected NetFlow data: (1) some new ﬂows will not be counted when cache is full; (2) sampling 8

Network Packet Cache Flow Collector Store & Analysis

New Flow

Export

Flow Data

Figure 2.1: NetFlow Process reduces the accuracy of ﬂows, especially when sampling rate is adjusted by the trafﬁc rate;

(3) exported flow records do not necessarily correspond to the order in which the flow traffic arrived at the router. There are varieties of NetFlow collectors and analysis tools from commerical vendors such as Cisco, freeware or developed in-house for special purposes [13, 15].

2.1.2 sFlow

Packet sampling of traffic flow [52] has been used even before NetFlow was developed. sFlow was developed by InMon Inc. and has become an industry standard defined in RFC

3176. It is a technology using simple random sampling and is supported by Alcatel, Ex- treme, Force10, HP, Hitachi by embedding the sFlow agent within switches and routers. The sFlow agent is a software process that combines interface counters and flow samples into sFlow datagrams and immediately sends them to sFlow collectors via UPD. Immedi- ate forwarding of data minimizes memory and CPU usage at the network device. Packets are typically sampled by Application-Specific Integrated Circuits (ASICs) to provide wire- speed performance. sFlow data contain complete packet header and switching/routing information, and provides up to the minute view of the network traffic. sFlow is able to run at layer 2 and capture non-IP traffic as well. The sFlow collectors are servers that collect 9

sFlow Analysis System

Collector Data Storage Analysis

Switch/Router

Management

sFlow agent

ASICs Counters Samples

Figure 2.2: sFlow Process the sFlow datagrams. The ofﬁcial sFlow web site [18] provides a list of available sFlow collectors. Figure 6.1 presents the basic components and processes of sFlow analysis.

2.1.3 IPFIX

IP Flow Information Export (IPFIX) is an IETF standard for exporting network ﬂow based on NetFlow version 9, and is deﬁned in RFC 5101 for information transmitting protocols,

RFC 5102 for information model, and RFC 5103 for exporting bidirectional ﬂow. IPFIX was designed to meet the fast growing requirements of observing network trafﬁc, to provide

an extensible and flexible data model that can be customized, and to support reliable and secure data transfer through SCTP, TCP, and UDP. IPFIX flow definition is less restrictive than traditional flow definition. 10

2.1.4 Network Flow Analysis

Network ﬂow analysis is the process of discovering useful information by using statistics or other sophisticated approaches. The basic process includes capturing, collecting and storing data, aggregating the data for query and analysis, and analyzing the data for useful information. This information is mostly related to network management, measurement, and network security. There are different ways to collect network ﬂow data including

SNMP, NetFlow, raw packet, or auditing data from network infrastructure such Firewalls, and VPN gateways. Typically, there are two strategies: (1) depth-ﬁrst when there is known information and clear purpose, or (2) breadth-ﬁrst when looking for a general view of the network without a particular purpose.

In general, deep packet inspection needs packet level information and consumes more computational resources. Flow level analysis, such as NetFlow and sFlow, typically consumes less computational resources. There are many products and tools developed by industry or open source community. Analysis of network ﬂow information has become crucial as the Internet has become the living blood in our society and is expanding at a fast pace around the world. There are many challenges in analyzing network ﬂow data such as, huge amount of data due to networks becoming larger and faster, limited high-level information, and complex statistical properties. As discussed in sections 3.1 and 3.2, various perspectives have been analyzed and many algorithms have been developed.

2.2 Big Data and Related Technologies

Big data technologies are still in its inception stage. Its concept is constantly growing as we try to understand its size, complexity, and methods for data collecting, storing, processing, analyzing, sharing, and visualizing. In general, Big data [5] refer four dimensions or four Vs: 11

• Volume: data size is from terabytes to petabytes in a data set.

• Variety: data type can be structured (e.g., data in traditional SQL database), unstructured (e.g., conversation between people), or semi-structured (e.g., emails are

semi-structured with structured data of ’from’, ’to’, ’subject’, and unstructured email body).

• Velocity: processing the big data in a short time or real time to maximize the data value for users.

• Veracity: truthful intelligence can be gained from more noise, less truth, and high uncertainty data (e.g., the NetFlow data has high uncertainty to identify a user in this dissertation).

Applications of big data technologies are tremendous, at scientiﬁc data of physics, bioinformatics, computational social science, climate simulation, government, and more. In this dissertation, I monitor and analyze security aspect of a large amount of data generated by computer network. Research related to big data technologies is very active and is supported by government and private sectors. Big data require technologies from many aspects of computer science, from high performance computing, collecting, and storing. The Apache Hadoop project [3] is a open source project from Apache, which includes a collection of tools for reliable, scalable and distributed data collecting, storing, and processing. Its core components include the Hadoop Distributed File System (HDFS), the parallel data processing Map reduce, and other common core modules. In the following, I brieﬂy introduce some open source projects of big data technologies related to this dissertation. 12

2.2.1 File System

Hadoop Distributed File System (HDFS) is a distributed ﬁle system that is designed to handle big data (in the order of petabytes), to provide high-throughput access to a large amount of data and highly fault-tolerant, and to run on low-cost hardware. HDFS is the basics for other Apache Hadoop related technologies.

2.2.2 Distributed, Parallel and Concurrent Computing

Big data demand more computing power than traditional data processing. Distributed, parallel and concurrent computing provide these need. Parallel and concurrent programming,

however, is a nightmare for many programmers. Fortunately, there are several open source projects to support distributed, parallel and concurrent computing. The following applications are experimented in this dissertation.

2.2.2.1 Map Reduce

The map reduce is a programming model for distributed parallel computing, and is com- posed of a pipeline of map and reduce. A simple example is the word count example of

lines of words as ﬁgure 2.3. In the map stage, it outputs word as the key and 1 as the value; In the reduce stage, it outputs words as key and sum as the value. In general, map reduce is

good ﬁt for log processing, data mining, and machine learning. Map reduce has interface for Java, C++, and high level script Pig Latin.

2.2.2.2 Apache Pig Latin

Pig Latin [141] is a platform and has a scripting language for analyzing large data sets in parallel using map-reduce programs. It makes map reduce parallel programming easier. 13

Input Splitting Mapping Shuffling Reducing Final Result

Deer River Deer 1 Bear 1 Bear 1 River 1 Car 1 Car 1 Bear 1 Deer River Car 1 Car 1 Car River Car River Deer 2 River 1 Deer 1 Deer Bear River 2 Deer 1 Deer 2

Deer 1 River 1 Deer Bear Bear 1 River 1 River 2

Figure 2.3: Map Reduce Word Count Process

2.2.2.3 Storm

Storm is a distributed, reliable, fault-tolerant, and real time distributed computing system

developed by BackType (now Twitter) in 2011 [20]. It was designed for real time computing of streaming data. Now it is also used for continuous computation and distributed remote procedure call (RPC). Compared with map reduce which performs batch process-

ing of jobs, Storm uses the term topology which processes messages forever. The two main components of a topology are spouts which are sources of streams to accept streaming data, and bolts which are single threads to process the stream data as shown in Figure 2.4. Compared with traditional big data solutions such as Apache Hadoop, Storm supports continuous streaming data processing without termination.

Continuous Input Spout Bolt

Deer River Deer Bear Deer,River, Deer 3 Deer,Bear, Bear 2 Car River Bear Deer Car,River, Car 1 Bear,Deer River 2

Figure 2.4: A Simple Storm Topology with Word Count 14

2.2.3 Data Collection

Big bata collection requires distributed, reliable, scalable, extensible, and efﬁcient data collection, aggregation, and migration. There are several distributed message collecting systems available, and some of them are open source projects. In general, the main consid- erations are conﬁguration, fault-tolerance, and maintenance. Flume and Kafka stand out of and are experimented in this dissertation.

2.2.3.1 Flume

Flume [2] was built by Cloudera and is a Apache project. It was designed as a highly reliable, distributed logging service HDFS. It has two main components: agents and masters.

Masters are responsible to keep track and manage all agents. Agents are responsible to collect, aggregate, and move data to data store. Figure 2.5 presents the Flume agent data

ﬂow model. Multiple layers of agents can be added for different roles such as data collection and aggregation. In the same way, more masters can be deployed to make the system fault-tolerant so that any one of them fails, the system will not fail.

Flume-ng is the next generation Flume with many improvements that make it more conﬁgurable and provide more features. Flume Agent

Source Sink

Network Flow To Cassandra

Figure 2.5: Flume Agent Data Flow Model 15

2.2.3.2 Kafka

Kafka [99] is a distributed data collecting system developed by LinkedIn, and later became an open-source Apache project. It was designed as a distributed publish-subscribe messag-

ing system for high throughput. A agent in Kafka has three basic roles: a producer which produces message, a broker which handles message and routes them to the appropriate

topic queue, and a consumer which consumes the message in the message queue of the the speciﬁc topic. Figure 2.6 presents the Kafka architecture. Kafka can play as a traditional distributed event logger to save the message for ofﬂine

analysis as Flume does. Kafka can also support real-time streaming data analysis by the consumer. Flume and other distributed event log systems are typically push based system whereas Kafka is a poll based message processing system. Kafka sends data to consumers over network via Linux sendﬁle() API which is more efﬁcient.

Producer 1 Producer m Producer N

Broker 1 Broker N ------Topic1 Topic1 /part1 /part1 … … /partN /partN TopicN TopicN /part1 /part1 … … /partN /partN

Consumer 1 Consumer m Consumer N

Figure 2.6: Kafka Architecture 16

2.2.4 Data Storage

The data storage of big data is NoSQL, i.e., Not Only SQL. NoSQL supports un-structured and structured data, multiple replications, and large volume of data. There are many

NoSQL solutions from commercial to open source. Each solution has its unique char- acteristic such as Apache Hadoop based, document store, key value store, and graph. In this dissertation, Cassandra is chosen as the data storage.

Apache Cassandra [1] is an open source, distributed NoSQL storage system. Starting at Facebook, it combines Amazon’s Dynamo architectural aspects and Google’s Bigtable data model as an open source project. It provides scalability, reliability, durability, and high performance. It is one of the most widely used among other NoSQL solutions, and has been adopted by companies such as NetFlix, Twitter, and Cisco. Eric Brewer’s well-known ”CAP Theory” [70] claimed that it is impossible for a web service to provide consistency, availability, and partition-tolerance at the same system. Cassandra is known for availability and partition-tolerance, but arguable provides consistency by specifying a consistency level. Compared with traditional relational databases, Cassandra has a simple schema com- prising keyspace (like database in Microsoft SQL or schema in Oracle database), column families (like tables in relational databases), rows and columns. Each row is stored as a single ﬁle on a machine, and column size and name can be static or dynamic. Cassandra provides application programming interfaces for Java, C++, Python, R, and Pig for data processing.

2.3 Machine Learning

The machine learning approach is a collection of methods for discovering knowledge by searching for patterns in a data set. The machine learning approach reﬁnes and improves its knowledge base by learning from experience. The basic learning types are listed below: 17

• Classiﬁcation: classify inputs to labeled outputs.

• Clustering: group inputs into clusters.

• Association: discover interesting relations between features.

• Prediction: predict outcome in terms of a numeric quantity.

Machine learning schemes include information theory, neural networks, support vector machines, genetic algorithms, and many more [165]. Machine learning applications require the collection of training and test data sets and depend on algorithms for feature extraction, feature selection, and learning. Initially, the system is trained using example data to learn specific data associations; then, the system is deployed in a similar environment where test data are used for classification. The Decision Tree and Support Vector Machine are two supervised machine learning algorithms that are often applied to network flows and typically perform better than others [112]. Therefore, in our experiments we chose both of these supervised machine learning algorithms.

2.3.0.1 Decision Tree

Decision Tree [65] uses inductive inference to produce a tree like structure which includes a root node, internal nodes, and leaf nodes. Classification is processed from the root node and moving down toward some leaf nodes. The strength of Decision Tree algorithms come from performing fast classification without requiring much computation, and providing information of which features are most important for classification. The weakness of Deci- sion Tree is that the training process is computationally expensive as the tree grows bigger. 18

2.3.0.2 Support Vector Machine

Support Vector Machine was ﬁrst introduced in the 1960s [31]. It has evolved over the time and has been successfully used for classiﬁcation of different applications. A Support Vector

Machine separates the classes with optimal hyperplanes that the largest possible points of the same class are on the same side. Support vectors are built based on the data points close to the hyperplanes. Different kernels are used to map data to feature spaces. The most common kernels are linear, polynomial, radial basis function, and sigmoid. Selecting a kernel is very important for success of the classiﬁcation. Support Vector Machines are scalable for high dimensional data. Online Support Vector Machine [31] is an extensive version that stores the model for each training, and then adds new examples while removing the least relevant examples for the new data set. It saves memory and provides ﬂexibility for scenarios involving stream data.

2.4 Web Technologies

The Internet is providing an exponentially growing content and is moving toward the cloud computing. Web related technologies have improved signiﬁcantly in the recent. The following are several web development technologies used in this dissertation.

2.4.1 AJAX

Asynchronous Javascript XML (Ajax) is a new model of web development. In Ajax development model, web application can send data to or retrieve data from a server syn- chronously or asynchronously. Ajax provides fast performance and opens new doors for interactive user experience and rich contents. 19

2.4.2 HTML 5

Hyper Text Markup Language (HTML) 5 is built on the latest development of web applications: multimedia, multi-platform, more complex web applications, low-powered devices such as smartphone, and more. A server-sent event is a new feature of HTML 5 that the web page automatically gets update from the server. Different from traditional stateless web page, WebSocket provides a full-duplex communications over a single TCP connection.

2.4.3 Nodejs

Rapid web development is on demand because of quick prototype, short time development and low cost. Many new platforms and programming languages have been developed in recently. Nodejs [17] is a server-side JavaScript for writing scalable Internet applications, and is a high performance and concurrent network application framework. Nodejs runs on V8 which is an open source JavaScript engine developed by Google [21]. Nodejs highly leverages event-driven model to implement non-blocking operation and everything inside nodejs runs in a single thread to make it lightweight.

2.4.4 Data Visualization on the Web

Data visualization on the web has gone through a breakthrough in recent years, especially with the new features of HTML 5 such as the canvas element and SVG images. WebGL is another JavaScript web graphical library supported by most browsers. Hardware and high performance JavaScript engines make these possible for interactive graphs on the web. Interactive visualization allows users to interact with the graph and provide dynamic information about the data by user inputs. There are many open source JavaScript libraries that make data visualization on the 20

web easier. D3 (d3js.org) and Flot (www.flotcharts.org) are used in this dissertation to visualize the network status. 21

Chapter 3. A Survey of Network Flow Applications

Research in network flow analysis has become very active in the recent years as observed in figure 3.1, which shows the published paper distribution with respect to publication year. It is necessary to look back what perspectives have been achieved, and what methods have been used and are more effective in order to move forward. This chapter presents a survey of NetFlow-like applications that studies were published between 1998 and early 2012 with emphasis on network flow applications. The rest of this chapter is organized as follows. Section 3.1 reviews the key perspectives which have been addressed in the literature including networking monitoring, analysis and management, application classification, inferring user identity, and network security awareness. Section 3.2 explains the key methodologies which have been employed for data analysis including statistic, machine learning, profiling, behavior-based approaches, sampling, visualization, and computational infrastructures. Section 3.3 discusses the limi- tations of existing approaches, challenges and future directions.

3.1 Perspectives

In this section, I survey the main research perspectives of network ﬂow applications. In particular, it covers network monitoring, measurement and analysis, application classiﬁca- tion, user identity inferring, security awareness and intrusion detection, and issues related to error and bias in NetFlow collection and analysis. Figure 3.2 shows the distribution of 22

0 1998 2000 2002 2004 2006 2008 2010 2012 Figure 3.1: Publications by Year published papers with respect to ﬁve main perspectives: monitoring, classiﬁcation, user identifying, security, and issues related to errors. As it can be observed, network security has been the main research topic using NetFlow data.

3.1.1 Network Monitoring, Measurement and Analysis

Network monitoring and measurement provide valuable information to network administrators, as well as ISPs and content providers. Compared to other technologies, such as

SNMP or Windows Management Instrumentation(WMI), network ﬂow data contain additional information for further analysis. For example, they can provide bandwidth analysis, speciﬁc protocol monitoring, and system performance, etc. Monitoring based on NetFlow can be categorized as:

• Network monitoring: provides information about routers and switches as well as network-wide basis view, and is used for problem detection along with efﬁcient trou- 23

20 monitor classification identity security issues

0 1998 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Figure 3.2: Analyzed Perspectives by Years

bleshooting.

• Application monitoring: provides information about application usage over the network, and is used for planning and allocation of resources.

• Host monitoring: provides information about user utilization of network and applications, and is used for planning, network access control, violating security policy.

• Security monitoring: provides information about network behavior changes, and is used to identify DoS attacks, viruses and worms, and network anomalies.

• Accounting and Billing: provides network metering, and is used for billing.

In this section, we focus on network, application, user and resource monitoring while section 3.1.4 focuses on security related monitoring. In the following, we discuss speciﬁc research perspectives. 24

3.1.1.1 Network Monitoring

Many aspects of networks have been studied using NetFlow data, including network performance based on round-trip time [170], delay measurement [97, 108], connectivity [157],

misuse of bandwidth [121], trafﬁc characterization [101], ﬁnding heavy hitters [178], monitoring for special purpose QoS [113], and diagnosis of troubleshooting [171].

3.1.1.2 Application Monitoring

Liu and Huebner [115] investigated the stochastic characteristics of distributions of ﬂow length, packet size, throughput, etc. for the popular and bandwidth consuming applications. Kalafut et al. [86] proposed a heuristic method to differentiate wanted and unwanted trafﬁc based on the sampled NetFlow data.

3.1.1.2.1 VoIP Voice over IP (VoIP) service is widely used, however, it introduces security threats that include Session Initiation Protocol (SIP) scanning, SIP ﬂooding, and Real- time Transport Protocol (RTP) ﬂooding. Lee [106] developed a system that can monitor VoIP service and detect VoIP network threats based on NetFlow statistics and behaviors.

Kobayashi [96] presented a method for measuring VoIP trafﬁc ﬂuctuation by using Net- Flow and sFlow based on the variance of the interval of the target RTP packets. Lucas [47] provided an open source VoIP monitoring system based on protocol characteristics.

3.1.1.2.2 Mobile Network Sinha et al. [163] analyzed the flow-level upstream traffic behavior from Broadband Fixed Wireless (BFW) and Digital Subscriber Link (DSL) to provide traffic characteristics of these networks. Moghaddam [128] used wireless NetFlow data to measure and simulate user behavior and provide information for future mobile network design. 25

3.1.1.2.3 IPv6 With the transition from IPv4, there is a need to understand IPv6 usage including user behavior, traffic volume, transitional technologies, assignment of IPv6 address, IPv6 percentage of network traffic. NetFlow can provide this information including application types, usage of transitional technologies of IPv6 to IPv4, interface identifier assignment schemes, etc [161, 202].

3.1.1.3 Host Monitoring

Host proﬁle and relationships in the network can be used for resource planning as well as for network security analysis. Caracas et al. [34] proposed an algorithm based on Net-

Flow data to describe the dependencies among computer systems, software components, and services. Kind et al. [94] presented a method to uncover the relationships between IT infrastructures using NetFlow data. Chen et al. [39] developed novel heuristics to analyze characteristics and correlations between inter-data centers and client trafﬁc, provided insights into data center design and operation. Several methods have been proposed for proﬁling behaviors on the end hosts [189, 194, 195]. Behavior-based approaches will be discussed in detail in section 3.2.4.

3.1.2 Network Application Classiﬁcation

Network application classification classifies network traffic into certain application categories which can be coarse- or fine-grained. Network application classification is a challenging task because of obfuscation techniques such as content encryption, dynamic ports, and proprietary communication protocols. Classification approaches can be divided into four categories: port based, payload-based, heuristic based using transport layer statistics, and machine learning based. Port based approaches are no longer reliable because of certain applications that randomly assign ports. Payload based approaches do not work on encrypted traffic, and are resource-intensive and scale poorly with high bandwidth. Ap- 26

proaches based on heuristic and machine learning approaches provide alternative methods. There are many reasons for network traffic classification. Network administrators need information for applications running at the network (e.g., file sharing), whether they are legitimate users or worms. ISPs and content providers need the information for quality of service assurances. Research in this area has conducted for over ten years, but it is still growing. There is a list of 68 published papers and 86 data sets collected in CAIDA web pages [12]. There are several surveys on network application classification using traffic classification approaches [92] and machine learning approaches [139]. We discuss machine learning approaches in detail in section 3.2.2.1. It is worth mentioning that Perelman et al. proposed a method that investigates the application signatures of web browsers, mail client, or media-players in network flow [145]. Peer to peer networks have become a major security concern and the focus of most network classification studies. We discuss peer to peer classification in detail below.

3.1.2.1 Peer to Peer Network

Peer to Peer (P2P) networks have been widely utilized for file-sharing, video distribution and voice communications. They consume more Internet traffic than traditional applications, and have been a concern for network administrators and a challenge for network security. There is interest from ISPs and network administrators to identify and control the P2P network traffic [72, 200]. NetFlow provides an alternative approach that is more efficient in terms of storage and processing than deep packet inspection (DPI). Recently, there has been considerable effort on NetFlow P2P analysis. These include methods based on: (a) default P2P port for heavy-hitters [183], (b) port usage pattern of specific P2P network such as BitTorrent [30, 72], (c) flow statistic characteristics such as packet length and time-interval [30, 148, 193], (d) TCP flags that a host, as both client and server, send/re- ceive a packet with both SYN and ACK at the same time [85], (e) machine learning that 27

using features such as IP address and port, packet size, bytes exchanged. Among machine learning approaches we discuss in section 3.2.2.1, six out of eight classiﬁy P2P trafﬁc.

3.1.3 User Identity Inferring

Identifying a person based on extrinsic biometric is not new; well-known examples include

signatures and keystrokes. Inferring user identity based on network ﬂow patterns however is a new ﬁeld. Melnikov et al. [125] discussed the potential of inferring user identity using NetFlow feature distribution and cross-correlation of various trace parameters and relationships among packets. Even though the reported results were preliminary, additional research may yield more promising results.

3.1.4 Security Awareness and Intrusion Detection

In this section, we focus on security related awareness, detection and monitoring. Ta- ble 3.1 provides a list of studies that provide perspectives on security awareness and intrusion detection. Table 3.3 also lists approaches that use machine learning approaches. IDSes can be categorized based on how they identify intrusions: anomaly-based, misuse- based (knowledge-based or signature-based), or combination of both anomaly and misuse- based [167]. Alternatively, IDSes can be categorized based on what they target: host-based, network-based or both. Network anomaly detection refers to ﬁnding patterns that are not expected users behaviors, also known as anomaly-based IDS. Compared with misuse-based IDS, these patterns are previously unknown. Most content-oriented systems belong to knowledge-based detection, which looks for known signatures of malware by inspecting trafﬁc packets. Most behavior-oriented systems belong to anomaly-based detection, which differentiate anomalous behavior from normal behavior. NetFlow based IDSes use existing NetFlow data and limited information, and avoid privacy issues compared to content-oriented approaches. 28

However, NetFlow based IDSes are more difﬁcult because of limited information in the NetFlow data. Consequently, recent research has shown that machine learning approaches

are better than statistical and streaming methods. Sperotto et al.s [167] conducted an overview of IP flow-based intrusion detection that focused on flow-based IDS, concept of flows, classification of attacks, and defense techniques. In the following, we discuss perspectives of security awareness and intrusion detection that can be achieved using NetFlow data.

3.1.4.1 Top N

Top N refers a set of statistic and models of NetFlow data. They reflect the basic network status. It is relatively simple with NetFlow analysis. It can be used to find the big talkers or heavy-hitters. It also can be used for abnormal traffic detection [201].

3.1.4.2 Port Scanning

Port scanning is the act of systematically scanning a computer’s ports, and is usually done by using small packets that probe the target machines. In most network attacks, port scanning is the ﬁrst reconnaissance step. Port scanning can be classiﬁed in three categories: scanning many ports on a single host, scanning a single port on many hosts, and combination of both. Detection of port scanning is addressed in most studies cited in table 3.1.

Approaches include host incoming/outgoing connections, probability of entropy, Bayesian logistic regression, distances from baseline models, and machine learning.

3.1.4.3 Denial of Service

A denial-of-service (DoS) or distributed denial-of-service (DDoS) attack is an attempt to make the target host or network resource unable to respond to its requests. Detection of DOS or DDoS is addressed in most ﬂow based IDSes. Gao et al. [68] proposed a resilient 29

DoS detection based on sketch-based schemes that use a hash table for storing aggregated flow measurement. Kim et al. [93] described different DoS attacks based on traffic patterns and presented a network anomaly detection method that can detect flooding attacks. New developments include using novel dynamic entropy to measure the anomaly [89], an attack detection method based on statistic aggregation that can detect DDoS and port scanning [66]. Table 3.1 provides a list of related studies.

3.1.4.4 Worms

A worm is a standalone malicious program that replicates across the networks by exploit- ing software vulnerabilities or tricking users to execute it by social engineering. Worms can cause mildly annoying effects, damaging data or software, DoS, stealing data, etc.

Detection of worms can be categorized as trap-oriented, packet-oriented and connection- oriented [37]. Detection of port scanning is one of the important steps for worm detection, and hence many similar approaches are used in both types of detection. NetFlow-like approaches are connection-oriented and include: analysis of host behavior on the basis of incoming and outgoing connections [50], correlation between NetFlow data and honeypot logs [49], and detecting hit-list worms using protocol graphs [44]. Chan et al. [37] proposed FloWorM system that includes tracker, analyzer and reporter based on NetFlow data. Abdulla et al. [23] presented a Support Vector Machines (SVM) method based on the fact that a scanning activity or email worm initiates a signiﬁcant amount of trafﬁc without

DNS.

3.1.4.5 Botnet

Botnets are malware at the infected target and controlled by a remote entity known as bot- master. They have become one of the major security threats credited for DDoS, spamming, phishing, identity theft, and other cyber crimes. Many botnets rely on communication chan- 30

nels varying from centralized IRC and HTTP to decentralized P2P networks. Detection of a botnet is relatively more difﬁcult than detection of port scanning and worms. Zhu et al. [37]

conducted a survey on understanding, detection and tracking, and defending against botnets. Recent approaches use advanced methodologies and combine host and network level

information. Zeng et al. [199] proposed a method that combined host and network-level information with protocol-independent detection. BotCloud’s detection is based on MapRe- duce and combining host and network approaches [63]. BotTrack’s tracking is based on PageRank of NetFlow data and host behavioral model [62]. Finally, Barsamian used a network statistical behavioral model for botnet detection [26], and Weststrate used heuristic methods to ﬁnd botnet servers [191].

3.1.4.6 Policy Validation

Peer-to-peer networks can be used legitimately, or misused by botnet, or violate network usage policy. Section 3.1.2.1 details peer-to-peer classiﬁcation using NetFlow data. Kr- micek et al. [100] proposed an approach to detect the use of unauthorized Network Ad-

dress Translation (NAT) via a heuristic method based on NetFlow data. NetFlow data can also provide information about legitimate ﬂows denied by the security policy, and help network administrator with troubleshooting. Frias-Martinez [64] proposed a behavior-based

network access control mechanism with a true rejection rate of 95%.

3.1.5 Issues of Data Error

NetFlow data is exported using UDP. Data can be lost due to overloaded segments between routers and collectors, an overload of collectors with benign trafﬁc increases, burst nature of NetFlow trafﬁc, or attacks in progress. Similarly, errors may happen in the process of

sampling, transporting and collecting. In order to address these problems, several methods have been proposed. Cohen et al. [43] proposed a framework for calculating conﬁdence 31

Table 3.1: Summary of Security Awareness and Intrusion Detection

Year Methodology Perspective 2001 [57] Histogram and chart IDS 2001 [98] Statistic DoS and DDos 2004 [197] Links between machines or do- IDS mains 2004 [93] Statistic patterns DoS and DDoS 2005 [50] Host behavior based Worm outbreaks 2005 [51] IP aggregation Detection and monitoring 2006 [152] Flow aggregation IDS 2007 [151] Trust and reputation model IDS 2007 [49] Flow signature and honeypot logs Worm detection 2008 [37] Heuristics Worm detection 2008 [204] Statistic Anomaly detection 2009 [100] Heuristics NAT detection 2009 [182] Decision tree Dictionary attack 2009 [64] K-means Behavior-based NAC 2009 [196] Information theory Risk detection 2009 [201] Statistic Top N detection 2009 [181] Statistic Spam machines 2010 [180] NBA Malware 2010 [80] Spatial-temporal aggregating Malicious website detection 2011 [156] Statistic of host behavior Attack Detection 2011 [89] Dynamic entropy DoS 2011 [66] Statistic DDoS and port scan 2011 [62] Host behavior and PageRank Botnets detection 2011 [168] Time series IDS 2011 [63] PageRank Botnets detection 32

intervals to address the estimation errors in a multistage combination of sampling and aggregation. Trammell et al. [176] characterized, quantiﬁed, and corrected timing errors, which are consequence of Cisco NetFlow version 9 protocol design that estimates the true base time from derived base time information. Rohmad et al. [154] proposed an enhanced

NetFlow version 9 using nProbe GPL. Fioreze et al. [59] investigated the trustfulness of NetFlow measurements and found that octets and packets are reliably reported, but the

ﬂow duration of samples are shorter than the actual duration. Zhu et al. [205] studied the errors of utilized bandwidth measurement of NetFlow, and provided guidance for correctly estimating the utilized bandwidth. Finally, Ricciato et al. [153] described a methodology to estimate one-way packet loss from IPFIX or NetFlow ﬂow records.

3.2 Methodologies

In this section, we review various methodologies used to analyze NetFlow data. Figure 3.3 provides a chronological summary of the methodologies discussed in this section. As it can be observed, a considerable number of studies have focused on using machine learning algorithms and real time analysis.

3.2.1 Statistics

Statistic approaches are the most common methods in NetFlow analysis. In general, it is the basic step before applying heuristic-based approaches, machine learning and visualization.

NetFlow data contains statistics of network flow information generated and exported from routers. Duffield et al. [55] investigated the resource usage of NetFlow formation and exportation as well as statistical properties of original traffic from sampled traffic data. Proto et al. [147] proposed a statistical model for network intrusion detection system. Sawaya et al. [156] proposed an approach of attack detection based on traffic flow statistics of hosts. 33

10 statistic ML profiling behavior-based visualization 8 anonymization system

0 1998 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Figure 3.3: Methods by Years

Barsamian [26] proposed a botnet detection method using statistical signatures. Liu et al. [29] proposed an analysis and monitoring system using NetFlow statistic, and an IDS

based on variance similarity. Compared to other approaches, statistical approaches are usually easier to implement, provide accurate results and consume less resources. However, statistical approaches are

good only for known cases and lack the ability to adapt to new cases.

3.2.2 Machine Learning

In this section, we provide a survey of machine learning approaches in NetFlow applications, which include traffic classification, anomaly detection, and security awareness. Selecting an appropriate set of features for a specific problem is critical. Example of features are shown in Tables 3.2 and 3.3, and are categorized as: (1) basic features such as NetFlow data fields, source and destination IP address and port, network interface, trans- 34

port protocol, type of service, start and finish timestamps, cumulative TCP flags, number of bytes and packets transmitted, and MPLS labels; (2) derived features such as flow length

(ﬁnish time - start time), average packet size (bytes / number of packets), average ﬂow rate (bytes / length), average packet rate (number of packet / length), aggregation of IP subnet

and traffic load bytes, percentage of traffic load on a node, percentage of traffic load at the current sub-tree with time period and aggregation threshold [184, 186]; (3) application specific heuristics such as webmail traffic [158] that has properties as close service proximity, daily and weekly pattern, and duration of client session; and (4) advanced features such as abacus signature, degree distribution, self-similarity of flow interval, entropy, kernel function, mutual information and Hellinger distance [179], or data fusion with other log files such as Snort, DNS related requests [23] (number of DNS requests, response, normals, and anomalies for each host over a certain period of time). Methods for feature selection include symmetric uncertainty [84], information gain [169], subgroup, keyword selection, gradually reduction based on efficiency [116], and rough sets.

The type of data sets and features being employed are very important for a successful machine learning approach. Typically, a large dataset is necessary to cover various relations in the data, including temporal and spatial relations. Training data has to be attack-free or attack-speciﬁc, both of which are difﬁcult to obtain. The data sets in Table 3.3 can be categorized as (a) Internet backbone of more than one week period, (b) Internet backbone of less than one week period, (c) intranet of more than two weeks, and (d) simulated data or honeypot log.

3.2.2.1 Application Classiﬁcation

Nguyen et al. [139] surveyed the application of machine learning techniques for traffic classification from 2004 to 2007; Even though NetFlow was not specified as analysis data set,

but the basic methodologies are applicable to NetFlow data. Kim et al. [92] conducted an 35

evaluation of trafﬁc classiﬁcation using traces with collected payloads. Their evaluation included seven machine learning algorithms: Naive Bayes (NB), Naive Bayes Kernel Esti- mation (NBKE), Bayesian Network (BN), C4.5 Decision Tree (DT), k-Nearest Neighbors,

Neural Networks, and Support Vector Machines (SVM). They concluded that SVM consis-

tently achieved higher accuracy. Soysal et al. [166] conducted more specific evaluations and comparisons of BN, DT and Multilayer Perceptons on flow-based network traffic clas-

sification using flow trace data. They concluded that BN and DT are suitable for Internet traffic flow classification. Nor et al. [129] evaluated a large number of machine learning algorithms in terms

of their performance on NetFlow data with the objective of classifying HTTP, gmail, and video streaming. The highest accuracy machine learning algorithms had an accuracy more

than 99.33%. Unfortunately, they did not provide information about the features used. Table 3.2 summarizes the algorithms, accuracy, features and data types for trafﬁc classiﬁ- cation using NetFlow data. Since accuracy varies considerably, there is a need to evaluate

these algorithms and features on the same data set.

3.2.2.2 Security Awareness and Anomaly Detection

Table 3.3 provides a summary of machine learning algorithms for anomaly detection in

terms of algorithms, features, and research perspectives. The highest reported detection rate is 98% [192]. Sommer et al. [165] found that applying machine learning for network anomaly detection is harder than in other domains. This is mainly due to the great variety of trafﬁc and the fundamental nature of machine learning approaches that are better suited at ﬁnding similarities than identifying relationships that are not present in the training data. 36

Table 3.2: Summary of Machine Learning Approaches of Network Application Classiﬁca- tion

Year Algorithm Accu.(%) Feature Application 2007 [84] NBKE 91 Basica and P2P, email, Multi- derivedb media 2009 [35] DT 90 Basic P2P, VoIP, DNS, email, FTP 2010 [38] Clustering 90 Applicationc SNMP, email, DNS, IRC 2010 [155] SVM 90 Advancedd P2P 2010 [158] SVM 94 Application Webmail 2010 [25] DT 90 Basic and P2P, HTTP, VoIP, DNS, derived FTP, email, games 2011 [179] SVM 70 Advanced P2P 2012 [114] BN 95 Derived BULK, email, P2P

aBasic NetFlow data ﬁelds bCalculation and aggregation of basic features cApplication speciﬁc properties from basic and derived features dAbstract information from basic and derived features Table 3.3: Summary of Machine Learning Approaches of Anomaly Detection

Year Algorithm Feature Dataset Perspective 2005 [102] Cluster Advanceda Internet Anomaly 2007 [116] Multiclass Advanced Internet NSSA SVM 2008 [187] GA-based Derivedb non-NetFlowc DDoS 2010 [186] Kernel Derived Internet Monitoring 2010 [169] SVM Derived Intranet Masquerade 2011 [23] SVM Applicationd non-NetFlow Worm 2011 [184] SVM Derived Internet Attacks 2011 [185] SVM Advanced Internet Attacks 2011 [192] SVM Basice and derived non-NetFlow IDS

aAbstract information from basic and derived features bCalculation and aggregation of basic features cSimulation or log data dApplication speciﬁc properties from basic and derived features eBasic NetFlow data ﬁelds

3.2.3 Proﬁling

Network profiling is an important step for further analysis. Various profiling levels have been discussed in the literature including user, application, host, and network profiling. 37

• User profiling: There is limited work on the user profiling based on NetFlow data. Melnikov et al. [125] proposed a set of correlation and distribution of user flow data related to time and packet to identify a users. Different user behavior based ap-

proaches have employed various features that are discussed in section 3.2.4. User proﬁling research, however, may provide helpful information for network security in the future.

• Application proﬁling: Liu and Huebner [115] discussed the stochastic characteristics of some of the most popular applications (i.e., FTP, HTTP, SNMP, NNTP, DNS, and

Napster): ﬂow length and time by probability density function and tail distribution, average packet size distribution, and average throughput distribution. Karagiannis et

al. [87] proposed trafﬁc patterns of social behavior, function (provider or consumer), and application ports that were used to classify trafﬁc based on heuristic rules.

• Host proﬁling: Wei et al. [189] proposed an approach for Internet host proﬁling using a data structure that can be expressed in XML-like format at listing 3.1, where the communication similarity is the average of Dice similarity values for the host. Kuai

et al. [194] proposed an approach based on bipartite graphs to represent host communication and one-mode projection of bipartite graphs to capture the social-behavior

similarity of end hosts as ﬁgure 3.4. For networks with few hosts, we need more detailed information for further analysis. Minarik et al. [127] proposed a host behavior proﬁling based on the bi-directional NetFlow that use communicating peers (number

of servers contacted, clients answered, and single flows), amount of traffic (amount of requests, replies, and single flows), and traffic structure (number of client, server and

single flows). Frias-Martinez et al. [64] defined a host behavior profile that contains seven features: the total number of flows, average flow size, average flow duration,

total number of packets contained in all flows, average number of packets per flow, total number of unique IP addresses contained in all flows, and average packet size. 38

• Network profiling:Cho et al. proposed Aguri tree [41], an aggregation-based traffic profile that aggregates small volume flows with a fixed number of nodes in an IP tree for spatial measurement. Jiang et al. [83] characterized network prefix-level traffic

profiling as daily traffic volume, distributions (over time, direction, applications, and flow size), and ratio of upload-download. Lakhina et al. [103] described Origin- Destination flows using a routing metric, and further analyzed using Aguri tree to

include time, features (i.e., source and destination address, source and destination port) and volume to represent both time and space attributes [102] .

A A X B E B C Y D Z C D E

Figure 3.4: Bipartite Graph (left) and One-mode Projection (right)

3.2.4 Behavior-based Approaches

Recently, behavior-based approaches to network security have received attention [69].

Compared to signature-based approaches, behavior-based approaches ﬁrst learn normal behaviors, and then detect anomalies. This approach has been applied with many research perspectives: application classiﬁcation, anomaly detection, zero day attack detection, network access control [64], and network design [163]. Types of behavior-based approaches include threshold, statistic and learning-based. Levels of behavior-based approaches include ISP- based Internet backbone behavior [50, 194, 195], network behavior [50, 150, 181], user behavior [125], host behavior [50, 87, 120, 173, 180, 194] and application (or protocol) behavior [87, 106, 166]. 39

i p address

daily destination number daily byte number average TTL

port1 ,port2 , ... port1 ,port2 , ...

destination address daily byte n u m daily connection n u m

average duration time port1 ,port2 , ...

destination address

daily byte n u m daily packet number

port1 ,port2 , ...

communication similarity

Listing 3.1: Internet Host Proﬁle (Courtesy of [189]) 40

3.2.5 Visualization

Network visualization provides interactive visual displays for exploration of network traf- ﬁc. It is a challenging task to visualize a large amount of information and provide sufﬁ-

cient level of detail to be meaningful and useful. Visualization can be at different levels of network abstraction (i.e., whole network, individual machine, and between whole network and individual machine), and described by different mechanisms (histogram, chart or

glyph-based and 3D graph). Table 3.4 presents a chronological summary of related studies with their abstract level, mechanism of data processing and visualization, and research

issues. Most applications use statistics and chart methods; whereas few applications use advanced methodologies such as machine learning, graph theory and quad-tree. In terms

of research perspectives, most of them focus on security detection while others provide network monitoring. Besides the approaches summarized in table 3.4, several other projects are worth mentioning. NFSen [16] is an open source, graphical web based front-end tool. It aggregates network traffic by protocols, direction or hosts using charts, and is used for network investigation. AURORA [4] is an IBM research project for traffic analysis and visualization. It was designed for large networks, supports multiple levels of abstraction, and uses chart and graph to visualize traffic, anomaly detection or real time traffic flow. Finally, the Spinning Cube of Potential Doom [105] is a 3D display of network links for anomaly detection and visualized as a cube.

3.2.6 Anonymization

There is a need to anonymize NetFlow data to protect the privacy when the data is shared among parties. There are several approaches for NetFlow speciﬁc anonymization. Slagell et al. [164] proposed an anonymization tool for sharing network logs for computer forensics. 41

Table 3.4: Summary of NetFlow Visualization Applications

Year Abstract Mechanism Perspective 2000 [146] Network Aggregate traffic volume of Network protocol and protocols, chart traffic amount 2001 [57] Multiple Histogram and chart IDS 2004 [197] Multiple Links between machines or IDS domains, graph 2004 [104] Multiple Activities of IP, histogram, Security situational glyph-based graph awareness 2004 [24] Multiple Map between internal and ex- Security ternal traffic, graph 2004 [123] Network Aggregation based on port, Security event detec- chart and graph tion 2005 [51] Network IP aggregation of traffic Worm detection and bursts, chart & graph backbone monitoring 2005 [144] Network Manifold learning, chart Monitoring, detection 2006 [140] Individual Extended the quad-tree,3D Internet traffic navigation and playback 2006 [143] Network Network statistics of proto- Network statistics cols, chart 2006 [152] Multiple Statistic, flow aggregation, IDS chart & graph 2007 [120] Multiple Host behavior, Force-directed Host behavior graph 2008 [126] Individual Graph theory, graph Network traffic 2008 [60] Network TreeMap with splines, chart Network security moni- and graph tor 2008 [173] Multiple Aggreate data per port, 3D Intrusive behavior graph 2009 [162] Network Based on Simple K-Means Detect anomalies clustering, chart 2009 [42] Network Pattern of shape, graph Network attacks 2009 [174] Multiple Aggregate and Map, graph Network monitoring and chart 2009 [71] Multiple Aggregate, tree view, Geo- Network security location, chart and graph 2012 [160] Multiple Sphere Network traffic 42

Their tool can anonymize the common ﬁelds in multiple ways. Similiarly, Foukaraki et

al. [61] proposed an anonymization tool with ﬂexible features and high-performance.

3.2.7 Analysis Systems

As the trafﬁc volume is very large, methodologies to improve performance of capturing, collection, and analysis are needed. There are three commonly used methods to reduce data size: aggregation, ﬁltering, and sampling [52]. In the following, we will survey optimization, sampling, and distributed analysis systems.

3.2.7.1 Optimization

Optimization can be applied in many stages of the NetFlow analysis process: capturing, collecting and analyzing. Bouhtou and Klopfenstein [32] proposed mathematical models to select the NetFlow interfaces based on robust optimizations to deal with probabilistic constraints. Sagnol et al. [157] proposed a method of Successive c-Optimal Design to select NetFlow interfaces and find the optimal sampling rates. Hu et al. [81] proposed an entropy based adaptive flow aggregation algorithm to improve efficiency of storage and

export, and improve the accuracy of legitimate flows. Zadnik et al. [198] proposed an architecture of network flow monitoring adapter based on hardware platform COMBO6, which is able to monitor one million simultaneous flows on an 2Gbps link. Nagaraj et al. [138] proposed an efficient aggregation techniques to speed up querying based on attributes and filter condition of queries.

3.2.7.2 Sampling

Sampling network flow reduces the burden of handling massive volumes of flow data in collection, storage and analysis. Duffield [52] conducted a review of Internet measurement sampling in 2004, focusing on classical sampling methods, new applications and sampling 43

methods, and applications areas. In 2007, Haddadi et al. [75] revisited the issues of Net-

Flow sampling which focuses on data distortion and techniques for the compensation of data distortion.

Sampling methods, impact of sampling, integration of system-wide sampling, and re- covering sampled data from distortion are mentioned in below studies. Duffield et al. [52, 54] developed a size-dependent sampling scheme suitable for billing purposes. Estan et al. [58] proposed an Adaptive NetFlow which dynamically adapts the sampling rate to achieve robustness without sacrificing accuracy. Brauckhoff et al. [33] evaluated the impact of sampling on anomaly detection metrics using flows with the Blaster worm, and found that entropy-based features are less affected. Barlet-ros et al. [25] analyzed the impact of sampling on the accuracy of traffic classification using machine learning methods, and proposed a solution to reduce the impact. Cheng et al. [40] proposed a resource-efficient sampling system that combines three models: a pre-sampling model that records the esti- mated value rather than the measured value, a sampling and holding model that process the sampled packets to update the cache, and a non-uniform sampling model and keep the long

ﬂows in cache. Hao et al. [78] developed a sampling scheme based on sampling two-runs to improve time and memory efﬁciency. Han et al. [76] proposed a pFlours tool that fetches

a packet and performs sampling to eliminate the synchronization problem during network trafﬁc sampling. Dufﬁeld et al. [53] discussed trajectory sampling, methods to eliminate duplications, and methods to join incomplete trajectories. Sekar et al. [159] presented a

system-wide approach that samples as a router primitive. To identify high-rate flow, Zhang el at. [203] developed two methods: fixed sample size test which uses user specified accuracy, and truncated sequential probability test through sequential sampling. Lee et al. [107] proposed a method for related sampling where flows from the same application session are given higher probability. Bartos et al. [27] proposed adaptive, feature-aware statistical sampling techniques to reduce the impact of sampling on anomaly detection. 44

3.2.7.3 Distributed Analysis System

More applications demand real time analysis, advanced detection and classiﬁcation. Cen- tralized analysis systems face the difﬁculties of performance, scalability, and robustness.

Although sampling provides an approach to reduce those burdens, there are tasks that can not be based on sampling data. Distributed systems provide new mechanisms for cap-

turing, accounting and monitoring [134]. Several distributed analysis systems have been mentioned below. Kitatuji et al. [95] proposed a real-time system with a bit-pattern based flow definition and round-robin mechanism to balance packet steams. Sekar et al. [159] proposed cSamp, a monitoring tool based on a coordinating mechanism for flow sampling, hash-based packet selection, and workload distributed. Morariu et al. [135] proposed a distributed IP traffic analysis system. DiCAP [133], a flow capturing system, uses round- robin and a Distributed Hash Table (DHT) to distribute the workload and uses off-the-shelf hardware at network links. DIPStorage [131] is a distributed flow storage platform for IP flow records based on DHT. SCRIPT [132] is a distributed flow analysis framework that distributed flow records equally to multiple nodes. Others used peer-to-peer communication infrastructure [62, 67] and map-reduce for efficient computation. Recent studies use the existing Hadoop based clustering platform and the map-reduce framework. Lee et al. [109] proposed using Hadoop based map-reduce to process packet trace files. Fran- cois et al. [62] proposed botnet detecting system based on Hadoop based clustering and

PageRank. Morken [136] compared two map-reduce frameworks of Apache Hadoop and Nokia Disco, and concluded that Nokia Disco provides fast response time while Hadoop provides rich features, and map-reduce model is a very good approach for ﬂow ﬁltering and

aggregation. 45

3.3 Discussion

Even though a large body of research has focused on trafﬁc ﬂows, many issues remain open.

In particular, NetFlow data analysis is challenging because of the difﬁculty in collecting real data, huge data sets with limited information, and lack of systematic methodologies. In this section, we discuss our view about datasets, research perspectives, methodologies, challenges, and possible future research directions.

3.3.1 Datasets

Because of privacy and other concerns, researchers lack effective trafﬁc ﬂow data sets. Simulated data and other log data have been used as alternatives. Even though there is some real data, this data is either old or does not cover a large enough time period. Acquiring training data sets is another challenge for supervised machine learning. Moreover, there is no publicly available data for comparing different methodologies. Accurate analysis depends on real-time data collection. In surveyed papers , very few discuss a real time data collection solution [67].

Despite the popularity of sFlow and its wide deployment, few papers have focused on sFlow as their data source.

3.3.2 Research Perspectives

Current studies have covered most perspectives of network monitoring, measurement, and network security. Application of network ﬂow data in network monitoring is more successful than in network security, while real time network security is in high demand for network management. Basic top N data is not enough to understand the current complex network security situations. More speciﬁc perspectives such as referring user identity will provide clear information for security and forensic purpose. New perspectives will probably from 46

network security because network security is becoming more important and challenge.

3.3.3 Methodologies

Heuristic approaches are easier to implement and seem more effective than machine learning approaches; however, practical experiences and findings are difficult to gain. Statistical approaches with heuristic methods give accurate results for known situations. For situations involving anomaly, more research is needed to develop advanced approaches that leverage information theory, machine learning and data mining. Much of the work has been limited to specific problems such as port-scan, DoS, or worms. A system that covers wide network security situations is needed for network security administrators. Moreover, visualization needs to focus more on IT operations and provides easy to understand and helpful information. For machine learning approaches, feature selection is a very important step that needs to be specific to the problem. Currently, there is no study available for understanding and comparing the affect of feature selection in the context of NetFlow data. Integrating data from other IT infrastructures will provide more information. As there is no publicly available dataset for comparing different approaches, researchers use their own private data sets in their experiments.

3.3.4 Challenges

With the constantly changing nature of networks, new applications and protocols being added to the Internet, network analysis will have to keep up with the speed of changes. For example, IPv6 addresses can be randomly generated and may not be identified as a unique host or user. Since IPv6 over IPv4 packets can bypass firewalls [73], new approaches for IPv6 measurements are needed. New applications and protocols, faster Internet speeds with increased backbone bandwidth, and more complex content will make the analysis more difficult. In particular, the cloud computing that is based on moving contents to cloud 47

services will make the analysis more complicated. In the following, we discuss speciﬁc challenges.

3.3.4.1 Feature Representation and Selection

Because NetFlow data only provides the header information, representing and selecting a set of appropriate features is challenging. For a speciﬁc task, the key question is how to effectively represent and extract features, and how to select the right features for a speciﬁc problem. With NetFlow version 9, it would be important to effectively leverage these new information.

3.3.4.2 Real Time Analysis and Data Storage

Analysis results need to be available in real time or within some fairly short period of time as the traffic is flowing. Furthermore, data needs to be continuously stored for certain amount of time for future need. Real time data collection is a challenging task because of the data size and the nature of the network traffic. Real time analysis requires understanding the dynamic nature of network traffic. As David [190] pointed out ”that is the face of knowledge in the age of the Net: never fully settled, never fully written, never entirely done”.

3.3.5 Future Directions

Despite signiﬁcant work in the ﬁeld, future research is needed to address the above mentioned challenges.

3.3.5.1 Distributed Data Collection and Analysis

Real time analysis is in high demand in network security. Centralized analysis systems have difﬁculty dealing with huge data and real time analysis. Scalability and robustness 48

are required to analyze data from multiple collectors. New technologies, such as Apache

Hadoop related distributed data collection and analysis systems, open up more opportunities for re-thinking the network trafﬁc analysis. Distributed applications and map-reduce

model will provide more power and bring more insight and understanding.

3.3.5.2 Advanced Analysis Methodologies

Advanced methodologies using behavior-based features have the potential to mine helpful

information. As Sommer et al. [165] pointed out, machine learning algorithms excel at find- ing similarity rather than at identifying anomalous behaviors. To make machine learning approaches more accurate and efficient, there is a need for better understanding of different types of features and heuristics for specific goals. In practice, selecting and understanding an effective set of features is challenging and labor-intensive.

3.3.5.3 Integration

Integrating with existing network infrastructures (e.g., IDS, firewall and VPN gateway), integrating with log file event activities, as well as integrating with host IDS (e.g., meta- events) all show a trend. NetFlow analysis can fill in the gap that IDS, firewall and host- based anti-virus tools can not provide. It can provide monitoring, reporting, security alter- ing, validating policy and configuration, assisting for forensic investigation, and serving as complimentary approaches for other network applications. Correlating with existing network infrastructures (e.g., NIDS may alert for an attack then NetFlow data will validate the alert) can give a high probability factor to remove false positives. Liu et al. [116] proposed a method using Snort logs and NetFlow data fusion with SVMs to create network security awareness. Integrating together with other approaches (such as deep packet inspection), NetFlow-like approaches can provide a breadth-first approach for early investigation, and cover more hierarchies of network layers. 49

Chapter 4. The Big Data Security Sys- tem Design

There are many network flow analysis tools as surveyed in chapter 3. Different from existing systems, as far as we know, this design is the first network flow analysis system that is completely distributed and fully leverages big data technologies. In this chapter, I will discuss the design approach, the components and featrures of the analysis system, and experience I have gone through.

4.1 Approach

The objective of this dissertation is to build a reliable distributed real time system for collecting, storing and analyzing network ﬂow data, log data, and other network security related data. To achieve this, it requires:

• Scalability: The system needs to be scalable to meet future requirement for processing and analyzing large volume of data, including network ﬂow, data from Simple Network Management Protocol (SNMP), security event log, and others. More hard-

ware and software components can be added without downtime or system construc- tion in the future. The system has clusters for multiple nodes, multiple processes,

multiple threads, distributed, and parallel computing.

• Real Time Continuous Monitoring and Interactive Visualization: Traditional security monitoring systems can not process these large amount of data in real time. Real 50

time continuous monitoring is the future of network security monitoring and analysis. TNational Institute of Standards and Technology (NIST) use the term Contin-

uous Monitoring in emphasizing the ongoing monitoring and awareness of network security [10].

Interactive visualization is related to real time monitoring and requires responding to

human input in a timely manner for illustrating the network security situation.

• Intelligent Analysis: The system needs to support more intelligent analysis. In particular, the system models network objects based on host proﬁling and user behavior

to provide a platform for network security analysis including classifying host and identifying a particular user among the other users.

The overall philosophy of the approach is

• Rather than looknig for an unknown and constantly changing object of malware or asking a unclear question of normal or anomaly, I focus on speciﬁc object, i.e., net-

work hosts and users, and ask speciﬁc questions such as: ”what is the role of the network object and does the object behave according to its role?”

• The system leverages the big data technologies to collect, store, and analyze these data.

4.2 Components of the Security Analysis System

This system is multi-layered as in ﬁgure 4.1. As each layer is loosely coupled, change in one component will not affect the other components. Similarly, new models can be added without affecting other modules. The system provides real time continuous network security monitoring and interactive visualization through web user interfaces, near real- time network measurement and advanced analysis. 51

Data Sources

Collector Collecting Producer Producer Producer Aggregating Broker Broker Broker Broker

Consumer Consumer Consumer

Spout Spout Spout Real Time Process Bolt Top N Bolt Bolt Statistic Online Classification Bolt Update Model Bolt ... Bolt

Bolt Bolt Bolt

NoSQL Storage

Gateway Web Server Data Gateway

Administrator & Use Case Web User Off-line Analysis

Figure 4.1: The Big Data Security System Architecture Design 52

4.2.1 Data Collection

I have experimented three distributed data collecting systems: Flume, Flume-ng, and Kafka. The big data community is very active in innovation and moves quickly into new application domains as there are many demands for big data collection and processing. Flume was experimented at ﬁrst. It provides ability for plug-in sources and sinks, and provides web user interfaces for conﬁguration. Unfortunately, it provides very little con-

figurable feature. A Flume client can have only one channel. However, we have many data sources, and one data channel of one Flume client is not sufficient. Then I experimented with Flume-ng, which is the next generation Flume that provides more configurable features and channels for a client. Flume-ng provides mechanisms fro our need of multiple data channels and configuration. Lastly, I experimented with Kafka [99]. Kakfa is considered more stable since it has been in the product line of large scale dynamic systems such as LinkedIn. Compared with the Flume and Flume-ng, Kafka is more efficient in memory management and supports multiple source producers, brokers, and consumers. Kafak also provides ability for real time analysis. I consider Kafka as the right solution for data collection in this dissertation.

4.2.2 Data Storage

Cassandra is the choice of NoSQL data storage for storing sFlow, NetFlow, and security event logs in this system. Compared with other NoSQL solutions, we select Cassandra because:

• It is scalable with nodes of clusters range from several to hundreds.

• It has build-in index mechanisms: the primary index, the secondary index, and the composite column. 53

• It is optimized for fast writing and reading.

• There are many client programming interfaces, tools, and third party index tools.

These Cassandra features support our goals of data storage and data schema with:

• Query by source or destination IP

• Query by time, protocol, source or destination port number

• Top network consumer (in term of package size)

• Top talkers (in term of connection counter)

• On-line detecting of port scanning

• Storing data for future off-line advanced analysis and scientiﬁc research

The experience with Cassandra is not very pleasant. We have experienced several release versions, and re-organized the cluster structure to meet the Cassandra hardware requirements. At this moment, the system uses Cassandra version 1.2.5. It is more stable compared to previous versions experimented in this system. Cassandra is a complex NoSQL solution and may take years to learn best practices of running Cassandra clusters.

4.2.3 Security Gateway

Protecting the big data is a critical challenge. There are many secure approaches inside the data clusters: nodes validation, client validation, data encryption, key management, and more. It is also important to protect the data from external adversaries and the security gateway protects the big data from outside access. 54

4.2.4 Data Processing

Network ﬂow data are processed in three levels: real time data aggregation in Kafka data collection, real time detection in Storm, and off-line advanced data analysis as shown in

figure 4.1. Kafka works as data collector and basic counter aggregator. Storm provides the online real time computation of tasks that need more computational resources such as network object online classification and identification. Finally, off-line advanced data processing is provided by Pig, R, Java, and C++ models.

4.2.5 User Interfaces

There are two types of user interfaces in the system. The data gateway is for administrators and off-line advanced network object modeling by using Pig, R, Java, C++, CQL, and

Cassandra console command lines. The web interfaces are for network security administrators who are not necessarily familiar with these tools or systems. The web interfaces

provide real time query to the network status, network security awareness, and interactive visualization.

4.3 Features

In this section, I discuss the features that this system supports.

4.3.1 Real Time Continuous Network Security Monitoring and Inter-

active Visualization

Network security administrators want to know what is happening in the network in real time continuously, and be aware of any security events. There are two categories of net- 55

work monitoring. One is normal network status such as trends of protocol usage, network throughput, top conversation, and top usage of sub-networks. Another aspect is hacking

activities such as port scanning and operating system ﬁngerprinting. For better understanding and analysis, these information need to be visualized in an interactive way. The detail

of real time continuous network security monitoring and interactive visualization are discussed in Chapter 5.

4.3.2 Network Measurement

There are needs for measuring the network for some statistic results or reports that are not existed in the data storage. One example we did is measuring the anonymity network usage

on the Internet. The detail of campus anonymity network usage measurement is discussed in Chapter 6.

4.3.3 Advanced Network Modeling

As network security becomes complicated, advanced methods and modeling are applied to analyze the network. For instance, role-based network object classiﬁcation and user

identiﬁcation provide a new approach to detect malicious activities such as intrusion, and is discussed in Chapter 7.

4.4 Discussion

The general trend of network ﬂow analysis is toward distributed systems. Big data technologies provide not just the computing power of distributed analysis but also methods to handle the large volume of data in real time. In this chapter, I present a system design that fully leverages the open source big data projects of Kafka, Storm, and Cassandra. The system supports real time and continuous network ﬂow analysis for network security. Com- 56

pared with systems discussed in Chapter 3, the system is a complete network ﬂow analysis system, more scalable, and low cost due to use of open source projects. The system components are relatively independent and is very easy to adopt the system for different goals. With additional nodes, the system can be scaled to handle more data for bigger organiza- tions. 57

Chapter 5. Real Time Continuous Network Monitoring and Interactive Visualization

Neither network monitoring nor interactive visualization is new. However, it is challenging to monitor network data in real time continuously and to visualize the information interactively in web browser for network security purpose. There are many factors that drive the need for real time continuous network monitoring. It is critical for network security administrators to know the information in real time continuously. Traditional approaches of data warehouse and data processing are infeasible to access a large amount of data in real time. The big data security analysis system design in Chapter 4 makes it feasible. In this chapter, I present how the real time continuous network monitoring and interactive visualization are achieved for these large volume of security data in web user interface. I focus more on methodology and theory rather than best user interfaces and rich features which can be done at development time. I demonstrate:(1) querying the campus network host status in real time continuously, (2) monitoring the network trends in real time and

(3) visualizing these information in interactive approaches in real time. For security and privacy reasons, the real identity is replaced by text description in the following web user interfaces. 58

5.1 Real Time Network Host Querying

The network ﬂow information of each host is sliced by time such as hourly or daily, and is

stored in the Cassandra data storage by IP address as the key. The Cassandra stores the whole row in a ﬁle and a disk, and is indexed by its key. Accessing these information is direct in O(1). It is still O(1) when the query is across several time slices. Hence, it is very rare to query the host across a big number of time slices, which is the limitation of the design. So it is important to ﬁnd a proper time slice for the most often used scenarios.

Figure 5.1 demonstrates that overall it took 56 milliseconds for querying 24 hours network ﬂow information of a host.

The network flow information can be presented in many levels after accessing in real time. Figure 5.1 presents an example with three levels: raw data, statistics, and network. Raw data are used for investigating detailed network flow information by experienced network security administrators. Statistic summary is used for providing brief information and is computed at teh web server to be presented to the web browser. These summary items are configurable. List 5.1 is an example of items in the summary. Network graph is used to allow users interactively investigate and track user connections with other hosts. Figure 5.2 shows the network for the target, its sources, and its destination. These are the most important information for security administrators. It provides drag-and-drop interactive ability so that the user can continuously trace the network. More detailed discussion of interactive network visualization is at Section 5.3. 59