Mining Web User Behaviors to Detect Application Layer Ddos Attacks

JOURNAL OF SOFTWARE, VOL. 9, NO. 4, APRIL 2014 985

Mining Web User Behaviors to Detect Application Layer DDoS Attacks

Chuibi Huang1 1Department of Automation, USTC Key laboratory of network communication system and control Hefei, China Email: [email protected]

Jinlin Wang1, 2, Gang Wu1, Jun Chen2 2Institute of Acoustics, Chinese Academy of Sciences National Network New Media Engineering Research Center Beijing, China Email: {wangjl, chenj}@dsp.ac.cn

Abstract— Distributed Denial of Service (DDoS) attacks than TCP or UDP-based attacks, requiring fewer network have caused continuous critical threats to the Internet connections to achieve their malicious purposes. The services. DDoS attacks are generally conducted at the MyDoom worm and the CyberSlam are all instances of network layer. Many DDoS attack detection methods are this type attack [2, 3]. focused on the IP and TCP layers. However, they are not suitable for detecting the application layer DDoS attacks. In The challenges of detecting the app-DDoS attacks can this paper, we propose a scheme based on web user be summarized as the following aspects: browsing behaviors to detect the application layer DDoS • The app-DDoS attacks make use of higher layer attacks (app-DDoS). A clustering method is applied to protocols such as HTTP to pass through most of extract the access features of the web objects. Based on the the current anomaly detection systems designed access features, an extended hidden semi-Markov model is for low layers. proposed to describe the browsing behaviors of web user. • Along with flooding, App-DDoS traces the The deviation from the entropy of the training data set average request rate of the legitimate user and uses fitting to the hidden semi-Markov model can be considered as the abnormality of the observed data set. Finally the same rate for attacking the server, or employs experiments are conducted to demonstrate the effectiveness large-scale botnet to generate low rate attack flows. of our model and algorithm. This makes DDoS attacks detection more difficult. • Burst traffic and high volume are the common Index Terms—HsMM, web user behaviors, DDoS, DDoS characteristics of App-DDoS attacks and flash Attacks , clustering crowds. "Flash crowd" refers to the situation when a very large number of users simultaneously access a popular Website, which produces a surge I. INTRODUCTION in traffic to the Website and might cause the site to Distributed Denial of Service (DDoS) attacks have be virtually unreachable [4]. It is not easy for become a serious problem in recent years. Generally, current techniques to distinguish APP-DDoS DDoS attacks are carried out at the network layer, such as attacks from flash crowds. ICMP flooding, SYN flooding and UDP flooding. From the literature, few exiting researches focus on the Without advance warning, a DDoS attack can easily detection of app-DDoS attacks during the flash crowd exhaust the computing and communication resources of event. This paper introduces the hidden semi-Markov its victim within a short period of time [1]. Because of the model (HsMM) to capture the user browsing patterns seriousness and destructiveness, many studies have been during flash crowd and to implement the app-DDoS conducted on this type of attacks. A lot of effective attacks detection. Our contributions in this paper are schemes have been proposed to protect the network and threefold: 1) We use the clustering method to extract the equipment from bandwidth attack, so it is not as effortless web objects' access features, which can well portray as in the past for attackers to launch the network layer current web user browsing behaviors. 2) We apply hidden DDoS attacks. In order to dodge detection, attackers shift semi-Markov model (HsMM) to describe the dynamics of their offensive strategies to application-layer attacks and access features and to implement the detection of app- establish more sophisticated types of DDoS attacks. They DDoS attacks. 3) Experiments based on real traffic are utilize legitimate application layer HTTP requests from conducted to validate our detection scheme. legitimately connected network machines to overwhelm The organization of the paper is as follows. In the next web server. These attacks are typically more efficient section, we review the related work of our research. In

Section 3, we explain our model and algorithm to detect crowd and app-DDoS attacks are unstable, bursty and the app-DDoS attacks. Experiment results are presented huge traffic volume. The app-DDoS attackers are in section 4. Finally, we conclude our paper in Section 5. increasingly moving away from pure bandwidth flooding to more surreptitious attacks that hide in normal flash II. RELATED WORK crowd of the website. It is a challenge to detect the app- DDoS attacks when they occur during a flash crowd Most current DDoS-related researches are conducted event. on the IP layer or TCP layer instead of the application To meet the above challenges, we focus on analysis of layer. These mechanisms typically utilize the specific the user behaviors. In this paper, we assume that it is network features to detect attacks. Mirkovic et al. [5] impossible for app-attacks to completely mimic the proposed a defense system called D-WARD located in normal web user behaviors. This assumption is based on edge routers to monitor the asymmetry of two-way packet the following consideration. Generally, web user access rates and to detect attacks. Yu et al. [6] discriminated behaviors can be described by three factors: HTTP DDoS attacks from flash crowds by using the flow request rate, page retention time, and request sequence. correlation coefficient as a similarity metric among The attackers can mimic the normal access behavior by suspicious flows. Zhang et al. [7] proposed the launching attacks with similar HTTP request rate and Congestion Participation Rate (CPR) metric and a CPR- retention time as the normal users. However, the app- based approach to detect and filter the TCP layer DDos DDoS attacks cannot simulate the dynamic process of the attacks. Cabrera et al. [8] depend on the MIB traffic web user behavior or the request sequence, because it is a variables collected from the systems of attackers to pre-designed routine and cannot capture the dynamics of achieve the early detection. Yuan et al. [9] capture the the users and the networks. Another assumption of our traffic pattern by using cross correlation analysis and then paper is that the user behavior can be described by the decide where and when a DDoS attack possibly arises. distribution of web objects popularity. Then the variation Soule et al. [10] used the traffic matrix, which represents of web objects popularity could represent the dynamic the traffic state, to identify various dynamic attacks at changing process of user behavior. Since app-DDoS early stage. In [11], DDoS attacks were discovered by attacker is unable to obtain historical access records from analyzing the TCP packet header against the well-defined the victim server, it cannot mimic the dynamic of normal rules and conditions and distinguished the difference user behaviors. We consider the app-DDoS attacks as between normal and abnormal traffic. In [12], attackes anomaly browsing behavior. Moreover, we also assume are detected by computing the rate of TCP flags to TCP that normal users always access the "hot webpages". packets received at a web server. However, there are little However, the attackers access the "cold webpages”. DDoS defense methods that utilize the application layer Therefore, we could build the browsing behavior model information. In [19], a suspicion assignment mechanism for both attacks and normal user by monitoring the Ranjan et al. [19] used statistical method to detect time dynamic change of the web objects popularity. Using this related characteristics of HTTP sessions, such as request model, we could distinguish the app-DDoS attack from interarrival time, session inter-arrival time and session normal user. arrival time. Yen et al. [21] defended the application DDoS attacks with constraint random request attacks by IV. MODEL AND DETECTION PRINCIPLE the statistical methods. Kandula et al. [3] designed a system to protect a web cluster from DDoS attacks by (i) A. Hidden Semi-Markov Model optimally dividing time spent in authenticating new Hidden semi-Markov model (HsMM) [14] is an clients and serving authenticated clients (ii) using extension of hidden Markov model (HMM) [23] with CAPTCHAs designing a probabilistic authentication variable state duration. It is a stochastic finite state mechanism, but the task of requiring users to solve machine, specified by λ = (,,,,Q π ABP )where: graphic puzzles causes additional service delay. As a • Q result, the graphic puzzle cause annoying legitimate users is a discrete set of hidden states with cardinality = ∈ as well as act as another DDoS attack points. [16] M , i.e., QN{1,...,} . qQt denotes the introduced a web browsing model represented by the state that the system takes at time t ; transition of the click pages. In a few previous studies, • π is the initial state probability distribution, i.e., analysis of user behaviors have been applied in many π =∈{π | mQ} π ≡=Pr⎡⎤qm . q research fields [13, 15, 17, 20, 22]. m , m ⎣⎦1 t denotes the state that the system takes at time t III. MODEL PRELIMINARIES AND ASSUMPTIONS and mQ∈ . The initial state probability π = 1; When detecting the app-DDoS attacks, we are faced distribution satisfies ∑ m m with the following challenges: 1) attacker may launch an • A is the state transition matrix with probabilities: app-DDoS attack by mimicking the normal web user a ≡=Pr⎡⎤qnqmmnQ | = , , ∈ ,and access behavior, so the malicious requests differ from the mntt⎣⎦−1 legitimate ones but not in traffic characteristics. the state transition coefficients satisfy Therefore, most current detection mechanisms based on a = 1; ∑ n mn traffic characteristics become invalid 2) Both the flash

• B is the output probabilities matrix bk()≡ ατ(,)md== Pr⎡ ot ,(, q ) (,). md⎤ m ttt⎣ 1 ⎦ (2) JJKJJK JJK Pr ⎡⎤oqm==v | , mQ∈ o A transition into state (,)qmdτ = (,)takes place ⎣⎦tkt and t tt t either from (,)(,1)qmdτ =+ or from denotes the observedJK JJvectorK at time , taking tt−−11 (,)q τ = (,1)n nm≠ values from {vv1,..., K } , K is the size of the tt−−11 for some . Therefore, we observable output set. For a given state m, could obtain the following forward recursion formulas: αα(,)md=+ (, md 1)( b o ) bk()= 1; tt−1 mt ∑ k m

• P is the state duration matrix with probabilities: +⋅()α (,1)na bopd ( ) (), d ≥ 1 (3) p ()ddqm≡==Pr⎡⎤τ | , τ ∑ tnmmtm−1 mtt⎣⎦ t denotes

q the remaining time of the current state t , απ= 11(,)mdmm b ( o ) p m (). d (4) ∈ ∈ mQ, dD{1,...,} , D is the maximum The backward variable is defined as follows: interval between any two consecutive state βτ(,)md== Pr⎡ oT ,(, q ) (,). md⎤ tttt⎣ +1 ⎦ (5) transitions, and the state duration coefficients It can be seen that when d > 1 the next state must be satisfy pd()= 1. ∑d m τ =− = (,)(,1)qmdtt++11 , and when d 1 it must be ()q ,τ Then, if the pair process tt takes on value τ = ≠ ≥ (,)(,)qndtt++11 ' for some nm and d ' 1 . ()md,,the semi-Markov chain will remain in the current We thus obtain the following backward recursion formula: +− ββ=−> state m until time td1 and transits to another tmtt(,)md b ( o++11 ) (, md 1), for d 1 (6) state at time td+ , fordD∈ {1,..., } . It is generally and ⎛⎞ not observable for these states. The observable variables ββ(,1)mbopdnd= ( ) () (,) JK JJK tmnntnt∑∑a ++11⎜⎟ (7) = nm≠≥ d1 are a series of observations Oo()1,..., oT where ot ⎝⎠ Especially whentT= : denotes the observable output at time t and T is the β = ≥ T(,)md 1, d 1. (8) number of samples in the observed sequences. bo() mab| Three joint probability functions can be expressed in represents the observation sequence from time a to time terms of the assumed model parameters and the forward JJK and backward variables defined above: b (i.e.,{otb : a ≤≤}) . When the “conditional t ξ (,)mn=== Pr⎡ oT , q mq , n⎤ ttt⎣ 11− ⎦ independence” of outputs is assumed, bo()= mba| ⎛⎞ =⋅αβ(,1)mabo ( ) pd ()(,) nd b tmnntnt−1 ⎜⎟∑ ∏ bo() d ≥1 t = a mt. ⎝⎠ (9) B. Problem Formulation ητ=≠==T (,)md Pr⎡ o , q− mq , m , d⎤ We use the hidden semi-Markov model to describe the tttt⎣ 11 ⎦ dynamic changing process of web user behaviors. The ⎛⎞ = αβ(,1)na bopd ( ) ()(,). md hidden state q is used to present the distribution of web ⎜⎟∑ tnmmtmt−1 t ⎝⎠nm≠ objects’ popularity, namely the user behavior, at timet . (10) In general, the distribution of web objects’ popularity is T γ ()moqm== Pr⎡ ,⎤ . (11) unobservable. The observable output is the web objects’ tt⎣ 1 ⎦ click rates: Then, the model parameters can be re-estimated by the following formulas: cit Click = (1) T it N qqmoˆ== arg max Pr⎡ | ⎤ c tt1≤≤mM ⎣ 1 ⎦ ∑i =1 it = γ =− arg maxt (m ), for tTT, 1,...,1. where cit is the click number of the ith time unit, and 1≤≤mM (12) N is the number of the web server’s objects. p m ()d γ ()m is the user behavior retention time distribution. The πˆ = 1 m N (13) change of web user behaviors can be considered as a γ ()n ∑ 1 transition of the hidden state (i.e., from qt −1 to qt ) . We n =1 can estimate the parameter with the following forward and back algorithm [14]. The forward variable is defined as follows:

T the app-DDoS attack to the victim server. But we also ξ (,)mn ∑ t assume that the app-DDoS attacks cannot simulate the ˆ = t =1 amn TN (14) dynamic process of the web user behavior or the request γ ()n sequence, because it is a pre-designed routine and cannot ∑∑ t tn==11 capture the dynamics of the user and the networks. Thus, when the attack begins, each compromised node replays a T snippet of another historical flash crowd trace. The η (,)md = ∑t =1 t interval between two consecutive attack requests is pdˆ(m ) (15) DTη (,)md decided by the attack rate. In our experiments, we ∑∑dt==11t simulate constant rate attack and increasing rate attacks. T γδ()(mo− v ) We set the time unit to 5s and group 12 consecutive bˆ()= ∑t =1 ttk mkv T (16) observations into one sequence. The "moving" step is one γδ()(mo− v ) ∑∑kt=1 ttk time unit and a new sequence is formed using the current observation and the preceding 11 observations. Where {vk } is the set of observable values, and if Algorithm 1 K = δ −= δ − The -means Clustering otkv , ()1otkv , otherwise ()otkv Input: Web objects' click rates dataset D = 0 . The average entropies AE of observed sequences Number clusters K fitting to the HsMM model are used to detect app- Output: Set of cluster representatives C DDoS attacks: = T λ Cluster membership vector m EntropyPr[ o1 | ] /*Initialize cluster representativesC */ ==Pr[oqT ,( ,τλ ) ( md , )| ] ∑ 1 TT (17) md, Randomly chooseK data pointsfrom D ln(Pr[oT | λ]) Use theseKC data points as initial set of ALE = 1 . (18) T repeat /*Data Assignment*/ C. Clustering Method Reassign points inD to closet cluster mean Clustering is the task of grouping a set of objects in Updatem such that m is cluster ID ofi th such a way that objects in the same group are more i similar to each other than to those in other groups. It is a point in D main task of exploratory data mining, and a common /*Relocation of means*/ technique for statistical data analysis used in many fields, including machine learning, pattern recognition, image UpdateCc such thatj is means of points analysis, information retrieval, and bioinformatics. inj th cluster Since the number of web objects is enormous, it is difficult to deal with the multidimensional observed data Until convergence of objective function without mass computation when training the hidden semi- N 2 (arg minx-c ) Markov model. Thus, we apply clustering method to ∑ i=1 j ij2 reduce the dimension of observed data. The specific Figure 1. The Clustering Algorithm clustering algorithm is shown in Fig.1. We assume the cluster results as: = A. Attacks during Flash Crowd Cccc{ 12, ,...,M } . (19) The emulation process lasts about 6 h. The first 2 h = Num{ n12, n ,..., nM } . (20) data are used to train the model, and the remaining 4 h of data including a flash crowd event are used for test. The emulated app-DDoS attacks are mixed with the trace Where C denotes the M classes, and n denotes the i chose from the period of [3.5h, 5.5h]. number of the web objects in ci . Then we calculate the Fig.2 shows the average entropies of observations click rates of each class according (1). varying with the time, when constant rate attacks are emulated. Curve a represents the dynamic entropy VI. EXPERIMENTS varying process of normal flash crowd and curve b represents the dynamic entropy varying process of flash We simulate 1200 client nodes which play as normal crowd mixed with constant rate app-DDoS attacks. We users from the semifinals of FIFA WorldCup98 [18]. We can see that the average entropy of observations does not randomly select 15% of these nodes as compromised change much during the flash crowd period of [2h, 3.5h], nodes. Furthermore, we assume the attackers can which implies that the main web user behaviors do not intercept a small portion of requests from normal web have obvious varieties during the flash crowd event. users and make the same request or "hot" pages to launch

However, during the period of [3.5h, 5.5h], constant rate the distributions of average entropy. It is easy to see that attacks appear and the average entropy of observations there exist significant differences in entropy distributions decreases sharply. In the duration of constant rate attack, between two groups: the entropies of normal web traffic there is significantly deviation from the average entropy are larger than -3, but most entropies of the traffic of normal observations. Therefore, we could make use containing attacks are less than -4. Therefore, we could of this phenomenon to detect app-DDoS attack. make use of this result to identify the app-DDoS attacks Fig.3 shows the average entropy of observation varying from the normal web traffic when conducting DDoS with the time, when increasing rate attacks are emulated. detection. As shows in Fig.5, if we take -2.7 for the Curve a represents the dynamic entropy varying process threshold value of normal web traffic’s average entropy, of normal flash crowd and curve b represents the the false negative rate (FNR) is about 2%, and the dynamic entropy varying process of flash crowd mixed detection rate (DR) is about 97%. We can see that the with increasing rate app-DDoS attacks. As shows in Fig.3, algorithm can correctly detect the app-DDoS attacks during the period of [3.5h, 5.5h], increasing rate attacks which happened with a flash crowd event by making use appear and the average entropy of observations decreases of the dynamic HsMM models of web user behaviors. gradually. At the end of increasing rate attacks, the average entropy increases gradually to the value range of the normal case. Similar to the situation described above, The distributions of average entropy the average entropy in the duration increasing rate attacks is significantly smaller than the one in normal flash 0.15 Normal crowd period. Attacks

Detection for constant rate attack 0.1

Frequence 0.05 -2 ↑ a -4 ← b 0 -7 -5 -3 -1 0 entropy entropy -6 Figure 4. Distributions of average entropy a(flash crowd) b(flash crowd with attack) -8 1 0 900 1800 2700 3600 4500 FNR index of time(*5s) DR (-2.7,0.97) Figure 2. Entropy versus time of constant rate attack 0.8

0.6 Detection for increase rate attack 0.4

0 FNR/DR

0.2 -2 (-2.7,0.02) ↑ a 0 -4 ← b -7 -5 -3 -1 0 entropy entropy Figure 5. Cumulate distribution of average entropies -6 a(flash crowd) VII. CONCLUSION b(flash crowd with attack) -8 In this paper, we applied the hidden semi-Markov 0 900 1800 2700 3600 4500 model to describe the web user behaviors which can be index of time(*5s) Figure 3. Entropy versus time of increasing rate attack represented by the click rates of web objects. By training the observed data of normal web traffic with forward and backward algorithm, we obtain the hidden semi-Markov B. Performance model parameters. The average entropies of observed In the above scenarios, based on the average entropy of sequences fitting to the HsMM model are used to detect observations fitting to the HsMM model, we can detect app-DDoS attacks. The experiments show that there is the abnormity caused by the DDoS attack. Fig.4 shows obvious obviation from the average entropies of normal

observations in the duration of app-DDoS attacks. In order to reduce the amount of calculation in training [9] J. Yuan, K. Mills, “Monitoring the macroscopic effect of HsMM model, we apply clustering method to reduce the DDoS flooding attacks,” IEEE Trans. Dependable and dimension of observed data. In addition, in future we can Secure Computing, v 2, n 4, p 324-335, October 2002. [10] A. Soule, K. Salamatian, N. Taft, “Combining filtering and consider how to cope with the app-DDoS attacks statistical method for anomaly detection,” In Proceedings launched by requesting dynamic webpage. of Internet Measurement Conference, p 331-344, 2005. [11] L. Limwiwatkul, A. Rungsawangr, “Distrubited denial of ACKNOWLEDGMENT service detection using TCP/IP header and traffic measurement analysis,” In Proc. Int. Symp. Commun. Inf. This work and related experiment environment is Technol., Sappoo, Japan, p 605-610, October 2004. supported by the National High Technology Research and [12] S. Noh, C. Lee, K. Choi, G. Jung, “Detecting Distributed Development Program of China under Grant No. Denial of Service (DDoS) attacks through inductive 2011AA01A102, the National Key Technology R&D learning,” Lecture Notes in Computer Science, v 2690, p Program under Grant No. 2012BAH18B04 and the 286-295, 2003. Strategic Priority Research Program of the Chinese [13] Jun. Cai, Shunzheng. Yu, Yu. Wang, “The community Academy of Sciences under Grant No. XDA06010302. analysis of user behaviors network for web traffic”, Journal We are sincerely grateful to their support. of Software, v 6, n 11, p 2217-2224, 2011. [14] S. Z. Yu, H. Kobayashi, “An efficient forward-back algorithm for an explicit duration hidden Markov model.” REFERENCES IEEE Signal Process. Lett., v 10, n 1, p 11-14, Jan 2003. [1] C. Douligeris, A. Mitrokotsa, “DDoS attacks and denfense [15] Hui. Chen, “Relationship between motivation and behavior mechanisms: Classification and state-of-the-art,” Computer of SNS User”, Journal of Software, v 7, n 6, p 1265-1272, Networks: The Int. J. Computers and Telecommunications 2012. Networking, v 44, n 5, p 643-666, Apr 2004. [16] Y. Xie, S. Yu, “A large-scale hidden semi-Markov model [2] “Incident Note IN-2004-01 W32/Novarg. A virus,” CERT. for anomaly detection on user browsing behaviors”, IEEE [Online]. Available: http://www.cert.org/incident_notes/ Transaction on Networking, v 17, n 1, February 2009. IN-2004-01.html [17] Wei. Yu, Shijun. Li, Yunlu. Zhang, Zhuo Zhang, “Mining [3] S. Kandula, D. Katabi, M. Jacob, A. W. Berger, “Botz-4- users similarity of interest in web community”, Journal of Sale:Surviving Organized DDoS Attacks that Mimic Flash Computers, v 6, n 11, p 2357-2364, 2011. Crowds,” MIT, Tech. Rep. TR-969, 2004[Online]. [18] [Online]. Available: http://ita.ee.lbl.gov/html/traces.html. Available: http://www.usenix.org/events/nsdi05/tech/kand- [19] S. Ranjan, R. Swaminathan, M. Uysal, E. Knightly, ula/kandula.pdf. “DDoS-resilient scheduling to counter application layer [4] Yi. Xie, Shunzheng. Yu, “Monitoring the application-layer attacks under imperfect detection,” In Proceedings of IEEE DDoS attacks for popular websites,” IEEE/ACM INFOCOM, April 2006. Transactions on Networking, v 17, n 1, February 2009. [20] Qingzhang. Chen, Yanqing. Ou, Hang. Sun, “Design and [5] J. Mirkovic, G. Prier, P. Reiher, “Attacking DDoS at the implement of customer communication behavior analysis source,” In Proc. Int. Conf. Network Protocols, p 312-321, system”, Journal of Software, v 6, n 8, p 1484-1491, 2011. 2002. [21] W. Yen, M.-F. Lee, “Defending application DDoS with [6] Shui. Yu, Wanlei. Zhou, Weijia. Jia, Song. Guo, constraint random request attacks,” In Proc. Asia-Pacific “Discriminating DDoS attacks from flash crowds using Conf. Commun., Perth, Western Australia, p 620-624, flow correlation coefficient,” IEEE Transactions on October 2005. Parallel and Distributed System, v 23, n 6, June 2012. [22] Xiaohua. Hu, Tao. Mu, Weihui. Dai, Hongzhi. Hu, [7] Changwang. Zhang, Zhiping. Cai, Weifeng Chen, “Flow Genghui. Dai, “Analysis of browsing behaviors with ant level detection and filtering of low-rate DDoS,” Computer colony clustering algorithm”, Journal of Computers, v 7, n Networks, v 56, n 15, p 3417-3431, October 2012. 12, p 3096-3102, 2012. [8] J. B. D. Cabrera, L. Lewis, X. Qin, W. Lee, R. K. Prasanth, [23] L. R. Rabiner, “A Tutorial on hidden markov models and B. Ravichandran, R. K. Mehra, “Proactive detection of selected applications in speech recongnition,” In Proc. of distributed denial of service attacks using MIB traffic IEEE, v 77, n 2, p 257-286, February 1989. variables a feasibility study,” In Proc. IEEE/IFIP Int. Symp. Integr. Netw. Manag., p 609-622, May 2001.