The Model of Trojan Horse Detection System Based on Behavior Analysis

The Model of Trojan horse Detection System Based on Behavior Analysis

Shumei Zhao Yanru Jia Zhengzhou Railway Vocational & Technical College Zhengzhou Railway Vocational & Technical College Zhengzhou, China Zhengzhou, China [email protected]

Abstract—This paper presents a Trojan-detection system model technology the Trojans can attain the regular executable path, based on behavior analysis. Through the abstract description of and then the system run the predefined functions, to hold up the the Trojan's behavior, according to certain rules to establish a Trojan. However, this method is difficult to detect the Trojan behavior feature database, and combining the heuristic analyzer which directly modifies the core of the system. to further analysis and judge whether the program is the Trojans, then do the appropriate processing at last. The In order to overcome these drawbacks of traditional Trojan experiments show this method is effective and efficient in real detection technology, Many researchers began to think about time and light load, and can be used to in practice to monitor the the Trojan detection technology in terms of behavior[4], which computer system in real time. is in according to the behavioral characteristics of Trojan horse programs (such as modify the registry, register system services, Keywords-Trojan; behavioral features; system call; behavior modify system files, etc.) to judge whether it is suspicious or analysis not. But comparing with existing anti-Trojan technology, it now just stay in the realm of theory. This paper presents the I. INTRODUCTION detection based on behavioral analysis, which can detect and judge the known and unknown Trojan. The Internet do not only bring great convenience to people, but also bring various security threats, such as computer Invoking function Heuristic Analysis results System Object of behavior viruses, hacker attacks, Trojan horses flood and so on., which analyzer responds [1] of the attacks from the Trojan are increasing. It is reported Registry that the new Trojan has explosion of growth in China, 2008, the total number has exceeded 10 million. The modularization File and professional of Trojan manufacturing and the internalize in Port The database of operating the virus are the three major features of the computer behavior feature virus development in China in 2008.Meanwhile, the Trojan's System service creator's "profit-driven" still has not changed, the web with horse and hole attack become the main channel for hackers Process profiting. So we must analyze the working principle technology of Trojans to prevent the Trojan do harm to our computer and Other network..At present, people take more attention to computer Figure 1. System model viruses research, but a little to Trojans. Now it is urgent to research how to prevent and detect known and unknown Trojan. II. THE SYSTEM MODEL DESIGN The Trojan-detection system model based on behavior At present, there are two kinds of technologies which can analysis is illustrated in Figure 1. The model is up to the CIDF detect Trojan horse: The first, A Trojan horse detection [2] standard. The system is composed with four modules, which technology based on features . This method can detect the are monitoring object, the database of behavior features, the Trojan program by the feature strings of the Trojan appearing heuristic analysis and the system response. The module of in the system, which method has high accuracy rate and low monitoring object can monitor all the objects which are false alarm rate for pre-existing Trojan horse. But it can’t be operated by the system. The behavior database records the used to detect unknown Trojans. The second: A Trojan horse [3] Trojan’s behavior features, and gives the features a value, in detection technology based on system call hook . In order to order to judge whether the program is Trojan. The module of hidden themselves successfully, the programs of Trojan must heuristic analyzer analyzes the information which is transferred modify the operating system execution path, or directly modify by the database of the behavior features, and gives judgment. the operating system information which are stored about the The module of system response can deal with the judgment. process, drive, network connection. With the system call hook

U.S. Government work not protected by U.S. copyright III. MODULE DESIGN This function has three parameters: The first parameter is pre-defined key value of the root key; the second parameter is A. The module of monitoring behavior object the primary key name, which include its path; The third Before analysis the Trojan's behavior, the system should parameter is a long integer pointer,. if the function returns 0, it summarize and classify the objects of the Trojan operate. This will save handle of the key. While The “RegCreateKey” call paper lists several common behavior objects, such as the “NtCreateKey” function in the Ntdll.dll, meanwhile, the registry, files, ports, processes, system services and so on. “NtCreateKey” has set parameter already, it can entry into the There are descriptions about these behavior objects of Trojan. kernel state by soft interrupt, and then request the Which are shown in Table 1. corresponding service. Because that there is a matching function start with Zw corresponds to the system services function in the Ntdll.dll, the system can find the TABLE I. THE DESCRIPTION OF THE TROJAN’S BEHAVIOR FEATURE “ZwCreateKey” function through the interrupt service number The description of the Trojan’s behavior in the table of the system service dispatch, and then execute Object of behavior feature this function. Therefore, when we use Hook function, we can Modify the startup items, related items and so Registry put the function pointer in the system dispatch table forward to on. our own function “MyZwCreateKe”, and then the program can Modify system files; Bundle the startup file; execute the “MyZwCreateKey” function first. In this function, File Copy Autorun.inf file; Copy files to the self- starting items; Add or delete files and so on. we can deal with the preprocessing function, and then execute Port Open or close ports. original “ZwCreateKey” function, or skip this function to Create process; Remote thread injection; Hidden execute the following codes. Process process and so on The operation of monitoring files: we take monitoring the Modify their own path, register themselves as System service operating of copying file in the system as example, and use system services “CopyFile” function, which function prototype is that: Record keyboard, screen interception operations Other and so an. Declare Function CopyFile Lib "kernel32" Alias In this strategy, we use the technology of the Windows API "CopyFileA" (ByVal lpExistingFileName As String, ByVal HOOK to monitor the behavior objects. The following is the lpNewFileName As String, ByVal bFailIfExists As Long) As introduction about the technology of Windows API HOOK Long application in this strategy. This function has three parameters, the first parameter API HOOK refers that before the application call the real “lpExistingFileName” is source file name; the second system API function, it would do some corresponding and then parameter “lpNewFileName” is target file name; the third call the real API function. In essence, The API HOOK parameter “bFailIfExists” is a long integer pointer, if it is set up Technology changes the execution path of normal procedure. It true , once the target file has exist, the system call will fail, or can intercept or monitor some information about executable the target file name will be modified. Likewise, in order to code in the execution process, which can change the behavior monitor this file operation, the system should define hook of the operate system, and also can help designers to function about copying file, which is “myCopyFile.” While understand the structure and operate mechanism of the system, using the hook function, we must put the function pointer in the so this technology is widely used in time tracking and system service dispatch table toward to self-defined function modifying system behavior. “myCopyFile”. At this time, when the program executes the The Trojan behavior is comprised of sections of program system service dispatch, it can execute function “myCopyFile”. code. If it will run in the computer system, it must call different In this function, we can deal with the preprocessing function, API functions. So, we can use Windows API HOOK and then execute original “CopyFile” function, or skip this technology to monitor and intercept the Trojan programs’ API. function to execute the following codes. Then take monitoring registry and operating file as example to Likewise, we can use the same method to deal with the analyze and block Trojans. other operations. Monitor registry: To achieve the registry monitoring, we must first find the function pointer which will be replaced in B. Behavior feature database the system service dispatch table, then put this pointer forward The behavior feature database collects the features of to our own function, when the program calling the appropriate Trojan behavior. In order to describe the behavior of the function pointer, it will execute our own function, so it finishes Trojan, and to make convenient for programmer to analyze, the system services hooking. Take monitoring the Trojan to this strategy need to build the behavior feature database, which write the self-starting item of registry for example. The Trojan are consisted of feature vectors N,,,={AA ⋅⋅⋅ A} , A want to modify the registry, it will use the “RegCreateKey” A 12 n n function which function prototype is that: represent a feature vector of a behavior, which can be described as the following parts: Private Declare Function RegCreateKey Lib” advapi32.dll” • P Alias”RegCreateKey”(Byralhkey As long,ByvalpSubkey As Abstract description of behavior feature i modifying String,phkResult As long) As long self-start item of the registry as example. it can be P Use yn() is the indicate predicted value of ynˆ(), there are described as:“ i = modify self-start item of the registry”. k ynˆ()=− a ∗ yn ( − i ), n ∈ N ∑ i • C i −1 Invoking function i : The Trojan behavior display different API function call in program, so we can − In the formula, N is positive integer, ai is weighting C describe the Trojan behavior: i :=The name of the coefficient. corresponding API. Take modifying self-start item of Predictive encoding is set: the registry as example. Because the Trojan program needs to call “RegCreateKey” function while it k en()=−= yn () ynˆ () a ∗−=∈ yn ( i ), a 1, n N modifying the registry, we can describe the API ∑ i 0 function call as “RegCreateKey”. i−1 • In this strategy, we use heuristic analyzer to analyze In order to define the predictive encoding, we can use the program behavior, the main task is that to analyze mean-square rule, likewise ξ = E[()]en2 . and refine the behavior of Trojan. Here we analyze the min behavior of every Trojan repeatedly and set a weight B according to some certain rules. The weight shows D. The module of behavior response the different importance of the different behavior of the For the behavior which are analyzed by heuristic analyzer, Trojan. the system can give the following response: If the behavior is determined Trojan, the system directly call killing module to Here we also use the modifying self-start item of the delete the program; If the behavior is not determined Trojan, （ P ， C ，） registry as a instance. It can be described as i i B .So the system pop the warning window., at the same time, this its complete description in the behavior database is that windows can ask the user whether allow, ban, or delete this (modifying self-start item of the registry, “RegCreateKey”， program; If the behavior is determined normal behavior, the B).About other action, we can draw the same conclusion. system allow the program run continually

C. The module of the behavior of heuristic analysis IV. THE EXPERIMENTAL RESULT The module analyze the data from the module of In order to verify the performance of this system, this monitoring behavior object, which can be realized as experiment work under Windows XP, and use C++ achieve this follows::firstly, we can calculate the total weight of the program。The experimental data is from the data set DARPA suspected behavior according to the data from the module of of the U.S. Department of Defense Advanced Intrusion monitoring behavior object and integrate the weight in the Detection plans. To make things convenient for the experiment, behavior feature database, and then we can compare the total this system randomly selected four sets of data to do this weight with the pre-scheduled threshold, if the weight is greater experiment, in which Group 1 includes 10 000 data, which than the threshold, we can conclude that it is Trojan. contains 100 abnormal data; Group 2 consists of 20 000 data, which contains 200 abnormal data; Group 3 includes30 000 According to the suspicious degree of every index, this data, which contains 300 abnormal data; Group 4 consists 40 strategy gives a weight for every behavior. And add weights of 000 data, which contains 400 abnormal data. Through the different behavior in this processing, and then compared with experimental results ,which shows that the first group has alarm the pre-specified threshold. If the right value is larger than the numbers 78, and the second group has alarm numbers 169, the threshold value, we can make the program as Trojan program. third Group has alarm number 271, the forth group has alarm The specific formula is as follows: number 374.From the analysis result, we can get the detection M =++ ab* ab* "" ab * rate, false alarm rate. The result as Figure 2. 11 22 nn 1.2 a is the weight of every behavior, and it is satisfied the i 1 following condition: a ++aa"" +=1, a ∈[0,1] . 12 n i 0.8 detection rate 0.6 Meanwhile, the value of a give expression to importance false alarm i 0.4 rate degree about the corresponding Trojan feature in the Trojans 0.2 feature database. n is the number of suspicious behavior which 0 a program’s performance in the operation; M is suspicious D1 D2 D3 D4 degree of Trojan program in operation. The principle of defining the weight coefficient is to make Figure 2. The experimental result the mean square value of forecast errors the minimum. Here, As can be seen from Figure 2, the system in the detection of we use k sample value to predict. Trojan horse have a high detection rate and low false alarm rate. V. CONCLUSIONS REFERENCES This paper presents the Trojan-detection system model [1] Kingsoft, Internet Security Report in China, based on behavior analysis in terms of the behavior analysis. http://ir.kingsoft.com/phoenix.zhtml?c=189890&p=irol- This system analyzes profoundly the behavior features of the reportsannual,2009-10-12 [2] J. Newsome,D. Song, “Dynamic Taint Analysis for Automatic Trojan, and analyzes the program, which is tested, combining Detection, Analysis,and Signature Generation of Exploits on with heuristic analysis, so this system can effectively indentify Commodity Software,” Proceedings of the 12th Annual Network and known and unknown Trojan. This strategy has several Distributed System Security Symposium. 2005 advantages. [3] L. Xiao, L. Chao,C. Jia , “Strathy Trojan Horse Detection Method Based on System Call Hook,” Computer Engineering,2007,33(10), pp. 181- First, establishing and manage the object is very easy. The 183. text referred to the regular operation of several Trojan objects, [4] Y. U. Ryu, H. S.Rhee, “Evaluation of intrusion detection systems under such as the registry, file, etc. Because the object of conduct a resourse contraint,” ACM Transactions on Information and Systems here are independent, so in the future the process can also be Security, 2008, 11(4),pp. 201-202 gradually add other acts of objects module, which can be easy [5] F. Cohen, “Reply to Comment on A Framework for Modeling Trojans to manage. and Computer Virus Infection,” The Computer Jouranl, 2001,vol. 44.No 4,pp.326-327 Secondly, establishing the Trojan behavior feature database [6] P. K Singh, A. Lakhotia, “Static Verification of Worm and Virus is rather easier than Trojan feature database. With the Trojan Behavior in BinaryExecutables using Model Cecking,” Proceedings of technology development, all kinds of Trojan's signature will be the 2003 IEEE Workshop on Information Assurance United States changed at a high speed. Example Trojan deformation, multi- Military Academy, West Point, NY June 2003, pp.298-300. state, packers can will cause the Trojan horse program [7] N.Belacel, M.R Boulassel, “Multicriteria fuzzy classification procedure procftn:methodogy and medical and medical application,” Fuzzy Sets signature changes, so that the Trojan feature changes very and Systems. 2004,vol. 141,pp.203-217 rapidly, so summary the Trojan's feature is very troublesome. [8] F. Sabahi, A. Movaghar, “ Instrusion detection: A survey,” Proceedings But the Trojan's behavior in the various operating systems is of the 3rd International Conference on Systems and Networks very similar, even identical, so the behavioral characteristics of Communications. Siema, Malta, 2008,pp. 23-26. Trojans easier to sum up, maintenance, and overhead are [9] W. Yimin, D. Beck, V. Binh, et al., “Detecting Stealth Software with relatively small. Strider GhostBuster,” Proceedings of the International Conference on Dependable Systems and Networks. Yokohama, Japan. 2005 Finally, in the processing of analyzing the tested procedures [10] You Chen,Hua wei Shen, “An efficient and lightweight feature with heuristic analyzer, and system give a weight to behavior, algorithm of intrusion detection system,” Chinese Journal of Computers. so when analyze the program not only attention one behavior, ，2007 but also many behaviors. Thus, this method is more accurate [11] D. Geer, “Behavior-Based Network Security Goes Mainstream,” than the traditional method. Computer, 2006, 39(3),pp. 14-17 [12] D. Kin,H. N. Nguyen,S.Y. Ohn, et al., “Fusions of GA and SVM for However, there are many areas need to be improved and anomaly detection in intrusion detection system,” Advanced in Neural strengthened in this strategy. Such as in the processing of Networks.Lecture Notes in Computer Science 2005,.3498.,pp. 415-420 heuristic analysis, the question of adjusting weight needs to be [13] L. Tao, “A New Model for Dynamic Intrusion Detection,” Lecture Notes improved in order to improve detection accuracy and so on. in Computer Science. 2005, 38(10),pp.72-84