Effective Detecting Method of Nmap Idle Scan

Journal of JAITC, Vol. 9, No. 1, pp.1-10, Jul. 31, 2019 1 http://dx.doi.org/10.14801/JAITC.2019.9.1.1

Jungsik Hwang1, Minsoo Kim2* 1Interdisciplinary Program of Information & Protection Graduate School, Mokpo National University 2Dept. of Information Security, Mokpo National University [email protected], https://orcid.org/0000-0003-2782-1514, [email protected], https://orcid.org/0000-0001-7056-1952, *(Corresponding Author)

Abstract In recent years, information collection of attacks through stealth port scanning technology has become more sophisticated. The most commonly used Nmap port scanner supports a variety of stealth scanning technologies along with the existing scanning techniques. Nmap also supports Idle scan that is different from conventional stealth scans. This is a more sophisticated stealth scan technique by applying the SYN scan and ACK scan techniques. In previous studies, the detection of Idle scanning was on zombie system, but was not on victim system. In this paper, we propose an effective detection method of Idle scan on victim system. The Idle scanning is composed of two stages; they are probing the zombie and victim system and scanning the victim system. We analyzed the characteristics of the two stages. The characteristics, we captured, are that SYN and RST packets are different from normal packet. We applied them to detection method, then Idle scanning is detected effectively. Keywords: Nmap, Idle scan, port scanner, probe, zombie PC

1. Introduction A cyberattack usually starts with vulnerability scanning. Accordingly, vulnerability scanning methods have been developed. Scanned objects are diversified into OS, Network and service. After an attacker gathered various information through the scanning, he attacks using the information. According to the InfoSec Report provided by SANS [1] from 2016 to 2018, there is an increase in port scanning attacks on services. The graph in Figure 1 represents the number of scans for the service port numbers provided on the network. Port scan techniques have been developed in various ways. Many tools for such attacks have also been developed. The typical attack tools are Nmap port scanner and McAfee- developed SuperScan scanner. These tools not only provide a common connection scan to detect open ports, but also provide the stealth scan techniques to avoid logging on victim systems. Many countermeasures against such stealth scan techniques have been proposed.

Received: Dec. 10, 2018 Revised: Jan. 24, 2019 Accepted: Jan. 27, 2019

J. Hwang et al.; Effective Detecting Method of Nmap Idle Scan 2

Figure 1. SANS InfoSec report (PortScan)

FIN scan, NULL scan, XMAS scan and ACK scan among the stealth scans use wrong TCP flags. The detection method of them is checking the TCP flags in TCP connection. In addition, the Idle scan [2] provided by Nmap performs a SYN port scan attack using a zombie PC. The victim system recognizes the zombie PC as an attacker and responds to it. The attacker can create new attack opportunities using the other zombie PC, even if the victim detects and blocks the zombie PC. Patel [3] proposed a detection method for Idle scan. But there is a high possibility of false positives due to network environment between zombie PC and victim. In this paper, we propose the effective detection method of Idle scan. To do this, we analyze the scanning process and packets of Nmap Idle scan. Nmap Idle scan was composed of two stages; the probing and scanning. We found the characteristics of the SYS/ACK packets in the probing stage and the RST packets of zombie PC against the scanning. Therefore, we propose a detection method for Nmap Idle scan by comparing SYN/ACK and RST packets with the characteristics that we found.

2. Related Work A port scan is an attack that detects an open port on a particular system using specific program or technique. Port scan attacks in Nmap have two basic operating principles: Positive scan and Negative scan [4]. Positive scan uses packets that are answered by the open ports of the system, which are SYN scan, ACK scan, Idle scan, Ping scan and Connect scan. Negative scan uses packets that are answered by the closed ports of the system, which are FIN scan, NULL scan and XMAS scan.

2.1 Port scan tools There are a variety of tools that support port scans. Nmap and SuperScan support stealth scans and normal port scans.

Nmap is a scanning tool that has been under study since September 1997 [5]. The functions of Nmap are host detection, TCP/UDP port scanning, and OS(Operating System) detection. Nmap uses ping responses to detect hosts connected on the network. The port scan of Nmap lists open ports and services of target computers. Nmap detects the OS version using a different response for each OS and network devices.

2.1.2 SuperScan Superscan is offered by McAfee. SuperScan 3.0 and 4.0 can scan TCP and UDP ports such as Nmap. In addition, they can check the services using the ports and execute user queries. SuperScan4.0 additionally supports SYN scan.

2.2 Stealth Scan Detection The stealth scan [6] includes SYN, ACK, FIN, NULL, XMAS, and Idle scans. These technique is characterized by not leaving a log information on the victim system during the scanning process. So hackers often use the stealth scan to avoid leaving traces. Efficient Port Scan Detection Rules (EPSDR) [7] shows the method to detect SYN, FIN, NULL, and XMAS scans through the flag value of TCP packet header. EPSDR uses Snort to realize the method. For detecting these scans, EPSDR proposed Snort rules to check TCP flags of TCP packet header. EPSDR detects a scanning attack as SYN scan when Reserved bit 1, 2, and SYN flag are set. Likewise, EPSDR detects it as FIN scan when FIN flag is set and session is not established, NULL scan when all TCP flags are not set, and XMAS scan when FIN, PSH, and URG flags are set. Patel [3] proposed a method to detect Idle scan using Snort. The method is to check increasing IPid value, which is a characteristic of the idle scan. He explained that IPid value of the RST packet, which is the response packet from the zombie PC, increased by 2 during Idle scan. He collected RST packets from the zombie PC and compared IPid value of each other to detect Idle scan. However, if there are several sessions in the network, the IPid value of the RST packet can be changed by another session. This means that there is possibility of false alarms. 3. Analysis of Nmap Idle Scan Principle

Nmap manually creates the packet to perform stealth scan, So Nmap packets have different characteristics from normal packets. Nmap Idle scan includes features of SYN and ACK scans.

3.1 Characteristics of SYN Scan Figure 2 shows the SYN scanning process. SYN scan uses half-open connection to target system [8]. In normal connection, the SYN packet is transmitted using connect system call. In step 3 in Figure 2, the packet is transmitted with raw socket made by Nmap instead of the system call. When raw sockets are used, the attacker does not leave any information in the own local TCP/IP stack [9]. Therefore, the attacker system judges that the SYN/ACK packet returned from the victim system is an unauthorized TCP handshake as in step 5 to 6. Because of this process, the system, instead of Nmap, sends RST packet back to the victim system. pISSN 2234-3072/eISSN 2234-0963 Copyright ⓒ KIIT

J. Hwang et al.; Effective Detecting Method of Nmap Idle Scan 4

Figure 2. SYN scanning process

Table 1 shows the packet characteristics of SYN scan supported by Nmap. The characteristics of the SYN scan packet are as follows: The length of the TCP header is 24 bytes, and the window size is set 1024 to 4096 in most cases. The TTL has a random value in the range of 37 to 59, and the TCP option value is fixed to \x02\x04\x05\xb4. The source port of SYN scan packets is a random number, and the destination port is an import service ports of the target.

Table 1. Nmap SYN scan packet features SYN packet Packet option Explanation TCP length 24 Bytes Window size 1024 ~ 4096 Bytes TTL 37 ~ 59 Source port Random number Option \x02\x04\x05\xb4 TimeiInterval One packet in 0.01 seconds

3.2 Characteristics of ACK Scan Figure 3 shows the ACK scanning process. The attacker sends ACK packet to the victim system, then the victim regards the ACK packet as an inappropriate response. This is because there is no connection information with the attacker system in the victim system's local TCP/IP stack.

Figure 3. ACK scanning process

3.3 Characteristics of Idle Scan Idle scan is a stealth port scan using zombie PC. The zombie PC used for Idle scan is a system that is rarely used or is already occupied by an attacker. Idle scan can hide the position of the attacker by keeping the zombie PC in the middle. Therefore, the information of attacker does not remain in the victim system after scanning [2]. In Idle scan, if attacker sends SYN or SYN/ACK packets to victim system and the zombie PC, they will respond RST packets. Because IPid value of RST packets is sequentially increased, if the increased value of the IPid of the RST packet is checked, the attacker can confirm the success of scanning or the port status of target system. However, since the idle scan has a zombie PC between the attacker and the victim, the accuracy of the attack depends on the network environment. If there are many other network connections between two systems, the IPid will increase unexpectedly. The Idle scan of Nmap have two stages of probing and scanning. Probing is the process of applying the principle of ACK scan to judge whether the zombie PC and victim system are connected or not. Scanning stage is the process of applying the principle of SYN scan to confirm whether target ports of victim system are opened or not.

3.3.1 Probing Stage At the probing stage, the attacker sends 10 probe packets to the zombie PC. All probe packets have the SYN/ACK flag set. It is similar to ACK scan. This stage causes an improper 3-Handshake to receive the RST packet from target system. The zombie PC and the victim judge the SYN/ACK packets as inappropriate packets and then respond with RST packets.

J. Hwang et al.; Effective Detecting Method of Nmap Idle Scan 6

Figure 4. Idle scanning process (probing)

In step 2 of Figure 4, 6 probe packets are sent to the zombie PC. If the zombie PC does not respond the RST packet as shown in step 3, the Idle scan is terminated because zombie PC is not available. If the attacker receives a certain number of RST packets, Nmap performs step 4 and 5. At this point, the source IP address of packets in step 4 is forged to victim system. Therefore, the zombie PC responds RST packets to the victim system as in step 5.

3.3.2 SYN Scanning Stage In step 7 of Figure 5, the attacker sends the SYN/ACK packet to get the current IPid value of the zombie PC. In step 9 and 10, the attacker makes SYN packet whose source IP address is spoofed as zombie PC, and sends it to the victim as if it is sent from the zombie PC. If target port is open, the victim sends a SYN/ACK packet to the zombie PC in step 11. Since the zombie PC has never sent a SYN packet to the victim, the zombie PC judges that the connection is inappropriate and responds RST packet to the victim system in step 12. After completing the process, the attacker sends the SYN/ACK packet to the zombie PC as in step 13. Like the above principle, the zombie PC returns the RST packet to the attacker. The attacker compares the IPid value of the RST packet in step 13 with the IPid value in step 8. If the value has increased by 2, the attacker makes it sure that the target port of the victim system is opened. Table 2 shows the characteristics of the SYN/ACK packet in the Idle scan. The SYN/ACK packet is the same as the SYN packet characteristic. The RST packet of the Idle scan uses 80 or 443 to source port, because the attacker uses the HTTP or HTTPS port of the zombie PC.

Figure 5. Idle scanning process (SYN scan using zombie PC)

Table 2. Nmap Idle scan packet features SYN/ACK Packet Option Explanation TCP Length 24 Bytes Window Size 1024 ~ 4096 Bytes TTL 37 ~ 59 Source Port Randomly set number Option \x02\x04\x05\xb4 Time Interval One packet in 0.01 seconds RST Packet Option Explanation TCP Length 20 Bytes Window Size Fixed by 0 or 32767 TTL Depends on the target system Source Port 80 or 443 Destination Port Random number pISSN 2234-3072/eISSN 2234-0963 Copyright ⓒ KIIT

J. Hwang et al.; Effective Detecting Method of Nmap Idle Scan 8

4. Proposed Detection Method for Nmap Idle Scan

4.1 Detection Method To detect the Nmap Idle scan, we use the characteristics of the SYN packet and RST packet in Table 1 and 2. If the detection of Idle scan is performed using the characteristic of Table 1, we can judge it SYN scan using Nmap. However, it is not known whether this is the SYN scan by zombie PC. Therefore, the RST packet in Table 2 should be used together. Nmap Idle scan can detect the combination of SYN and RST packet characteristics. First, when a SYN packet is not received from the source of the RST packet, the RST packet is determined as a probe packet. As shown in step 4 and 5 of Figure 4, the victim can judge that the RST packet is originated from a packet forged by the attacker. Second, when a SYN packet is received from the source of RST packet, the victim compares the SYN packet with the characteristics of Table 1. If it matches, victim judges that it is SYN scan using zombie PC. As shown in step 10 to 12 of Figure 5, it is based on the principle that the attacker transmits a forged SYN packet to a victim system to perform an Idle scan.

4.2 Experimental Result The result of detecting Idle scan is shown in Table 3. The method of Patel [3] can detect the Idle scan when the IPid value of the RST packet in step 14 of Figure 5 increases by 2. However, this method can occur a false alarm when target port is multiple. It is because the IPid value of the RST packet increases irregularly. Patel's method can detect Idle scan only at the step 14 of Figure 5. And the detection of this method is performed on zombie PC. Generally, the zombie PC is occupied or rarely used by attacker. For these reasons, it is more appropriate to place the detection position in front of the victim system.

Table 3. Result of detecting idle scan

Patel’s method Proposed method Detection position Zombie Zombie Victim Multiple port scan X O O Step 3 of Figure 4 X O X Step 5 of Figure 4 X O O Step 8 of Figure 5 X O X Step 10 of Figure 5 X X O Step 12 of Figure 5 X O O Step 14 of Figure 5 O O X Using characteristics of SYN/ACK and Comparing IPid RST packets in Table 2 Method of Detection values of RST Packet Using characteristics of SYN packet in Table 1

Our method is applicable to both zombie PC and victim system. And also the detection of Idle scan is performed at various stages of probing and scanning. Our method uses the characteristics of the SYN and RST packets transmitted by the attacker. Therefore, detection is performed irrespectively with the attacker’s success or failure.

5. Conclusion Nmap supports the stealth port scanning techniques and is used to steal services and OS information about the target system. Idle scan among the stealth scans does not expose attacker’s information by using zombie PC. The Idle scan consists of probing stage and SYN scanning stage using zombie PC. Each stage applies the principle of ACK scan and SYN scan. Idle scan can’t be detected with TCP flags setting like as other stealth scans methods. There is a disadvantage that the detection method for the existing Idle scan using the increase value of the IPid of the RST packet can’t be detected in the victim system. Therefore, the method can cause false positives if the attacker scans multiple ports or the network is busy. In this paper, we have analyzed the features of Nmap Idle scan. And we proposed a more effective detection method for Idle scan by using characteristics of SYN and RST packets in probing and SYN scanning stages of the scan. Using this method, it is possible to detect in all stage of Idle scan and in both victim system and zombie PC. Through experiments, we confirmed that the detection of the Idle scan is possible through the packet characteristics in each stage. And we also confirmed that it can be detected in the zombie PC and the victim system.

References

[1] SANS, Threat Feeds, https://isc.sans.edu/threatfeed.html. [Accessed: Dec. 01. 2018]. [2] R. Ensafi, J. C. Park, D. Kapur, and J. R. Crandall, "Idle Port Scanning and Non-interference Analysis of Network Protocol Stacks Using Model Checking", 19th USENIX Security Symposium, Washington, DC, USA, August 11-13, 2010. [3] S. K. Patel, "Detection of Idle Stealth Port Scan Attack in Network Intrusion Detection System using Snort", https://www.slideshare.net/skpatel91/detection-of-idle-stealth-port-scan- attack- in-network-intrusion-detection-system-using-snort?from_action=save [Accessed: Dec. 01. 2018]. [4] OPRIX, Analysing NMAP, http://www.hackerschool.org/HS_Boards/data/Lib_network2/ AnalyzingNmap.pdf [Accessed: Dec. 01. 2018]. [5] G. F. Lyo, Nmap Network Scanning: The Official Nmap Project Guide to Network Discovery and Security Scanning Paperback, Nmap.org, Jan. 2009. [6] B. Claypool, "Stealth Port Scanning Methods", Global Information Assurance Certification Paper, 2002. [7] S. K. Patel and A. Sonker, "Rule-Based Network Intrusion Detection System for Port Scanning with Efficient Port Scan Detection Rules Using Snort", International Journal of Future Generation Communication and Networking, Vol. 9, No. 6, pp. 339-350, June 2016. [8] G. Taleck, "SYNSCAN: Towards Complete TCP/IP Fingerprinting", CanSecWest, Apr. 2004. [9] M. Rash, "Linux Firewalls Attacker Detection and Response with iptables psad, and fwsnort", No Starch Press, 2007.

J. Hwang et al.; Effective Detecting Method of Nmap Idle Scan 10

Authors

Jungsik Hwang Hwang graduated with the Bachelor’s degree at the Department of Information Security, Mokpo National University, Korea, in 2017. He received Master’s degree from the same university in 2019. His Master thesis topic was related to the network scanning system. He is a penetration testing researcher in RAON Whitehat at Korea. His research interests include penetration testing and network forensics.

Minsoo Kim Prof. Kim graduated with the Bachelor’s degree at the Department of Computer and Statistics, Chonnam National University, Korea, in 1993. He received Master’s degree and PhD degree respectively from the same university in 1995 and 2000. His PhD thesis topic was related to the intrusion detection system. He is a professor in Department of Information Security of Mokpo National University at Korea. His research interests include malware analysis and computer forensics.