CAUDIT: Continuous Auditing of SSH Servers To Mitigate Brute-Force Attacks Phuong M. Cao, Yuming Wu, and Subho S. Banerjee, UIUC; Justin Azoff and Alex Withers, NCSA; Zbigniew T. Kalbarczyk and Ravishankar K. Iyer, UIUC https://www.usenix.org/conference/nsdi19/presentation/cao

This paper is included in the Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19). February 26–28, 2019 • Boston, MA, USA ISBN 978-1-931971-49-2

Open access to the Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19) is sponsored by CAUDIT: Continuous Auditing of SSH Servers to Mitigate Brute-Force Attacks

Phuong M. Cao1, Yuming Wu1, Subho S. Banerjee1, Justin Azoff2,3, Alexander Withers3, Zbigniew T. Kalbarczyk1, Ravishankar K. Iyer1 1University of Illinois at Urbana-Champaign, 2Corelight, 3National Center for Supercomputing Applications

Abstract While only a small fraction of such attempts succeed, they This paper describes CAUDIT1, an operational system have led to major misuses in 51% of 1,800 surveyed organi- deployed at the National Center for Supercomputing Applica- zations, with a financial impact of up to $500,000 per organi- tions (NCSA) at the University of Illinois. CAUDIT is a fully zation [7]. automated system that enables the identification and exclusion This paper describes the production deployment of of hosts that are vulnerable to SSH brute-force attacks. Its CAUDIT at the National Center for Supercomputing Ap- key features include: 1) a honeypot for attracting SSH-based plications (NCSA) at the University of Illinois over a period attacks over a /16 IP address range and extracting key meta- of 463 days. CAUDIT is a fully automated system to identify data (e.g., source IP, password, SSH-client version, or key and exclude hosts that are vulnerable to SSH brute-force at- fingerprint) from these attacks; 2) executing audits on the live tacks. The system is composed of existing techniques with production network by replaying of attack attempts recorded custom-built components, and its novelty is to execute at a by the honeypot; 3) using the IP addresses recorded by the scale that has not been validated earlier (with thousands of honeypot to block SSH attack attempts at the network border nodes and tens of millions of attack attempts per day). The by using a Black Hole Router (BHR) while significantly re- key components of the proposed system are as follows. ducing the load on NCSA’s security monitoring system; and 4) • An SSH-based honeypot deployed on an entire /16 class- the ability to inform peer sites of attack attempts in real-time less inter-domain routing (CIDR) network2 that mimics to ensure containment of coordinated attacks. The system is a realistic server farm of 65,536 machines. In contrast composed of existing techniques with custom-built compo- with other honeypots [8–12], ours is non-interactive, nents, and its novelty is its ability to execute at a scale that has i.e., it only records and immediately rejects any attack not been validated earlier (with thousands of nodes and tens attempts; thus, it has a small memory footprint that re- of millions of attack attempts per day). Experience over 463 duces the operational risk of exploiting the honeypot to days shows that CAUDIT successfully blocks an average of get into the internal network. 57 million attack attempts on a daily basis using the proposed BHR. This represents a 66× reduction in the number of SSH • A continuous SSH auditing tool (driven by attack at- attempts compared to the daily average and has reduced the tempts recorded using the honeypot) that automatically traffic to the NCSA’s internal network-security-monitoring replays the attempted attacks against an internal network infrastructure by 78%. in order to uncover vulnerable hosts. This approach 1 Introduction extends the notion of fire drills in production systems for reliability testing [13–15]. CAUDIT minimizes the Security auditing of large-scale networks is challenging auditing tool’s disruption of the production network, e.g., due to the constantly evolving configurations of networked de- by subscribing to SSH protocol activities by using the vices [1,2]. Critical devices often expose their remote access deep packet inspection capabilities of existing network interfaces to the Internet via the Secure Socket Shell (SSH) security monitors such as the Bro IDS [16]. protocol [3], often using default usernames/passwords [4]. The availability of stolen credentials [5] has added a new • An enhanced black hole router (BHR) deployed at the dimension to the problem: attackers can now remotely mas- production system’s borders to support automated re- querade as legitimate users and penetrate internal networks to sponse to malicious IP addresses identified by the audit- misuse computational resources and leak sensitive data [6]. 2The address space belonged to a Fortune 50 company but has been 1 https://pmcao.github.io/caudit transferred to NCSA.

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 667 ing tool. The BHR is integrated with a flow-shunting • We investigated a new observation on SSH attack at- tool that discards high-volume irrelevant packets, e.g., tempts that use keys. While traditional SSH attempts (VPN) traffic, before they get use known username and password pairs, the honey- parsed by the kernel’s networking stack. This enhances pot’s collected data on attack attempts indicate that most the BHRs ability to block brute-force SSH attacks and of the SSH keys used by attackers were not previously bypass legitimate network flows to reduce the load on known and were mutually exclusive to each source IP network security monitors. address. This finding suggests that attackers obtained the keys via direct compromise of file systems, either • A security alert-sharing network to provide timely alerts by using ransomware or through reverse-engineering of to peer sites to ensure containment of coordinated at- keys embedded in the firmware of IoT devices. tacks [17]. All attack attempts (recorded by the hon- eypot) are encrypted and shared (in real time) with ten • Companies make extensive use of SSH private keys authenticated peer sites (nine in the U.S. and one in for automated server management, but our analysis Singapore). strengthen the risk of improperly using SSH keys Placing CAUDIT in perspective. Prior work on SSH se- through discovered incidents. Our analysis has led to curity auditing has focused on preventing SSH brute-force the implementation of new security policies at NCSA. attacks [16, 18, 19] using multi-factor authentication [20], First, all known SSH hosts in the known_host file must deceiving attackers by using honeypots [9, 21, 22], Internet- be hashed to hide the actual host names in the event of a scale scanning of vulnerable hosts [23], and use of Black successful compromise, thus reducing attackers’ lateral Hole Routers to block blacklisting-based malicious IP ad- movements. Second, SSH passphrases must be enforced dresses [24]. Such approaches have helped network operators in private keys to prevent the use of the leaked keys. run ad-hoc scans and analyses of attacks after the fact. Par- While the current implementation of CAUDIT focuses on ticipants in red teams and bug bounty programs emulates brute-force SSH attacks, the proposed architecture can be malicious behavior of attackers to uncover security vulner- extended to address other types of attacks. We has open- abilities and bugs [25]. However, such processes are still sourced our implementation.3 manually driven by security experts and are therefore difficult to scale for large production networks. The above techniques 2 Background are often ineffective in practice, e.g., 1) SSH honeypots do not This section provides an overview of the daily operations attract a large amount of traffic; 2) large-scale network scan- at NCSA and the typical threats targeting its infrastructure. ning hampers the performance of production networks, and 3) 2.1 Daily operations at NCSA it is problematic to maintain and manage a large blacklist (e.g., NCSA provides integrated cyberinfrastructure that includes because of false positive filtering). In addition, coordinated computing, data, networking, and visualization resources to attacks can be thwarted using security intelligence sharing enable research of scientists and engineers at the Univer- and analysis between geo-distributed sites [26]. Most impor- sity of Illinois at Urbana-Champaign (UIUC) and across tantly, such efforts have never been integrated as a whole and the country. NCSA hosts Blue Waters [27], a sustained validated at a large scale in production workloads. Those petaflop system that is a prime attack target, as attackers limitations have motivated us to design a scalable auditing wish to exploit its processing power and exfiltrate sensitive system. data in storage. On a daily basis, thousands of researchers During 463 days production deployment at NCSA, the hon- access NCSA’s cyber-infrastructure remotely (using SSH pro- eypot attracted attacks that originated in 76% of the registered tocols) via a wide area network (WAN) to carry out exper- IPv4 autonomous systems (ASes), with a total of 405 million iments. NCSA observes 5,970 (variance = 1,541 users) le- attack attempts from 4 million unique source IP addresses. gitimate remote logins every day. At the same time, we ob- On a daily basis, the BHR blocked an average of 57 million served an average of 875,491 credential-guessing activities SSH attack attempts. On average, 875,491 attack attempts (405,352,245 attacks/463 days), which are 147× more frequent than per day passed the BHR and were recorded at the honeypot legitimate login activities (875,491 attack SSH/5,970 legitimate SSH). to support the fire drill. Our operational experiences were as Several computing and data services at NCSA, e.g., a Ku- follows. bernetes container cluster and NCSA’s dspace data reposi- • The BHR augmented with the fire-drill resulted in a 66× tory [28], are accessible from the Internet, which exposes reduction in the number of SSH attempts compared to them to targeted attacks. the daily average. The system identified eight vulnera- ble hosts, one of which was an unsecured HPC storage 2.2 System model device, and one of which was compromised. The sys- An SSH server is the most critical component of a host tem reduced the traffic to the internal network security because it is typically the single point of entry for authenti- monitoring infrastructure by 78%. 3 https://pmcao.github.io/caudit

668 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association cating remote users. Notwithstanding, many SSH servers are SSH daemon binary file /usr/bin/ssh was different not properly guarded, e.g., exposed to the Internet while still from the official version that should have been installed using default credentials. on the host. The modified file was related to suspi- 2.3 Threat model cious downloads 181.215.xxx.yyy:24221/op3.tgz and 182.215.xxx.yyy/sp.tgz from a remote server. The file, This paper considers SSH credential-guessing attacks that op3.tgz, contained the source code for OpenSSH v5.3.p1 originate in various sources, e.g., botnet-infected devices [29] and was compiled locally to create the ssh binary with or external attackers targeting personal accounts at a large- which the authentic file was replaced. Analysis of the mod- scale network. Once infected, these devices receive com- ified ssh binary and OpenSSH source code revealed that mands from attackers and constantly look for other ex- the malicious binary contained modifications of the original posed devices, so they account for the majority of credential- OpenSSH so that it would record SSH login credentials to a guessing attack traffic. We assume that attackers are not file, /usr/lib64/.lib/lib64.so. The location and name insiders, i.e., they are not aware of the /16 address space in of this file were designed to hide it from plain sight (e.g., which we deployed the honeypot. simply by running the ls command). An examination of the 3 Motivation lib64.so file revealed that the attacker had collected creden- This section describes a real credential-stealing incident tials of two users across three different systems. We suspect that could have led to a data leak at NCSA, and shows the ben- that the attacker logged in periodically to collect the stolen efit of continuous auditing in securing large-scale networks. credentials and the hosts to which they had connected, and then cleared the credentials from the file. A snippet of the ma- 3.1 A Motivating Example licious code is shown in Figure1. While previous work has In April 2018, NCSA’s security team was notified of sus- covered brute-force SSH attacks [18, 31, 32], none has cov- picious activity on a multiuser host supporting a major sci- ered a sophisticated attack with this level of detail. Forensic ence project. A legitimate user on that machine reported analysis of this attack has driven the design of our SSH au- that attempts to connect from the host to the Fermi National ditor (Section 4.2) to identify potentially compromised SSH Accelerator Laboratory (FNAL) [30] had failed. servers. Analysis of network logs indicated that this user’s ac- As a result of the compromise, the stolen user credentials count had been accessed a number of times from suspi- were used to access an iForge cluster [33], a high-performance cious IP addresses during the previous 2 weeks. Cross- computing cluster designed specifically for NCSA’s industry examination of the host’s file system revealed that the partners. Although the stolen user accounts were confirmed to have been accessed and the attackers tried to escalate priv- diff output for /sshconnect2. (truncated) ileges, the attack failed, as the stolen user accounts did not int userauth_passwd(Authctxt *authctxt){ have root privileges on the iForge system. Comprehensive + mode_t u; examination of other hosts accessible by this account did not + char *file_path = "/usr/lib64/.lib/lib64.so"; reveal any further indications of privilege escalation. + int fd = open(file_path, O_WRONLY | O_APPEND, Investigation of the legitimate user revealed that the real + S_IRWXG | S_IRWXO | S_IRWXU); user had accessed an NCSA server from a host in the United + if (fd != -1) { Kingdom (UK) on March 2018, as confirmed in login records. + int usize = strlen(authctxt->server_user); The NCSA team provided indications of the compromise to + int psize = strlen(password); the admin of the host in the UK, 148.197.xxx.yyy, and the + int hsize = strlen(authctxt->host); admin confirmed that they had indeed been compromised. + int out_size = usize+psize+hsize+4; Further examination suggested that the UK host had been + char *out = (char *) malloc(out_size); compromised as far back as February 2017. Fortunately, + if (out != NULL) { NCSA’s logs show that there was no access of the legitimate + strcpy(out, authctxt->server_user); user’s NCSA account at that time. + strcat(out, password); Remark. The compromise of this user’s password likely + strcat(out, authctxt->host); occurred on the UK host. Although the UK host had been + write(fd, out, out_size); compromised a year before, the attackers stayed dormant, in + free(out); part because they didn’t know exactly what systems the UK + } host could access. Upon making a successful connection from } the UK host to NCSA, the attackers compromised the host at Figure 1: A snippet of the malicious code that NCSA and tried to reach its peers, including Fermi lab and had been injected into the function userauth_passwd the iForge cluster. in OpenSSH server to record passwords to the file /usr/lib64/.lib/lib64.so

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 669 Non-interactive Interactive auditor Internal servers 400 MB 300 MB Legit controller SSH Bro Users Data syslog 200 MB osquery 100 MB Security monitors X SSH-auth-logger 0 MB 4 8 6 Attacker Black Hole 2 14 20 24 12 16 18 22 Router Alert Sharing 10 Network Figure 2: Overview of CAUDIT. Instead of exposing the Number of concurrent SSH connections internal servers of the existing infrastructure (dotted box) to the internet, CAUDIT uses a virtual server farm of SSH Figure 3: Comparison of memory footprint between an inter- authentication loggers that attracts attack attempts and uses active honeypot that provides a shell for each connection and an auditor to continuously mimic such attempts on internal our non-interactive honeypot on a commodity server. servers. An SSH database of excessive scanners is used by 1) Thus, one instance of our honeypot covers an entire /16 IP a controller that instructs a Black Hole Router to dynamically address space with significantly fewer resources (Figure3). create null routes to block attackers at the network border, The disadvantage of this technique is that it puts all the loads and 2) an alert-sharing network. of attacks on a single physical server. However, one can miti- 4 System Architecture gate that issue by using a load balancer in front of the server This section first describes our architecture and implemen- that is hosting the honeypot. tation details. Then, we explain the significant modifications As a result, our honeypot attracts an average of 875,491 to existing tools that are required to mimic attackers’ attempts attack attempts every day, i.e., 33× more than the one in [19]. and block malicious network flows at scale. Deceiving attackers. Making a honeypot look realistic is challenging, since sophisticated attackers will eventually 4.1 SSH authentication logger (SAL) discover that the honeypot does not offer any real system The core component of our approach is a lightweight, non- and network resources. Our goal is not to completely fool interactive SSH server (i.e., honeypot) that records brute-force attackers, but to make our honeypot realistic enough to attract attacks (shown as ssh-auth-logger in Figure2). a large number of attacks (as shown above). To realize that Attracting attackers. While any SSH server on the Inter- goal, our honeypot generates a host key deterministically net is susceptible to SSH attack attempts, the attack volume based on the destination IP address being scanned. Thus it for servers on an IP address is relatively low, in the order of creates the impression that a large network (of diverse and thousands of attempts per day, e.g., 27K attempts per day in a real machines) is responding to an attacker’s guesses, while relevant SSH honeypot deployed by the Naval Postgraduate in fact there is only one instance of the honeypot running. School [19] in 2017. To attract more attackers than existing Isolation and memory footprint. Properly isolating honeypots [9, 21, 34, 35], we deployed SAL on an enticing a honeypot is difficult. A traditional interactive honey- classless inter-domain routing (CIDR) /16 address space that pot [21,34] provides a shell for each attack attempt. Although mimics 65,536 (216) realistic SSH servers. Note that owning such a honeypot could be insulated in a virtualized environ- an entire CIDR address-set /16 is difficult, given that IPv4 ment (e.g., a container [36, 37] or a virtual machine [38–40]), addresses are being exhausted. NCSA is in a unique position nonetheless, attackers have network access and may bypass since it owns the entire CIDR address space that previously the virtualized environment with a vulnerability, e.g., CVE- belonged to a Fortune 50 company. In addition, NCSA can 2017-5123, allowing the attack to escape from the container. afford to reserve the entire address space for the honeypot, In addition, providing a shell for each attack attempt does while other organizations need IPv4 address space for their not scale, since the more realistic a honeypot is, the more physical or virtual machines. resources (e.g., memory) are needed for deceiving the attack- By deploying our honeypot on an entire /16 IP address ers. Figure3 compares the memory footprint of an interactive space, it gives our honeypot a large capacity because all in- coming credential guesses targeting the CIDR address space, for x in \$(seq 1 254); do ip route add local 143.x.\${x}.0/24 dev p5p4 i.e., the darknet, are redirected to only one instance of the ip rule add from 143.x.\${x}.0/24 table darknet honeypot (Figure4). Thus, we do not need to duplicate done the honeypot deployment on 65,536 physical machines in ip rule add from 141.x.y.z dev p5p4 table darknet the CIDR address space. Also, all connections targeting the CIDR address space of the honeypot can be automatically Figure 4: Sample ip route rules to setup a darknet for a labeled as malicious attempts, because none of the legitimate /16 CIDR address space for an interface named p5p4. servers is assigned an IP address in the CIDR address space.

670 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association honeypot that provides a shell (based on containers) as a fire drill, i.e., can automatically and continuously audit in- for a number of concurrent connection attempts vs. our non- ternal hosts in the same way that real attackers would, without interactive honeypot. While the non-interactive honeypot disrupting production workload in large-scale networks. maintains a constant memory usage of ∼ 98MB, the interative- To address the above challenges, we have implemented an honeypot uses linearly more memory for new containers as SSH credential auditor (SCA) that discovers weak and stolen each new connection is made. Thus, traditional interactive credentials in existing SSH servers as well as anomalous honeypots do not scale to millions of attack attempts. changes in SSH server configurations. In contrast to existing To address those challenges, our SSH authentication logger tools that use default dictionaries, the SCA is driven by pass- is implemented in Golang with only 159 lines of code. A words used by attackers that target the honeypot, collected small code base reduces the attack surface of the honeypot, by an SSH authentication logger. Thus, it closely mimics thus it has a low operational risk. In addition, our honeypot attackers’ attempts on internal hosts without exposing these does not provide any shell to each attack attempt, it rejects hosts to attackers. SCA works as follows. attack attempts by default to preserve memory, thus it attracts Discovery of internal SSH hosts. In a large-scale net- more attackers and has been able to log to millions of attack work, scanning an entire IP space, e.g., /16, for a particular attempts per day. Furthermore, a small code base eases its port takes a long time and disrupts network activities, e.g., trig- deployment, i.e., all dependencies of the honeypot can be gers false alerts. Similar to existing tools such as nmap [43], cross-compiled and contained in a single binary file that can SCA performs a full discovery on the entire network peri- run on embedded, e.g., ARM devices such as Raspberry Pi. odically but only weekly, because a full discovery would be Attack attempts records. The SAL logs a 5-tuple record disruptive. of the following data: SSH client version, SSH key fingerprint, SCA minimizes the disruption of full scans on the produc- source IP address, username, and guessed password (shown tion network by generating a list of suspected hosts based in Table1). The key enabler for the SAL is the fact that the on the basic information provided by Bro on the SSH proto- SSH protocols allow the server to read plain-text credentials col activities on the network. For example, Bro can output at authentication time, and also allow us to study different a source/destination IP, SSH server banner, and client key types of credentials. These measurements provide 1) visibility fingerprint, based on the handshake of an SSH connection. into the originating autonomous systems (ASes) from which Audit of SSH hosts. The SSH credential auditor performs the attacks came (based on the source IP addresses), and 2) following the audit schedules. a deep understanding of the SSH clients that carry the attack A full audit checks for 1) known weak or stolen credentials, traffic. 2) credentials collected from the SAL, and 3) stolen or leaked Identifying malicious scanners. Our goal is to have high SSH keys. A full audit is triggered in two cases: a new precision in identifying malicious scanners (i.e., a low rate host is added to the network (by a full discovery or by an of false positives). A simple approach is to rely on the count incremental discovery), or the SSH version or key fingerprint of the attack attempts to block aggressive scanners. Another of any known host changes. observation is that malicious scanners often use a fake SSH A partial audit only checks for new credentials, e.g., new client version banner; sometimes, these fake SSH client ver- passwords that attackers used against the honeypot, on exist- sions contain typos. For example, an attack might use PUTTY ing hosts with a customized interval, e.g., every day. or putty to masquerade as the PuTTY SSH client (note that Localization and isolation of weak/compromised SSH the correct SSH version has the lowercase u character). This hosts. Our experiences with past incidents [44] have shown issue is analyzed in detail in Section 5.3. that an unexpected change in a server key fingerprint and server version is typically an indication that an attack is com- Table 1: An example of an SSH record promising the OpenSSH daemon. Once a weak/stolen cre- Field Example Value dential or an unexpected change is discovered, an alert is source_ip 123.201.xxx.yyy automatically sent to network operators to isolate the host client_version SSH-2.0-libssh2_1.7.0 from the production network. It is followed by an email or key_fingerprint N/A a face-to-face meeting with the host owner to confirm the username dspace security status of the host. All network flows in/out of the password dspace@123 host are reviewed for possible redirection into the Black Hole Router. key_fingerprint is intended for key-based authentication 4.3 Black hole router (BHR) 4.2 SSH credential auditor (SCA) Although the Black Hole Router seems logically similar Despite that existing tools such as the UNIX passwd utility to blacklisting of IP addresses, our BHR’s goal is not to [41, 42] can give warnings for dictionary-based passwords block 100% of the attack attempts, but to reduce the loads on and misconfigurations, none of the available tools can be used existing network security monitors. There is only one global

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 671 blacklist in the BHR, at the network border. That makes is compared with the memory cache, and the malicious IP it easier to manage and to update the list (i.e., by removing address is re-inserted into the routing table if it is present in inactive IP addresses from the list). We only keep IP addresses the cache. that actively participate in attack attempts in the BHR, thus Flow shunting. We have implemented flow shunting for reducing false positives. the Bro IDS by using eXpress Data Path (XDP) [47], a pro- Although the SSH authentication logger and SSH creden- grammable network data path in the . XDP tial auditor provide a list of IP addresses for SSH scanners, preprocesses an incoming packet without early allocation of naive blocking such IP addresses does not work in large-scale the skbuff [48] data structure in the networking stack in the networks. Existing host-based blocking approaches such as Linux kernel or software queues. XDP works by looking fail2ban [45] cannot manage a global list of blocking IP at a specific offset in the packet, e.g., a flag identifying a addresses in a large-scale network. Although blocking is handshake in the SSL/TLS record, to determine whether it is possible, a malicious IP address is blocked only when the encrypted or it is part of a protocol handshake. kernel has already fully parsed and analyzed a sequence of As a result, packets coming from malicious IP addresses packets from the malicious IP address. On a large network and packets that contain encrypted data are shunted (dis- with heavy traffic (on the order of 100 Gbps) and a variety of carded) before any further parsing (by kernel-level packet- cryptographic protocols (e.g., IPSec, SSH, or TLS), network capture mechanisms such as the Berkeley Packet Filter [47]) security monitors (NSMs) such as Bro are often overloaded happens, except for that of the handshakes. and drop packets. While existing NSMs cannot analyze the encrypted contents of cryptographic protocols, they can still 4.4 Alert-sharing Network (ASN) provide valuable insights by analyzing the initial handshakes, Large-scale networks employ a variety of monitoring tools e.g., alerting on the use of outdated SSL protocols, expired and corresponding analysis techniques to provide compre- SSL certificates [46], or compromised SSH keys. Our goals hensive coverage. Network IDSs [16] perform deep packet are i) to block excessive brute-force attack attempts to reduce inspection of network traffic for detection of anomalous ac- the load on internal NSMs, and ii) to properly bypass flows tivity. In addition, network traffic analysis is augmented with that are unrelated to brute-force attacks for further analysis. host logs to detect coordinated attacks. While such alerts To realize the goals above, we implemented our Black are often handled by a dedicated incident response team, re- Hole Router (BHR) at the network perimeter by using the cent attacks have happened across multiple institutions at a Border Gateway Protocol (BGP). The BHR works with the global scale (Section3). Very few institutions can afford exit routers and manages a list of malicious IP addresses as the kind of dedicated security team NCSA has. Thus, a new follows. coordination effort is needed to prevent such attacks. To facil- Null route. Immediately after a credential-guessing at- itate cross-site incident response and forensic analyses, our tempt is observed, the network flows that originate at the honeypot provides a data feed of alerts that can be used to malicious IP addresses that are carrying out excessive attacks exchange SSH records with other national and international (e.g., guessing of multiple usernames within a short period) sites. Our honeypot is being used in bidirectional exchange of are redirected to a null route, which is a standard feature security-alert-related information with one site, the Singapore in BGP routers. The BHR discards the incoming network University of Technology and Design. The major participat- traffic without telling the source IP address that the network ing sites in the U.S. are the Pittsburgh Supercomputing Center, flow did not complete. Thus, the attackers are more likely to the Texas Advanced Computing Center, and Duke University, send more requests, with the intention of receiving a proper which, because of organizational policies, are only receiving response. As a result, the BHR reduces overall network load unidirectional alerts from NCSA. on the WAN border switch and allows the honeypot to avoid Site authentication and alert encryption. Each site has excessive attacks. a private key that is identified by: a corresponding public key, Catch-and-release. While the BHR can redirect flows a hostname, and a port. Sites must register their public keys from malicious IP addresses to the null route, the routing with the honeypot for authentication. Since sites exchange table is limited and cannot store too many IP addresses. In alerts that may contain sensitive personnel information and our router configuration, the upper limit of the routing table IP addresses, we encrypt alerts in transport by using NIST’s is a million entries, and the number of malicious IP addresses recommended Curve25519 cryptography [49]. To implement make up one-third of that (and could increase in the future). authentication and encryption, we utilize ZeroMQ [50], the To reduce the load on the routing table, the BHR implements a high-performance message queue library that has been proven catch-and-release technique, in which the list of malicious IP in financial applications with similar needs. addresses is stored externally in a memory cache. A malicious Site discovery. Sites use a simple gossip protocol for dis- IP address is initially present in the routing table as usual; then covery and alert exchange [50]. First, a new site needs to it is released after a period of time (i.e., an hour) of inactivity. be introduced to the network by a trusted peer. The trusted When there is a new flow from a new IP address, the flow peer then advertises the new site’s identity to its neighbors.

672 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association Table 2: Summary of attack attempts Table 3: Top 5 cloud/VPN providers and top 5 Internet ser- Field Unique Total vice providers (ISPs). 93% of attacks launched from the top SSH key fingerprint 159 103,554 5 Cloud/VPN providers originated in Microsoft Azure in Eu- SSH client version 171 405,352,245 rope and OVH. Over 31% of attacks from the top 5 ISPs Username 3,214 405,352,245 originated in Asia. Password 95,989 405,248,691 Cloud/VPN % ISP % Source address 4,035,975 648,333,747 Microsoft Azure 4.60 China Telecom 22.36 OVH 0.28 Indonesia Comnets 5.85 The SSH key fingerprint is an SHA-256 hash of the SSH public key Linode 0.20 China Unicom 3.19 Second, an encrypted alert is broadcasted from a site to its 21vianet 0.12 MCI Comm 0.13 neighbors and is further propagated to the entire network. FrootVPN 0.03 Infonet Comm 0.12 Although our ASN guarantees the confidentiality, authen- ticity and integrity of shared alerts, a man-in-the-middle ad- Table 4: SSH-2.0 client versions with a daily count percentage versary may cause network congestion and network partition. greater than 50%. Thus, the ASN prevents alerts from being shared within a time Client Version Count Release Year  0.1 76.7M 2010 limit in order to mitigate attacks at runtime. We will investi- sshlib gate techniques such as the use of redundancies [51] to send 0.5.2 1.8M 2011 alert replicas across multiple network paths, thus maximizing libssh2 1.7.0 26.8M 2016 the probability that the alert will be successfully delivered. paramiko 2.4.0 25.1K 2017 Go N/A 19.4M – 5 Measurement Results PUTTY N/A 20.4M – This section presents the main results from our operational (whose SSH client version strings are paramiko and Go, re- experiences with CAUDIT during a 463-day period of its spectively). deployment at NCSA. Attack device cloaking. While it is not possible to spoof the source address of an SSH client (because SSH is a TCP- 5.1 Dataset based protocol), our honeypot observed a deliberate tech- A total of 405 million SSH attack attempts were observed nique used by attackers (12% of the top six client versions) to in our longitudinal data collection period of 463 days (Feb. mask the SSH client version in order to bypass SSH firewall 7, 2017 to May 17, 2018). All the attacks were observed at rules that block unknown SSH clients. Specifically, attackers NCSA, as summarized in Table2. had their clients masquerade as PuTTY SSH clients, because 5.2 Attack sources PuTTY is popular (see Table4). The real PuTTY SSH client uses the banner PuTTY with a lowercase u, not the banner The majority of the attack sources were Cloud and VPN PUTTY used by attackers (as shown in Table4). Thus, it ap- providers and Internet service providers (ISPs) (shown in pears that malicious SSH clients use fake SSH client version Table3). For the listed top 5 Cloud/VPN providers, attacks banners to masquerade as legitimate PuTTY clients. from Europe (Microsoft Azure and OVH) accounted for 93% Age of SSH clients. We characterize the age of SSH and those from Asia (Linode and 21vianet) accounted for clients by the release data in the underlying SSH libraries. 6% of the attacks. For ISPs, over 31% of the attacks origi- Per this definition, aged devices account for nearly half of nated in Asia. Those attacks spanned over 15 cities in China the attacks. Our honeypot observed 77.5M attacks (47% of and accounted for over 80% of the listed ISP-based attacks. all attacks) that use SSH libraries released in 2010–2011 (we These findings highlight weaknesses in the security monitor- suspect that those belong to old devices that are still in op- ing infrastructure of outbound traffics in the listed Cloud/VPN eration), as indicated by their sshlib versions (see Table4). providers. On the other hand, the remaining attacks used relatively new 5.3 SSH clients in attacks SSH client libraries, e.g., Go, libssh2, and paramiko, because many of those libraries contain primitives for building SSH When a device initiates an attack, an SSH client version clients. can be observed, per the SSH 2.0 protocol, that can be used to identify the type of device. We observed a total of 171 unique 5.4 Attacks using personalized passwords SSH clients (Table2). Among the top six client versions The availability of leaked password databases [5, 52] en- observed in attacks in Table4, 63.8% of attacks used C/C++ ables attackers to run targeted attacks that use less popular, libraries that included libssh2 and sshlib. One reason e.g., personalized passwords [53]. Our goal is to character- could be that those C/C++ libraries are already installed in ize the intention of attackers, i.e., 1) are they broadly brute- embedded devices and allow one to generate attacks at very forcing devices by using many attempts based on a dictionary, high rates (in terms of the number of attacks per minute). or 2) are they specifically targeting personnel by using just We observe that 26.4% of attacks used Python and Golang a few attempts based on stolen passwords to stay under the

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 673 4 105 V!rtu@l 1.0 7ujMko0admin P@55word_. 3 123456

se7en-h0st# y n 4

Gu3stUs3r123 t

10 i o l i casey123 i s Zte521 b 0.9

n 2 raspberrypi bulldogs1 a

e uClinux welc0me 3 b

alpine 10 o m r i vagrant testing1234 p

d 1

kmroot 0.8 e

E QWERT5TGB5 Count 2 v

qwertyu i N 10 123456 t

S IoT

0 a l - luckygirl HIBP t u dictionary allo 0.7 d 1 cosamaloapan m IoT

n 10 1 random friends u

2 Dictionary 007 tyisfatohm C 0 dspace@123 0.6 5 0 5 10 10 5 6 7 8 1st t-SNE dimension Password 10 10 10 10 Daily count Figure 6: Histogram of 96K distinct pass- Figure 5: Projection of passwords in 2D Figure 7: Empirical cumulative distribu- words in our dataset, in which personalized space using t-distributed stochastic neigh- tion function (CDF) of daily count of passwords typically have low counts and bor embedding (t-SNE), which reveals password guesses. three types of passwords. lie at the tail. radar? itly targeting site-specific personnel or infrastructure based on Visualizing features of passwords. To support the secu- the leaked password database. For example, we observed one rity team in recognizing the intentions of the attackers, we instance of guess-based access to the data repository space divide the attempted passwords into clusters. Our expectation (dspace) cluster at NCSA, in which the attacker used the pass- is that dictionary-based passwords will belong to a cluster, word dspace@123, which has not been seen in any publicly while stolen passwords belong to another. A key requirement available dictionary. This observation indicates a targeted of clustering is a distance function between passwords. Sim- attack attempt at NCSA. Furthermore, Figure7 shows that, at ply using string distance or edit distance did not give a good 90 percentile, (from www.haveibeenpwned.com) HIBP pass- separation between password clusters. Thus, we have ex- words are attempted around 10× more frequent than default tracted key features from passwords, such as length, entropy, passwords when attacks target IoT devices. In the analysis, and related lexical statistics. Then, we projected a random we removed IoT and dictionary-based passwords from the sample of passwords in a 2D space by using t-distributed HIBP database, so that the three databases, i.e., HIBP, IoT, stochastic neighbor embedding (t-SNE) [54]. t-SNE allows and the dictionary, would be mutually exclusive. Our anal- us to capture local distances between the nearest password ysis emphasizes the popularity of credential-stuffing attacks features instead of the generic separation by running k-means concerned with automated guessing of leaked credentials that clustering. t-SNE captures the probability distribution of dis- target multiple sites. tances in high-dimensional space and low-dimensional map, Usernames in SSH attack attempts. Since usernames while PCA deals only with the linear transformation of fea- are crucial to SSH attack attempts, we analyzed the collected tures and thus may lose information. Figure5 shows three usernames against a database of NCSA usernames to find groups of passwords: i) default passwords of IoT devices, targeted attacks. However, we have not found any instance e.g., raspberry; ii) common passwords in dictionaries, e.g., of repeated attack attempts against a user account at NCSA. 007; and iii) other passwords, e.g., se7en-h0st#. This finding suggests that targeted attacks did not generate At run-time, via visualization and clustering of passwords, a lot of noises, i.e., attack attempts. Attackers carry out an t-SNE visualization helps the security team understand at- attack only when they are confident that a stolen account is tackers’ intentions better; in particular, is the attack is target- valid on the target network. ing devices, or targeting personnel accounts by using leaked passwords? Attacks that target devices should be shut down 5.5 Attack attempts using SSH keys immediately, but it’s better to closely monitor the second type In addition to guessing passwords, attackers have attempt of attack. Such monitoring allows the security team to infer to use SSH keys. Over the past few months, we observed more about the subsequent steps that attackers might take, up to 56.8K attacks that used SSH keys (see Table5). A which is important because this kind of attack behavior can total of 159 distinct SSH keys have been recorded during the lead to higher potential risk and loss if it is successful. deployment of the honeypot. An example of the top 5 SSH The long tail of password counts. We found that attack- key fingerprints in the SHA256 hash is provided in Table6. ers are shifting from using dictionaries to using leaked pass- We investigated the origin of such keys; however, we have words for targeted attacks. Figure6 is a histogram of 96K found no evidence that any one of the 159 distinct SSH keys distinct passwords in our dataset, it exhibits a long tail. Al- is leaked or bad. We used the Censys search engine [55] and though the most common passwords, e.g., 123456, have been a database of bad keys [56] to perform that assessment. being guessed for ∼ 105 times, attackers are changing their Interestingly, we grouped the source IP addresses by their strategy to guess unique and personalized passwords, explic- distinct SSH keys and found that the sources were mutually exclusive, i.e., no two of them used the same key in their

674 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association Table 5: SSH key and password attempts from 2017/12 to 6 Evaluation 2018/04). This section provides a brief history of NCSA’s production 17/12 18/01 18/02 18/03 18/04 network, in which CAUDIT has been gradually deployed. Key 0.2K 7.9K 13.6K 3.8K 56.8K It then describes our evaluation of the performance of each Pass 984.1K 951.0K 393.8K 174.0K 99.1M CAUDIT’s components. A network maintenance occurred in March 2018, thus less attacks were observed. Table 6: Top 5 SSH key fingerprints 6.1 Gradual deployment of CAUDIT in Key Fingerprint (SHA256) Count NCSA’s production network oHhjwxYH9v+ChV4Vr...Pk6KHla6P7g443w 20,307 NCSA has been a frequent attack target [32] due to its qOd/Gr8bWftEu8HDU...aNCXA3Q/0zWMCdo 17,026 unique networking infrastructure and vast computing re- YEYlq2GOCueBnJRoS...f7KzN5meQVVQFmA 9,542 sources [27]. In the past 18 years, NCSA has recorded an +UJNIlXcTgv4BLeaZ...QH//L2cG5GRQJUE 8,199 average of 19 security incidents per year (Figure8). Although oU4y6kZLH2kAdhwWU...1eBJCButjeEhIwo 7,870 in recent years, the number of security incidents has been attack attempts. Thus, we suspect that there are black markets decreasing, NCSA still observes an increasing number of for private trading of such undisclosed SSH keys. Since the brute-force attack attempts per day. Use of weak/stolen cre- honeypot attracted millions of attack attempts, this result dentials [44] to gain access in such attack attempts is still suggests that our honeypot could serve as an observatory the main method for gaining illicit access, as discussed in that measures bad SSH keys circulating in the wild. More Section3. To address this problem, we have deployed each importantly, through sharing of such SSH key attempts among component of CAUDIT as follows. sites, it will be possible to block such keys in a timely manner. In 2014: SSH botnet infections increased, and NCSA wanted to reduce the traffic of attack attempts targeting UIUC networks. 5.6 Impact of attacks In 2015: NCSA deployed a Black Hole Router to block excessive attack attempts. The credential-guessing attack attempts increased the load In 2016, as NCSA expanded to have more internal ma- and added significant noise on the network security monitor- chines and interdisciplinary researchers, it wanted to continu- ing infrastructure (i.e., the Bro IDS), even though the BHR ously audit machines on its network. Thus, it developed and mitigated the majority of the load (see Section 6.4). On deployed the SSH auditor tool. a typical day, e.g., on May 16, 2018, a total of 8,694,836 In 2017, after the /16 address space became available, alerts of attack attempts were observed. Of them, 8,361,159 NCSA developed and deployed the SSH authentication logger (96.2%) were brute-force attempts that targeted our honeypot. tool on the address space. The remaining alerts (3.8%) indicated other types of attack In 2018, as requested by peer computing sites that had attempts, such as exploitation of the Shellshock or Apache limited resources (in personnel and network bandwidth) to Struts vulnerabilities. secure their networks, NCSA started to establish an alert- In 463 days of our honeypot’s deployment, despite the sharing network and share SSH attack attempts with peer launch of 405 million key-based and password attacks, the sites. success rate of the attackers was extremely low (1 out of 405 million). There was only one major security incident, 6.2 Overall impact of our system in April 2018, in which the attacker used a stolen password Our system has contributed to annual decreases in the num- of an employee to get access to an internal cluster at NCSA. ber of critical security incidents at NCSA, i.e., from an aver- Also, there was one unsuccessful targeted attack attempt to age of 30 incidents per year during 2000–2010, down to an get access to an internal NCSA software repository (named average of seven incidents per year during 2011–2016, and "dspace"). finally down to an average of only two incidents per year in Our honeypot deployment helped to uncover eight vul- 2017-2018 (see Figure8). This is a counter-trend result, as nerable hosts before there were eslations to major security there have been increasing numbers of disclosed data leak incidents. Notably, of those hosts was a DataDirect Networks incidents in a variety of industries in recent years [57]. storage device for HPC research data that used the same pass- word that the honeypot had recorded; one host of these hosts 6.3 Honeypot was a smart device, and the other six vulnerable hosts were The honeypot is deployed on a physical server with a 14- in the internal NCSA network. The compromised core Intel Xeon CPU 2.00 GHz with 128 GB memory running smart device repeatedly scanned other hosts in the UIUC net- Enterprise Linux (RHEL). We perform stress-testing work 696 times before being shut down. This finding shows experiments to establish the capacity of the honeypot. We do that our honeypot can produce early indicators of internal so by establishing multiple SSH connections that target the network compromises. honeypot from outside of the network.

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 675 With BHR Without BHR Moving average for the number of incidents 1.0 55 60 46 0.9 y t i

38 l 36 36 i 40 34 b 31 a

b 0.8 o r

20 p 16 e

12 v 20 11 i t 0.7 a

6 6 5 l 2 2 3 3 1 u m 0 u

C 0.6 Honeypot maximum 2002 2009 2015 2018 2006 2010 2012 2004 2003 2007 2013 2017 2001 2000 2008 2005 2011 2014 2016 capacity 0.5 10 5 10 6 10 7 10 8 10 9 10 10 Figure 8: The numbers of annual security incidents at NCSA Daily count from 2000 to 2018 show a decreasing trend from an average of 30 incidents per year in 2000–2010, down to 7 incidents per Figure 9: Cumulative distribution function plot of daily attack year in 2011–2016, and 2 incidents per year in 2017–2018. counts that shows the effectiveness of the BHR. At the 50th percentile, the BHR blocks ∼ 100× of the attack attempts Capacity. The average load capacity of our honeypot is than the case when BHR is not deployed. The attack attempt 50,400 ± 4,115 SSH connections per second, which is equiv- traffic is well within the capacity of our honeypot. alent to 4.3 billion attack attempts per day, well above the observed average of 875K attack attempts per day (or the monitors for analysis. observed peak of 40M attacks per day). Thus, our honeypot 6.5 SSH credential auditor can capture all incoming attack attempts (see Figure9). The SCA performs regular audits of the NCSA network. 6.4 Black Hole Router During a year-long deployment, the SCA recorded 1,600 unique hosts and observed the following changes to the hosts. The BHR is deployed on a mixed set of Arista and Juniper Version changes. In past attacks, a change in an OpenSSH network routers that can handle at most 100 Gbps and has version meant that an attacker may have modified the an upper limit of one million routing entries for the BHR. OpenSSH server. However, that is not always the case. The goal of our BHR is to drop the most frequent attack An OpenSSH server version typically changes when an attempts at the border and thus reduce the load on the internal upgrade happens, e.g., SSH-2.0-OpenSSH_7.3 is upgraded monitoring system. We provide a representative measurement to SSH-2.0-OpenSSH_7.4. Interestingly, the SCA has also of the BHR on a typical day in September 2018. observed version downgrades. For example, when two VMs Effect of the BHR. The BHR had 300,000 unique IP ad- provisioned through OpenStack are assigned the same IP, they dresses in its block list, in which the BHR observed and might be brought up with different software configurations blocked 137,000 (45%) unique IP addresses that attempted or software stacks. Network scanners will report this change, attacks. Note that the BHR may block legitimate IP addresses however, such behavior is not necessarily malicious. Overall, (i.e., have false positives). We did not have the ground truth SCA has observed 5,500 changes to OpenSSH versions. for every IP address in the block list, except for IP addresses SSH server fingerprint changes. A server fingerprint that NCSA uses for legitimate scans. Thus, we cannot quan- uniquely identifies an OpenSSH server and is rarely changed, tify the false positives. However, NCSA’s security team unless the server is reinstalled. While version changes happen closely monitors their help desk inbox to help any legitimate often, SCA observed only 2,820 server fingerprint changes. users who have problems logging in using SSH. To date, we Thus, to use version changes and fingerprint changes as have not observed any issue from legitimate users. indicators of compromise, one needs to correlate them with The BHR also demonstrated its effectiveness in a main- upgrade cycles and VM provision events to filter out false tenance window in April 2018. Figure9 shows a ∼100× positives. increase in scanning traffic (in the 50th percentile) when the BHR did not operate. 6.6 Alert-sharing network Effect of flow shunting. The Arista network router records The alert-sharing network is capable of exchanging up to an average of 14 Gbps traffic in/out of the NCSA network. 5,000 alerts per second with all subscribed peer sites. How- Out of that, flow shunting provides a 78% (11 Gbps out of ever, it is still underused. While the deployed honeypot at 40 Gbps) reduction in network loads for network security NCSA collected the largest amount of traffic (a total of 405 monitors, e.g., by discarding encrypted traffic such as VPN, million attack attempts in 15 months, with an average of 27 SSH, and big file transfers that use GridFTP. The remaining million attack attempts per month), the other sites do not 22% of the traffic (3 Gbps) is forwarded to network security attract as much traffic because they do not have as much dedi-

676 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association cated network resources (CIDR /16 IP space) as NCSA. For from being shared or malicious insiders might intentionally example, at our international site at SUTD, only 2 million share irrelevant alerts in mimicry attacks. The second is that attack attempts have been collected in a one-month period, of ensuring the immutability of stored and shared alerts for which is 13× lower than NCSA’s number. forensic analysis. Adversarial Evolution and Adaptation. To address the 7 Discussions case in which an attacker may discover our /16 IP space and Extending CAUDIT to other networks. The measure- avoid targeting it, we will leave the address of the /16 IP ments and analyses in this paper were performed at an aca- space out of our public dataset. In the future, we will not demic site (NCSA) that implements a more open networking use the /16 IP space exclusively for the honeypot. Instead, infrastructure than corporate networks have, e.g., allowing we may start deploying a small number of legitimate servers, inter-institutional access to its internal resources. Even so, using random IP addresses, in the IP space. These legitimate NCSA makes significant use of industry-standard networking computers act as canaries and allow us to assess how they components, e.g., by extending OpenSSH to support single perform under heavy attack related traffic. sign-on4. Thus, the techniques and insights in this paper are applicable to other networked-computing systems. 8 Related Work The unique value of our honeypot deployment is not only This section discusses prior work in honeypot design, secu- that it blocks incoming SSH requests, but also that it has col- rity auditing, black hole router, and alert sharing networks. lected as many attack attempts as possible. Our measurements Honeypots. HoneyStat has been deployed for local worm aim at characterizing SSH attackers behaviors (e.g., the use of detection. However, 1) it is deployed on local networks SSH keys and personalized passwords) in the wild. As a large whereas ours is deployed on NCSA’s global peer-to-peer shar- number of observed attack attempts come from cloud/VPN ing infrastructure; and 2) it only carries out logit analysis providers, they could deploy CAUDIT internally to localize for worm detection, and thus lacks a mechanism for protect- and isolate such attempts before they mature. In the future, ing inner network infrastructure from honeypot intrusions, we will explore programming of protocol-independent packet while NCSA applies auditing tools to preempt compromises processors [58] and advanced flow-control algorithms [59,60] based on honeypot intrusion data [63]. John et al. [64] de- to deal with larger-scale traffic. ployed Web honeypots in a university network, which at- Integration with Machine Learning- based IDS. In our tracted ∼ 44,000 attacker visits from ∼ 6,000 distinct IPs, previous work, we have shown the benefit of aggregating which inspired NCSA’s honeypot deployment in a similar alert-information from a variety of network- and host-based campus deployment environment. However, NCSA’s hon- security monitors to provide machine learning based preemp- eypot traffic is at least 1,000 times greater. With that rela- tive intrusion detection capabilities to a networked system [6]. tively limited attack surface, John et al.’s honeypots rely on The password clustering analysis (described in Section 5.4) other Web pages with high page ranks and dynamic linking presented in this paper naturally feeds into such probabilistic of search engines to attract up-to-date or zero-day attacks. graphical model (PGM) based multi-stage attack detectors. Therefore, our non-interactive honeypot is scalable: it can For example, the “sophistication” of an ongoing attack can handle an order of magnitude more attack attempts compared be extracted from t-SNE model and incorporated with the to interactive honeypot such as Kippo [34], which is also decision model in a PGM, e.g., block a bot-based attack us- more expensive to maintain and pose an unnecessary risk to ing dictionary immediately or enable additional monitoring our system. Such interactive honeypots allow attackers to in- for sophisticated attacks using personalized passwords us- teract with a shell: thus, they require more resources and need ing deep packet inspection (DPI). In future work, we plan careful network configuration (blocking of new outgoing con- to study the differences between behaviors of automatically- nections) to isolate attackers. There have also been studies in and manually-generated attacks. VoIP honeypots [65,66]. The main limitation of [65] is a lack Consensus in Distributed Alert Sharing Network. With- of decisions in reaction to attackers, compared to the NCSA out coordinated alerts sharing among the sites, it is challeng- honeypot’s deployment of real-time decision infrastructure ing to preemptively detect coordinated attacks across sites based on the collected data from the honeypots. [66] has only as illustrated in our motivating example. We plan to work been implemented in a preliminary stage; it has not been with peer sites to simulate coordinated attacks, i.e., attacks deployed on a large scale, and its honeypots do not maintain that occur at the same time at multiple sites to achieve the interactions with the rest of the security components, leading overarching attack goals. We anticipate two main challenges to delayed enforcement of security policies in response to in alert sharing. The first is using redundancy to ensure the real-time dynamic attacks. timely arrival (availability [51,61,62]) of the shared alerts un- Provos [21] presented a framework that simulates virtual der a stronger threat model. For example, man-in-the-middle honeypots and Vrable et al. [22] built a prototype of vir- attackers might deliberately prevent or delay critical alerts tual honeyfarm system, both in opposition to a physical one, 4www.grid.ncsa.illinois.edu/ssh with Varable’s honeyfarm system motivated NCSA’s honey-

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 677 pot deployment. Provo’s work was driven by the IP space data among retail sites in a manner that is neither real-time limitations placed on traditional physical honeypots, and that nor encrypted. The publish of new threats in Facebook Threa- is not an issue for the NCSA honeypot. [67] looked into the tExchange [80] is not automated. However, it is promising to effectiveness of building deceptive honeypots within a virtual integrate NCSA’s alert-sharing network with Facebook Threa- environment that uses Linux containers. However, [11] illus- tExchange and IBM X-Force to make use of the APIs for trated the limitations of these virtual honeypots from both threat intelligence sharing [80, 81]. attackers, and system architecture’s points of view. 1) From the attackers’ viewpoint, there is ease of detection without any 9 Conclusion privileges; and 2) in the underlying system architecture, there This paper presents the operational experiences with the were fundamental flaws in virtualization. Other honeypots in proposed framework at the National Center for Supercomput- the literature have inspired our design decisions. However, ing Applications. Our experience over 463 days shows that their source codes are typically not available for immediate CAUDIT successfully blocks an average of 57 million attack use. attempts on a daily basis by using the proposed BHR. This Security auditing. In [68], the testbed is embedded in represents a 66× reduction in the number of SSH attempts the production network, while our system runs with the real compared to the daily average, and has reduced 78% of the production traffic that includes a mixture of attack and benign traffic to the internal network-security-monitoring infrastruc- traffic in a large-scale deployment: [69] emulated a wider ture. We posit that the measurements and insights presented range of virtual topologies, not just physical hardware but in this paper can be used to propose new research directions only to the extent of more than a thousand virtual nodes; the in IDS systems deployed in adversarial environments. NCSA honeypot scales well, such that one physical server 10 Code and Data Availability takes all the loads of attacks from 65,536 virtualized servers. It will likely be beneficial to apply the automated network We have open-sourced CAUDIT’s implementation and its monitoring tools [70,71] to prioritize and customize the net- dataset at https://pmcao.github.io/caudit. work flows [72] of the honeypot data; and to extend tech- Acknowledgement niques in [73] to formally verify authentic SSH login flows. We thank the NCSA security team, students participated Routing malicious traffic. Yu et al. proposed a precise in the SDAIA project, and the partnering sites for supporting network instrumentation framework to mitigate malicious traf- CAUDIT operational deployment; DEPEND group members, fic, e.g., by forcing the user to change the default password anonymous reviewers, and our shepherd, Prof. Vyas Sekar, of an IoT device [70] instead of redirecting the user to the for providing valuable feedback; and Ms. Jenny Applequist null route as it is done in our approach. Wu et al. presented a for proofreading. This material is based upon work sup- packet filter [74] with low filter update latency and high-speed ported by the National Science Foundation under Grant No packet processing. 007 is an application deployed to diagnose, 1535070,1547249. The opinions, findings, and conclusions detect, and trace source causes for TCP connection packet stated herein are those of the authors and do not necessarily drops [75]. APUNet integrated GPU in APU platforms to ac- reflect those of the sponsors. celerate packet processing in network applications [76], while, on the other hand, Netmap enables rapid network packet de- livery without requiring for customized hardware or modified applications [77]. Sarma et al. broadened the availability of hardware switches for network resource allocation algo- rithms, thus making the implementations of network protocols more flexible [78]. To achieve easier and more efficient net- work flow processing development in stateful middleboxes, Jamshed et al. designed and implemented a reusable network stack [79]. Network auditors or scanners. Compared to ZMap, for which analyses of new protocols were performed on random samples [23], NCSA’s honeypot has adopted a more intelli- gent scanning methodology that subscribes to new protocols logged by IDS to incrementally discover newly added SSH servers, therefore lowering the burden of probe traffic on the production network. Alert-sharing network. R-cisc is a cybersecurity sharing center for retail ecosystem [26]. However, unlike NCSA’s sharing network, that sharing center shares security incident

678 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association References [12] Vladimir B Oliveira, Zair Abdelouahab, Denivaldo [1] Theophilus Benson, Aditya Akella, and David A Maltz. Lopes, Mario H Santos, and Valéria P Fernandes. Hon- Unraveling the complexity of network management. In eypotlabsac: a virtual honeypot framework for android. NSDI, pages 335–348, 2009. International Journal of Networks & Commu- nications, 5(4):159, 2013. [2] Taous Madi, Suryadipta Majumdar, Yushun Wang, Yosr [13] Todd Hoff. Netflix: Continually test by failing servers Jarraya, Makan Pourzandi, and Lingyu Wang. Auditing with chaos monkey, 2010. security compliance of the virtualized infrastructure in the cloud: Application to openstack. In Proceedings [14] Yury Izrailevsky and Ariel Tseitlin. The netflix simian of the Sixth ACM Conference on Data and Application army. The Netflix Tech Blog, July, 2011. Security and Privacy, pages 195–206. ACM, 2016. [15] Catello Di Martino, Ugo Giordano, Nishok Mo- [3] Nicholas DeMarinis, Stefanie Tellex, Vasileios Ke- hanasamy, Stefano Russo, and Marina Thottan. In pro- merlis, George Konidaris, and Rodrigo Fonseca. Scan- duction performance testing of sdn control plane for ning the internet for ros: A view of security in robotics telecom operators. In 2018 48th Annual IEEE/IFIP research. arXiv preprint arXiv:1808.03322, 2018. International Conference on Dependable Systems and Networks (DSN), pages 642–653. IEEE, 2018. [4] Tianlong Yu, Vyas Sekar, Srinivasan Seshan, Yuvraj [16] The bro network security monitor. 2018. https:// Agarwal, and Chenren Xu. Handling a trillion (unfix- bro.org. able) flaws on a billion devices: Rethinking network security for the internet-of-things. In Proceedings of the [17] Yu-Ming Ke, Chih-Wei Chen, Hsu-Chun Hsiao, Adrian 14th ACM Workshop on Hot Topics in Networks, page 5. Perrig, and Vyas Sekar. Cicadas: congesting the internet ACM, 2015. with coordinated and decentralized pulsating attacks. In Proceedings of the 11th ACM on Asia Conference on [5] Have i been pwned, 2018. https://haveibeenpwned. Computer and Communications Security, pages 699– com/. 710. ACM, 2016.

[6] Phuong Cao, Eric Badger, Zbigniew Kalbarczyk, Rav- [18] Mobin Javed and Vern Paxson. Detecting stealthy, dis- ishankar Iyer, and Adam Slagell. Preemptive intrusion tributed ssh brute-forcing. In Proceedings of the 2013 detection: Theoretical framework and real-world mea- ACM SIGSAC conference on Computer & communica- surements. In Proceedings of the 2015 Symposium and tions security, pages 85–96. ACM, 2013. Bootcamp on the Science of Security, page 5. ACM, [19] Ryan J. McCaughey. Deception using an ssh honeypot. 2015. [20] D M’raihi, M Bellare, F Hoornaert, D Naccache, and [7] You’re already compromised: Exposing ssh as an attack O Ranen. Hotp: An hmac-based one-time password vector, 2016. https://www.cisco.com/c/en/us/ algorithm. Technical report, 2005. about/security-center/ssh-honeypot.html. [21] Niels Provos et al. A virtual honeypot framework. In [8] Drawing the foul: Operation of a ddos honeypot, USENIX Security Symposium, volume 173, pages 1–14, 2017. https://www.usenix.org/conference/ 2004. enigma2017/summit-program/presentation/ [22] Michael Vrable, Justin Ma, Jay Chen, David Moore, drawing-foul-operation-ddos-honeypot. Erik Vandekieft, Alex C Snoeren, Geoffrey M Voelker, [9] Ping Wang, Lei Wu, Ryan Cunningham, and Cliff C and Stefan Savage. Scalability, fidelity, and containment ACM SIGOPS Zou. Honeypot detection in advanced botnet attacks. in the potemkin virtual honeyfarm. In International Journal of Information and Computer Se- Operating Systems Review, volume 39, pages 148–162. curity, 4(1):30–51, 2010. ACM, 2005. [23] Zakir Durumeric, Eric Wustrow, and J Alex Halderman. [10] Payas Gupta, Bharat Srinivasan, Vijay Balasubra- Zmap: Fast internet-wide scanning and its security ap- maniyan, and Mustaque Ahamad. Phoneypot: Data- plications. In USENIX Security Symposium, volume 8, driven understanding of telephony threats. In NDSS, pages 47–53, 2013. 2015. [24] Warren Kumari and Danny McPherson. Remote trig- [11] Maximillian Dornseif, Thorsten Holz, and Und Sven gered black hole filtering with unicast reverse path for- Müller. Honeypots and limitations of deception. 2005. warding (urpf). Technical report, 2009.

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 679 [25] Andreas Kuehn and Milton Mueller. Analyzing bug Emerging networking experiments and technologies, bounty programs: An institutional perspective on the pages 253–264. ACM, 2012. economics of software vulnerabilities. 2014. [37] Phuong Cao, Eric C Badger, Zbigniew T Kalbarczyk, [26] R-cisc, 2018. https://r-cisc.org/ and Ravishankar K Iyer. A framework for generation, #homeResources. replay, and analysis of real-world attack variants. In Proceedings of the Symposium and Bootcamp on the [27] Catello Di Martino, Zbigniew Kalbarczyk, Ravis- Science of Security, pages 28–37. ACM, 2016. hankar K Iyer, Fabio Baccanico, Joseph Fullop, and William Kramer. Lessons learned from the analysis of [38] Cuong Pham, Zachary J Estrada, Phuong Cao, Zbigniew system failures at petascale: The case of blue waters. In Kalbarczyk, and Ravishankar K Iyer. Building reliable Dependable Systems and Networks (DSN), 2014 44th and secure virtual machines using architectural invari- Annual IEEE/IFIP International Conference on, pages ants. IEEE Security & Privacy, 12(5):82–85, 2014. 610–621. IEEE, 2014. [39] Cuong Pham, Zachary Estrada, Phuong Cao, Zbigniew [28] Nds services. 2018. https://wiki.ncsa.illinois. Kalbarczyk, and Ravishankar K Iyer. Reliability and edu/display/NDS/NDS+Services. security monitoring of virtual machines using hardware [29] Manos Antonakakis, Tim April, Michael Bailey, Matt architectural invariants. In Dependable Systems and Bernhard, Elie Bursztein, Jaime Cochran, Zakir Du- Networks (DSN), 2014 44th Annual IEEE/IFIP Interna- rumeric, J. Alex Halderman, Luca Invernizzi, Michalis tional Conference on, pages 13–24. IEEE, 2014. Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, [40] Robert P Goldberg. Architecture of virtual machines. Kurt Thomas, and Yi Zhou. Understanding the mirai In Proceedings of the workshop on virtual computer botnet. In 26th USENIX Security Symposium (USENIX systems, pages 74–112. ACM, 1973. Security 17), pages 1093–1110, Vancouver, BC, 2017. USENIX Association. [41] man page for passwd, 2018. https://www.unix.com/ man-page/linux/1/passwd/. [30] Fermi national accelerator laboratory, 2018. http:// www.fnal.gov/. [42] Ssh server auditing, 2018. https://github.com/ arthepsy/ssh-audit. [31] Antonio Pecchia, Aashish Sharma, Zbigniew Kalbar- czyk, Domenico Cotroneo, and Ravishankar K Iyer. [43] Gordon Fyodor Lyon. Nmap network scanning: The Identifying compromised users in shared computing in- official Nmap project guide to network discovery and frastructures: A data-driven bayesian network approach. security scanning. Insecure, 2009. In 2011 30th IEEE International Symposium on Reliable Distributed Systems, pages 127–136. IEEE, 2011. [44] Aashish Sharma, Zbigniew Kalbarczyk, R Iyer, and James Barlow. Analysis of credential stealing attacks in [32] Aashish Sharma, Zbigniew Kalbarczyk, James Barlow, an open networked environment. In Network and System and Ravishankar Iyer. Analysis of security data from a Security (NSS), 2010 4th International Conference on, large computing organization. 2011. pages 144–151. IEEE, 2010. [33] Iforge cluster, 2018. http://www.ncsa.illinois. [45] fail2ban, 2018. https://www.fail2ban.org/wiki/ edu/industry/iforge. index.php/Main_Page. [34] Kippo - ssh honeypot, 2016. https://github.com/ desaster/kippo. [46] Liang Zhang, David Choffnes, Dave Levin, Tudor Du- mitras, Alan Mislove, Aaron Schulman, and Christo [35] Cláudia J Barenco Abbas, L Javier García Villalba, and Wilson. Analysis of ssl certificate reissues and revoca- Victoria López López. Implementation and attacks anal- tions in the wake of heartbleed. In Proceedings of the ysis of a honeypot. In International Conference on Com- 2014 Conference on Internet Measurement Conference, putational Science and Its Applications, pages 489–502. pages 489–502. ACM, 2014. Springer, 2007. [47] Bpf and xdp reference guide, 2018. https://cilium. [36] Nikhil Handigol, Brandon Heller, Vimalkumar Jeyaku- readthedocs.io/en/latest/bpf/. mar, Bob Lantz, and Nick McKeown. Reproducible network experiments using container-based emulation. [48] Gianluca Insolvibile. Kernel korner: Inside the linux In Proceedings of the 8th international conference on packet filter. Linux journal, 2002(94):7, 2002.

680 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association [49] Recommendation for pair-wise key establish- In Proceedings of the 9th USENIX conference on Net- ment schemes using discrete logarithm cryp- worked Systems Design and Implementation, pages 29– tography, 2018. https://csrc.nist.gov/ 29. USENIX Association, 2012. CSRC/media/Publications/sp/800-56a/rev- 3/draft/documents/sp800-56ar3-draft.pdf. [62] Cuong Pham, Phuong Cao, Zbigniew Kalbarczyk, and Ravishankar K Iyer. Toward a high availability cloud: [50] Pieter Hintjens. ZeroMQ: messaging for many applica- Techniques and challenges. In Dependable Systems and tions. " O’Reilly Media, Inc.", 2013. Networks Workshops (DSN-W), 2012 IEEE/IFIP 42nd International Conference on, pages 1–6. IEEE, 2012. [51] Sushant Jain, Michael Demmer, Rabin Patra, and Kevin Fall. Using redundancy to cope with failures in a delay [63] David Dagon, Xinzhou Qin, Guofei Gu, Wenke Lee, Ju- tolerant network. In ACM SIGCOMM Computer Com- lian Grizzard, John Levine, and Henry Owen. Honeystat: munication Review, volume 35, pages 109–120. ACM, Local worm detection using honeypots. In International 2005. Workshop on Recent Advances in Intrusion Detection, [52] Joe DeBlasio, Stefan Savage, Geoffrey M Voelker, and pages 39–58. Springer, 2004. Alex C Snoeren. Tripwire: Inferring internet site com- [64] John P John, Fang Yu, Yinglian Xie, Arvind Krishna- promise. In Proceedings of the 2017 Internet Measure- murthy, and Martín Abadi. Heat-seeking honeypots: ment Conference, pages 341–354. ACM, 2017. design and experience. In Proceedings of the 20th inter- [53] Phuong Cao, Hongyang Li, Klara Nahrstedt, Zbigniew national conference on World wide web, pages 207–216. Kalbarczyk, Ravishankar Iyer, and Adam J Slagell. Per- ACM, 2011. sonalized password guessing: a new security threat. In [65] Mohamed Nassar, Radu State, and Olivier Festor. Voip Proceedings of the 2014 Symposium and Bootcamp on honeypot architecture. In 2007 10th IFIP/IEEE Interna- the Science of Security, page 22. ACM, 2014. tional Symposium on Integrated Network Management, [54] Laurens van der Maaten and Geoffrey Hinton. Visu- pages 109–118. IEEE, 2007. alizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008. [66] Rodrigo Do Carmo, Mohamed Nassar, and Olivier Fes- . Artemisa: An open-source honeypot back-end to [55] Censys, 2018. https://censys.io/. support security in voip domains. In Integrated Net- work Management (IM), 2011 IFIP/IEEE International [56] Ssh bad keys, 2017. https://github.com/rapid7/ Symposium on, pages 361–368. IEEE, 2011. ssh-badkeys. [67] Alexander Kedrowitsch, Danfeng Daphne Yao, Gang [57] Verizon RISK Team and R Team. 2018 data breach Wang, and Kirk Cameron. A first look: Using linux investigations report. 2018. containers for deceptive honeypots. In Proceedings of [58] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick the 2017 Workshop on Automated Decision Making for McKeown, Jennifer Rexford, Cole Schlesinger, Dan Active Cyber Defense, pages 15–22. ACM, 2017. Talayco, Amin Vahdat, George Varghese, et al. P4: Programming protocol-independent packet processors. [68] Rob Sherwood, Glen Gibb, Kok-Kiong Yap, Guido Ap- ACM SIGCOMM Computer Communication Review, penzeller, Martin Casado, Nick McKeown, and Guru M 44(3):87–95, 2014. Parulkar. Can the production network be the testbed? In OSDI, volume 10, pages 1–6, 2010. [59] Zaoxing Liu, Antonis Manousis, Gregory Vorsanger, Vyas Sekar, and Vladimir Braverman. One sketch to [69] M Hibler R Ricci L Stoller, Jonathon Duerig, Shashi rule them all: Rethinking network flow monitoring with Guruprasad, Tim Stack, Kirk Webb, and Jay Lepreau. univmon. In Proceedings of the 2016 ACM SIGCOMM Large-scale virtualization in the emulab network testbed. Conference, pages 101–114. ACM, 2016. In USENIX Annual Technical Conference, Boston, MA, 2008. [60] Seyed Kaveh Fayaz, Yoshiaki Tobioka, Vyas Sekar, and Michael Bailey. Bohatei: Flexible and elastic ddos [70] Tianlong Yu, Seyed Kaveh Fayaz, Michael P Collins, defense. Vyas Sekar, and Srinivasan Seshan. Psi: Precise security instrumentation for enterprise networks. 2017. [61] Costin Raiciu, Christoph Paasch, Sebastien Barre, Alan Ford, Michio Honda, Fabien Duchene, Olivier Bonaven- [71] Michael Golightly and Jack Brassil. Automating net- ture, and Mark Handley. How hard can it be? de- work monitoring on experimental testbeds. In CSET, signing and implementing a deployable multipath tcp. 2011.

USENIX Association 16th USENIX Symposium on Networked Systems Design and Implementation 681 [72] Kimberly C. Claffy, H-W Braun, and George C. Polyzos. A parameterizable methodology for internet traffic flow profiling. IEEE Journal on selected areas in communi- cations, 13(8):1481–1494, 1995. [73] Shuo Chen, Matt McCutchen, Phuong Cao, Shaz Qadeer, and Ravishankar K Iyer. Svauth–a single-sign- on integration solution with runtime verification. In In- ternational Conference on Runtime Verification, pages 349–358. Springer, 2017. [74] Zhenyu Wu, Mengjun Xie, and Haining Wang. Swift: A fast dynamic packet filter. In NSDI, volume 8, pages 279–292, 2008. [75] Behnaz Arzani, Selim Ciraci, Luiz Chamon, Yibo Zhu, Hingqiang Liu, Jitu Padhye, Boon Thau Loo, and Ge- off Outhred. 007: Democratically finding the cause of packet drops. In 15th {USENIX} Symposium on Net- worked Systems Design and Implementation ({NSDI} 18). USENIX Association, 2018. [76] Younghwan Go, Muhammad Asim Jamshed, Young- Gyoun Moon, Changho Hwang, and KyoungSoo Park. Apunet: Revitalizing gpu as packet processing accelera- tor. In NSDI, pages 83–96, 2017. [77] Luigi Rizzo. Netmap: a novel framework for fast packet i/o. In 21st USENIX Security Symposium (USENIX Security 12), pages 101–112, 2012.

[78] Naveen Kr Sharma, Antoine Kaufmann, Thomas E An- derson, Arvind Krishnamurthy, Jacob Nelson, and Si- mon Peter. Evaluating the power of flexible packet processing for network resource allocation. In NSDI, pages 67–82, 2017. [79] Muhammad Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han, and KyoungSoo Park. mos: A reusable networking stack for flow monitoring mid- dleboxes. In NSDI, pages 113–129, 2017.

[80] Getting started with threatexchange, 2018. https: //developers.facebook.com/docs/threat- exchange/getting-started/v3.1. [81] Ibm x-force, 2018. https://www.ibm.com/ security/xforce.

682 16th USENIX Symposium on Networked Systems Design and Implementation USENIX Association