The Forseti Distributed File Integrity Checker

Cluster Security with NVisionCC: The Forseti Distributed File Integrity Checker Adam J. Lee†‡, Gregory A. Koenig†‡, and William Yurcik† †National Center for Supercomputing Applications ‡Department of Computer Science University of Illinois at Urbana-Champaign {adamlee, koenig, byurcik}@ncsa.uiuc.edu Abstract for a number of malicious actions. In many cases, the password used to log into a single node can be used Attackers who are able to compromise a single node to access a large number of nodes in the system, al- in a high performance computing cluster can use that lowing the attacker to utilize the vast computing and node as a launch point for a number of malicious ac- storage capabilities of the compromised cluster to carry tions. In many cases, the password used to log into out brute-force password cracking, launch distributed a single node can be used to access a large number of denial of service attacks, or serve illegal digital content. nodes in the system, allowing the attacker to utilize the Additionally, an attacker can listen to network traffic vast computing and storage capabilities of the compro- for other users to log into their compromised nodes in mised cluster to sniff network traffic, carry out brute- hopes of learning passwords for other accounts on these force password cracking, launch distributed denial of systems. service attacks, or serve illegal digital content. Often, The recent rise in popularity of grid computing has these types of attackers modify important system files exacerbated this problem by increasing the amount to collect passwords to other accounts, disable certain of damage that can be caused with a single compro- logging facilities, or create back-doors into the system. mised account. Often times, a password used to log In this paper, we present Forseti, a distributed file into one cluster node can be used not only to log into integrity checker designed specifically for the high per- other nodes in the same cluster, but also to log into formance computing cluster environment. Forseti was other clusters located throughout the computational designed to address the shortcomings exhibited by ex- grid. This implies that the compromise of a single ac- isting host-based intrusion detection systems when used count on a single cluster node could give an attacker the in the cluster environment and to provide a means of ability to access many thousands of geographically dis- detecting changes to critical system files made by root- tributed cluster nodes. The high utilization of clusters level adversaries. We discuss the design and implemen- participating in grid computing systems allows an at- tation of the Forseti system, present a security analysis tacker sniffing for passwords on a compromised node to of Forseti, examine the performance of the system, and gain access to an extremely high number of other user explore how Forseti can be used in concert with other accounts. This technique is not conjecture and was, security monitoring techniques to enhance the security in fact, used during the TeraGrid compromises that af- of the HPC cluster environment. fected a large number of clusters world-wide during the Spring of 2004 [11]. During this time period, attackers installed Trojan-horse sshd processes designed to cap- 1 Introduction ture the passwords of users logging into compromised nodes so that these passwords could then be used to Over the course of the last two years, the security compromise other nodes throughout the grid. of large-scale commodity clusters has become a topic Host-based file integrity checkers (e.g., [2, 4, 6, 10, of increasing research and practical interest. Attack- 12, 15, 18, 24]) have been used to help detect these ers who are able to compromise a single node in one types of attacks on hosts in enterprise computing sys- of these clusters can use that node as a launch point tems for a number of years. These tools are activated periodically (usually by means of a scheduler such as acts. In Section 4, we discuss the implementation of cron), to check various properties of critical system our distributed integrity checker and detail its inter- files, including access permissions, timestamps, and actions with cluster nodes and NVisionCC. We revisit content hashes. These properties are compared to ref- our threat model and discuss the performance of this erence values stored in a read-only database; files whose scanner in Section 5, and address related work in Sec- values differ from the reference cause an alert to be tion 6. We then present our conclusions and directions raised, such as an entry in the system log or an email for future work in Section 7. being sent to the system administrator. Though these systems have worked well in enterprise computing, they 2 Threat Model suffer several shortcomings when used in large-scale commodity clusters. Installing and managing these tools on the hundreds or thousands of nodes in a large Throughout this paper, we focus on the use of file cluster is a time-consuming process involving changes integrity checking as a single tool at the disposal of sys- to each node when system software is upgraded. Ad- tem administrators tasked with monitoring the security ditionally, if the signature database is stored on each state of large-scale commodity clusters. As such, we do node, attackers can relatively easily change the the ref- not advocate the use of Forseti to detect all forms of erence copy of the file information. Lastly, effectively cluster intrusion, but rather to locate only a particu- logging the access violations can pose a problem if an lar class of attacks. Any anomalies detected through attacker has gained root access to the machine, as they the use of file integrity checking are meant to be cross- could disrupt the logging mechanism and easily cover referenced and correlated with the findings of other se- their tracks. curity monitoring techniques—a task which our com- prehensive cluster monitoring tool, NVisionCC, is de- To address the shortcomings of host-based file in- signed to carry out (e.g., by examining running system tegrity checkers while retaining their strengths, we have processes [13], open network ports [14], or detecting designed Forseti, a distributed file integrity checker privilege escalations [20]). that meets the unique needs of the HPC cluster en- File integrity checking allows system administrators vironment. In our tool, the file integrity database is to be notified when the contents or permissions of key separated from the nodes to be monitored and by lever- system files is changed. Many times, attackers who aging the emergent properties of the HPC cluster com- compromise nodes in a cluster do so with the intent puting environment [25], we can significantly reduce of carrying out password gathering attacks to escalate the size of this database. Cluster nodes cannot access their privilege set and gain access to other nodes or the database, meaning that an attacker who has com- clusters. These attacks are usually carried out through promised some number of nodes in the system cannot the use of rootkits or Trojan-horse system utilities. The alter the stored file signatures. Our system relies on a tool presented in this paper focuses on attacks executed collector node (not necessarily part of the cluster) to by a root-level adversary that involve the modification carry out the integrity scans by making remote connec- of system files. In particular, we assume our adversary tions to the nodes in the system, pushing the necessary has the ability to insert, reorder, and replay messages software to the system, and carrying out the scan. In sent on the network, arbitrarily modify files stored on this way, Forseti has no software footprint on the nodes the compromised nodes, and perform any number of to be monitored and adding a new node to be scanned administrative tasks (e.g., starting and stopping sys- is as simple as updating the database to inform the tem services) on the nodes which have been compro- collector node of this change. This system is robust mised. Throughout this paper, we assume that any to individual node compromises and, though not fool- individual node in the cluster can be compromised and proof, is significantly harder to defeat than other host- used to carry out arbitrary tasks on behalf of an at- based systems performing similar function; had Forseti tacker. We relax this assumption and discuss the use been deployed during the Spring of 2004, the TeraGrid of Forseti in single-image clusters (e.g., Clustermatic attacks could have been detected and stopped much systems) in Section 5.4. earlier than they were. The rest of this paper is organized as follows. In Section 2 we describe our threat model and discuss the 3 Design conditions in which integrity monitoring can aid in the security monitoring of large-scale commodity clusters. In this section, we comment briefly on the emergent Section 3 briefly discusses the design of NVisionCC, properties of large-scale commodity clusters and de- the cluster monitoring tool with which Forseti inter- scribe the ways in which these properties can be used Information Collection Host Cluster Nodes Security Collector Web Server Parameters Database 1 2 4 Alerts 4 3 Security Analysis Security Analyzer Management Nodes Figure 1. NVisionCC system architecture during security monitoring. We then discuss the ar- properties can be leveraged to increase security situ- chitecture of NVisionCC, a cluster security monitoring ational awareness in the cluster environment. Rather system designed to leverage these emergent properties. than being concerned with managing thousands of pos- Lastly, we overview the architecture of the Forseti dis- sible node configurations, as in an enterprise comput- tributed file integrity checker and discuss how it fits ing environment, the emergent structure of large-scale into the NVisionCC monitoring framework.

Load more