lsof COVER STORY
Looking for intruders with lsof QUICK CHECK
Track down and expose intruders with the versatile admin tool lsof.
BY CASPAR CLEMENS MIERAU
as your server been cracked? obtaining information that would other- has gotten comfortable. If you are seri- Are your processes running wise require a whole collection of admin ous about using lsof for intrusion detec- Hwild? If you suspect an intru- utilities. tion, leave out the make install step after sion, you’ll need accurate information As the adage goes, “everything is a compiling and manually move the bi- on what’s happening with the system. file” in Unix. Almost all activities on a nary to a write-protected medium such Open file handles are a useful source Unix-like system bear some relation to as a CD ROM. Of course, if a sophisti- for this information. lsof [1] scans the an open file. Unix-style systems use reg- cated attacker has directly modified the depths of the filesystem for these files ular files, special block files, executables, kernel (through a kernel rootkit, for ex- and then returns comprehensive and libraries, directo- detailed output. ries, internal data To be fully prepared for an attack, streams (Unix Do- you’ll need an Intrusion Detection Sys- main Sockets), tem (IDS) like Snort, Tripwire or Aide to and network con- check the filesystem and data streams nections. lsof is for suspicious patterns. However, if you able to centrally don’t have the time or resources for a collect and syn- full-blown intrusion response, Linux has thesize all this a number of standard command line information into programs capable of discovering tell-tale meaningful clues traces on a system. The usual suspects about the nature for server diagnosis are ps, netstat, top, of an attack. fuser, and other friendly helpers. Like any utility, lsof is a single tool that provides a lsof is subject to summary of similar system information. manipulation Figure 1: The promising glsof tool gives GUI fans easy access to filter You can use lsof as a single source for once the attacker settings.
WWW.LINUX - MAGAZINE.COM ISSUE 77 APRIL 2007 29 COVER STORY lsof
ample) the output of lsof will be unreli- You can manipulate the output for pro- able even if the tool itself is untouched. cessing with other tools using lsof -F. However, as you’ll learn in this article, Special formatting helps the downstream many attackers try tricks that aren’t tools parse the individual fields (see the especially sophisticated and are easily manpage section Output for other pro- exposed with a tool like lsof. grams for details). lsof is no substitute for a full-featured IDS, but if you are too late for that or if Flood of Information you aren’t interested in implementing Calling lsof without setting parameters or managing a more comprehensive returns too much information to provide system, you can still use lsof to look a useful overview – the flood of informa- for footprints. tion would scare off many users. How- Figure 2: The Jlsof filter dialog tells you the ever, command line parameters can help correct lsof parameters. Although this might Investigations lsof concentrate on the data you need. If confuse GUI-only users, it does make it eas- Table 1 lists a number of examples for you combine multiple parameters, lsof ier for users to move to the command line. investigating a system. If you enable assumes a logical OR operation by de- lsof’s security option, only root will re- fault; however, you can specify -a for an ceive detailed output for these com- AND operation (last line in Table 1). However, this method is as drastic as it mands. In secure mode, lsof will only Identifying processes that are prevent- is effective. show users the details that directly affect ing users from unmounting a storage them, however, even in insecure mode, medium is a typical task for lsof. Calling Expectations lsof gives users without root privileges lsof with the -t directoryname option The commands listed in Table 1 are fine fewer details, as you need root privileges returns a list of numeric process IDs for discovering important facts about a to access the details in /proc. accessing the CD-ROM: system before or after an attack. To de- lsof uses a tabular format to output fend yourself against invaders, you need the information filtered as specified by $ umount /dev/cdrom to be familiar with the normal status, the parameter list, including the follow- umount: /cdrom: device is busy and to be aware of where suspicious ing columns by default: $ kill -9 `lsof -t /dev/cdrom` entries are likely to appear. • Process name: COMMAND $ umount /dev/cdrom The following example uses a tradi- • Process ID: PID $ eject tional LAMP system (Linux, Apache, • Name of system user account under which the process is running: USER Finding and Building lsof • File descriptor: FD Lsof supports a number of Unix deriva- obtained from an insecure source), con- • File type: TYPE tives, and it is probably part of your basic figure the source code, and compile it. • Device: DEVICE Linux system, or at least it should reside During the configuration phase, you are • Size: SIZE in the standard repository. To install on prompted to make a few decisions. The • Connection: NODE Debian, for example, you just need to HASSECURITY and HASNOSOCKSECU- • Full name: NAME issue a apt-get install lsof command. RITY options are important. If you would The lean package has no dependencies, only like the root user to be able to use Listing 1: Compiling lsof apart from the mandatory Libc 6. lsof to list open files and sockets for all This said, there are two reasons for users, you need to answer [y] and [n]. 01 wget ftp://lsof.itap.purdue. avoiding the prebuilt binary: system The inconsistent terminology does tend edu/pub/tools/unix/lsof/lsof. compatibility and security. As lsof’s de- to be confusing. tar.bz2 veloper Vic Abel points out in the FAQ On completing the build, ./lsof -v tells 02 tar xjf lsof.tar.bz2 [2], you can only guarantee a full feature you the options it was compiled with. set and optimum stability, if you build The Only root can list all files message 03 cd lsof_4.77 the current lsof version on the target ma- means that normal users will be unable 04 wget ftp://lsof.itap.purdue. chine, since lsof digs deep into the sys- to misuse the program to list system- edu/pub/Victor_A_Abell.gpg tem architecture and kernel. It is always critical information. (Restricting access 05 gpg --import Victor_A_Abell. better to obtain tools you will be using to lsof is a rather cosmetic security solu- gpg for preventive or forensic analysis from tion, since much of the information avail- a safe source and not to mix them with able through lsof can also be obtained 06 gpg --verify lsof_4.77_src. standard system tools to avoid the dan- with tools such as ps and netstat, al- tar.sig lsof_4.77_src.tar ger of manipulation by rootkits. though the process may not be quite as 07 tar xf lsof_4.77_src.tar The lsof sources are easily compiled, convenient. The prebuilt versions in vari- ous distributions handle security differ- 08 cd lsof_4.77_src so you might as well build a version that matches your system. The commands ently. Debian grants non-administrative 09 ./Configure linux in Listing 1 grab the sources of the net- users unrestricted use of lsof, whereas 10 make -s work; use GnuPG to check the signature Red Hat Enterprise applies restrictions. 11 ./lsof -v (note that the key in our example was
30 ISSUE 77 APRIL 2007 WWW.LINUX - MAGAZINE.COM lsof COVER STORY
be able to open investigates the suspicious processes for log files. In con- details of network connections, libraries trast to this, data that have been loaded, open files, and communications many other things. are handled by As malevolent hackers tend to use unprivileged pro- their own FTP, IRC, telnet, or SSH serv- cesses. Thus, a ers, initial analysis should include web server offers searching for open ports. The lsof -a -i -u a guessable con- www-data | grep LISTEN command lists figuration of users, all the IP sockets (-i), which sockets the executable files, Apache user has opened, (-u www-data), and open ports. and which are listening for connections For example, (this explains grep LISTEN). Figure 3: The Java-based JLsof tool converts lsof output to a simple www-data runs Everything apart from 80 (HTTP) and table. /usr/sbin/apache2 443 (HTTPS) is suspicious. Although a on Debian. call to netstat will give you similar MySQL, PHP). The administrator notices results, lsof can help you perform more an enormous increase in network load What’s Running Where? detailed analysis without needing to that doesn’t reflect the number of page In this context, we need a call to lsof -a switch to another tool. hits. The administrator suspects that an -d txt -u www-data to list processes that attacker has injected a trojan that copies execute the file /usr/sbin/apache2 as the The Real World files over the wire, launches distributed www-data user account. The -a option Apache and PHP exploits are fairly com- network attacks or sends spam mail. In gives us a logical AND, -d txt lists exe- mon. Listings 2a and 2b show two ex- a LAMP environment, the PHP system cuted files only, and -u www-data re- cerpts from the lsof logs on compro- interface is one of the major targets, as stricts the output to just one user. Under mised servers, and they are all I need to PHP suffers from a couple of design normal circumstances, this will give you diagnose an attack. The output results weaknesses [6], but poorly crafted just the Apache processes. from analyzing processes belonging to scripts can just as easily give an attacker If an attacker manages to manipulate the www-data account. See Listing 3 for a foothold. PHP or your PHP scripts and execute another abridged example. If you are familiar with typical PHP system commands and programs on the In the first example, the attacker attack patterns, you will probably al- server, these commands and processes exploits an obsolete version of W-Agora ready have guessed what kind of infor- will typically run under the same ac- (online forum software) and a directory mation you need to look for with lsof. count as Apache – that is, unless the without write protection (Listing 2a, The Apache web server runs under its attacker has escalated his privileges and Line 2: /home/user/public_html/ own user account by default, www-data gained root access by exploiting other w-agora/). (Debian), apache, httpd, or if the worst security holes. comes to the worst, nobody (this ac- Finding processes that belong to the Listing 2a: Bash count is normally reserved for NFS). Apache user and that also access other Camouflage Typically, additional processes will run binaries or open unexpected ports, 01 COMMAND PID USER FD as root to support privileged ports and to should set off the alarms. lsof -p PID TYPE DEVICE SIZE NODE Table 1: lsof Examples NAME 02 bash 30334 www-data cwd Command Explanation DIR 3,8 4096 1571340 / lsof Without any parameters, the command gives you an overview. lsof /bin/ bash Lists all processes that use bash. home/user/public_html/ lsof -p PID Lists the open files for the process with the specified process ID. w-agora/.m lsof +D /tmp Lists all open files in /tmp and its subdirectories without symbolic links. 03 bash 30334 www-data txt lsof -u Benutzer Lists all open files for the specified user. REG 3,8 496231 1571405 / lsof -u ^root Lists all open files, except for those opened by root. home/user/public_html/ lsof -d txt Displays a process list, similar to ps aux, by listing entries with the file w-agora/.m/bash descriptor entry txt, instead of the normal number (txt refers to program code and data, that is, for executed files). 04 bash 30334 www-data 0w lsof +L1 Displays all deleted files that are still open, and thus still occupy disk REG 3,8 125 1571408 / space, but are not part of any directory (files with less than one link). home/user/public_html/ lsof -i Network-related files. w-agora/.m/LinkEvents lsof -i -P -n All network-related files without the port number as a service identifier, 05 bash 30334 www-data 2u and without resolving hostnames (for faster response). IPv4 4709341 TCP lsof -i6 Shows IPv6-related files. server.com:40001->undernet. lsof -i | grep ‘\->‘ All active connections. lsof -a -i -u www-data All open network files for the www-data account (AND relation -a). xs4all.nl:ircd ESTABLISHED)
WWW.LINUX - MAGAZINE.COM ISSUE 77 APRIL 2007 31 COVER STORY lsof
The attacker has created a new direc- count has a num- tory .m to use as a working directory ber of open ports, (Line 2, Column FD: Current Working including on the Directory). The attacker has uploaded C Psybnc IRC proxy. files to the directory and then compiled The unique pro- and executed the files using a harmless- cess name of sounding account name of bash. psybnc (Lines 5 However, as you can see in Line 5, the through 8) is a programs are not as harmless as the real give-away, name might suggest; this bash has an but at least there open connection to an IRC server. Plus, is an attempt to bash has written data to the LinkEvents hide the processes file, which is obvious from the file de- behind a familiar scriptor 0w (that is, bash has opened name – as the stdout for writing). name server bind in Line 9. Cheeky but Dumb In fact, this is a Our cyber criminal is really cheeky, but patched SSH the attacker’s methods reveal more self- server that grants Figure 4: Sloth provides a native lsof interface for Mac OS X. confidence than technical ability – espe- system access to cially considering the fact that he has www-data without requiring a password. lsof, it makes sense to monitor a list of not bothered to cover his tracks. Hiding There also is a server process with the open ports, adding process names, user- the directory by starting the directory suspicious name of a (see Lines 2 names, and interfaces. with a dot and using bash as the account through 4). The command shown in Line 1 of List- name for the processes, are both begin- ing 3 handles the first part of this task in ner’s tricks. Automatation In the second example (Listing 2b), You may have the need for a script that Listing 2b: Injecting an IRC the attacker has found a similar security compares a known system status with Proxy hole and installed several applications. the current status and responds in a pre- 01 COMMAND PID USER FD Again, the attacker has not taken the defined way in case of deviations – that TYPE DEVICE SIZE NODE NAME trouble to cover up; the www-data ac- is, an anomaly detection system. With 02 a 10555 www-data 266u GUI Tools IPv4 2808 TCP *: https (LISTEN) GUI fans may find it hard to locate a time when it was called. As this ap- graphical front-end for lsof. The Libg- proach is not event-based, glsof can 03 a 10555 www-data 267u nome-based glsof tool [3] is fairly new, easily overlook short-term access. IPv4 2809 TCP *:www and its developers are still extremely The fairly ancient Java front-end, JLsof (LISTEN) busy, although they have not made it [4], which has not been updated since 04 a 10555 www-data 543u past the alpha stage thus far. The release 2003, has less in the way of functionality IPv4 757852768 TCP cycles are fairly short, so you might like than glsof, but it also has fewer depen- *:9713 (LISTEN) to download the latest version from the dencies. To install JLsof, you need to Subversion repository. This actually download and unpack the archive. You 05 psybnc 10615 www-data 266u turned out to be the only way to get may need to modify the path to the Java IPv4 2808 TCP *: things working in our lab. The glsof interpreter and to lsof in the jlsof start https (LISTEN) homepage has the usual howtos, and script. JLsof has a far more spartan look 06 psybnc 10615 www-data 267u the developers will answer your email if than glsof, but it does show you how IPv4 2809 TCP *:www you get stuck. your filter settings resolve to lsof com- Glsof gives users the ability to set filters mand line parameters (Figure 2), which (LISTEN) by pointing and clicking (Figure 1) and to is a good thing if you are trying to under- 07 psybnc 10615 www-data 543u store the settings so that they can run stand the filter rules. Although you can’t IPv4 757871322 TCP *: the same queries later. Filters support actually store filters, JLsof will export the ircd (LISTEN) fairly complex rulesets, which you can output (Figure 3) to an XML document. 08 psybnc 10615 www-data 549u view and analyze in the query debug- If you use a Mac, there is no need to do IPv4 762054917 TCP ging window. Thus, glsof considerably without a native lsof GUI. Sloth [5], server.com:35614->oslo1. shortens the learning curve for lsof new- which was written in Objective C, scores comers. The ability to set up file moni- with a nicely organized interface (Figure no.eu.undernet.org:ircd tors in glsof, to watch freely definable re- 4) that offers predefined filters catego- (ESTABLISHED) sources, and to notify administrators in rized by resource type and the ability to 09 bind 22004 www-data 543u case of access is also useful. Under the terminate processes by clicking with the IPv4 696149859 TCP hood, all glsof does is repeatedly call mouse (kill). lsof, which lists access for the point in *:1982 (LISTEN)
32 ISSUE 77 APRIL 2007 WWW.LINUX - MAGAZINE.COM lsof COVER STORY
Listing 3: Open TCP Ports launching the program, is characteristic for the Read initial 01 $ lsof -i TCP -n -P | awk '/ web server. port assignments LISTEN/ {print $1"/"$3"/"$8}' | sort -u Do-It-Yourself 02 apache/root/*:443 IDS The miniature lsof-based 03 apache/root/*:80 IDS in Listing 3 works Read current Current port 04 apache/www-data/*:443 as depicted in Figure 5. port assignments assignment be- 05 apache/www-data/*:80 When launched, the comes new standard 06 mysqld/mysql/127.0.0.1:3306 script remembers (List- ing 4, Lines 4 through 8) 07 sshd/root/*:22 the current port configu- ration. Every 10 seconds, No port an elegant way. It tells lsof to output net- it calls lsof to fetch the Email assignment to work-related files (-i) without writing list of open ports and No changed? Yes out the port numbers as service names compares the list with administrator (-P) and without resolving IP addresses the last known status to hostnames (-n). Awk checks the out- (Line 12). If a change Figure 5: The script shown in Listing 4 alerts the administra- put for listening ports (LISTEN status) occurs, the script mails tor whenever a change occurs in the port assignments. To do and formats the output as: username/ the before/ after status so, it compares the original status with the current status processname/IP:Port, where an IP ad- (Lines 14 through 0) and every 10 seconds. dress of * stands for a server that listens uses the new status for to all interfaces. further comparisons (Line 22). certain circumstances, processes like this The final sort organizes the output in To test your do-it-yourself anomaly de- can cause false alerts, although it's fairly alphabetical order, and -u ensures that tection system, you might like to tempo- easy to avoid by modifying the logic be- each combination of user, process, and rarily open a port. Netcat offers an easy hind the query. Adding | grep -v tempo- service occurs only once. way to do so. Give a command like nc rary service in Line 5 should do the trick. The output shown in Line 2 of Listing -l -p 12345 to launch Netcat in LISTEN 3 was taken from a Debian Sarge server mode (-l) and keep port 12345 open. Conclusion with Apache 1.3, MySQL, and an SSH Within 10 seconds, the shell script in the A simple shell script can’t hope to re- daemon. In our example, MySQL only infinite loop should have noticed the sta- place a full-fledged IDS, but if you’re binds to the local interface (Line 6), tus change and responded accordingly. looking for a no-frills detection tool or while Apache and SSH are accessible via Be aware that some processes change an extra line of defense, lsof could be any interface. lsof’s view of the port assignments. For a part of the solution. Useful additions The grouping of the Apache processes example, some email servers fork addi- might be cryptography-based configura- in root and www-data, which results tional processes, depending on the sta- tion management in the style of Aide, from dropping root privileges after tus of the incoming connection. Under checks on executed files, evaluation of the UDP configuration, and many other Listing 4: Port Monitoring things. Repeated calls to lsof can also 01 #!/bin/bash 13 echo "Port assignments open up new fields of application, as evi- denced by the file monitor options in 02 MAILTO="root" changed! Notify administrator by email" Glsof. Whether you script with lsof or 03 HOSTNAME=`hostname` use it as a fast, universal administration 14 mail -s "Attention: 04 getports() { tool, lsof is an easy, if limited, tool for $HOSTNAME LISTEN status spotting intruders. ■ 05 lsof -i -n -P | awk '/LISTEN/ changed" $MAILTO <
WWW.LINUX - MAGAZINE.COM ISSUE 77 APRIL 2007 33