SYSADMIN Nmap Scripting
SCANAutomated penetration testing with TIME Nmap’s new scripting engine
Nmap is rolling out a new scripting engine to automatically investigate vulnerabilities that turn up in a security scan. We’ll show you how to protect your network with Nmap
and NSE. BY ERIC AMBERG
map is the tool of choice for pen- functions into custom scripts. Fyodor, expressions – a useful option for pen tes- etration testing [1]. Experts use the Nmap project leader, placed the de- ters who want to check existing vulnera- NNmap to search out security velopment of this scripting engine in the bilities by launching exploits. However, holes and scan for open network services. capable hands of Diman Todorov. The the developers point out that Nmap is But what happens when you find a prob- result is the Nmap Scripting Engine not looking to compete with the lem? Many administrators prefer to follow (NSE) [2], which has been an integral Metasploit framework [3]. up the discovery with additional tests. For part of Nmap since version 4.21. NSE provides an easy means for build- instance, if Nmap finds an http service, NSE extends the core functionality of ing automated solutions around Nmap. why not query to determine the web the Nmap scanner, providing detailed in- The NSE option works well for small to server version? formation on services such as NFS, SMB, mid-sized networks. Tools such as Nes- Thus far, administrators have written or RPC. You can also use NSE to search sus [4], GFI LANguard [5], or ISS Inter- their own scripts to parse Nmap output for active systems using domain look- net Scanner [6] might be better suited files – a slow and time-consuming pro- ups, Whois searches, or other source for large-scale scanning operations. You cess. The Nmap project recently decided network discovery techniques. To can download the NSE source code from it was time to introduce a scripting en- discover backdoors, NSE checks any the project server [1] or check out your gine so that users could automate Nmap version strings it detects against regular distributor’s repository for a binary.
68 ISSUE 87 FEBRUARY 2008
068-072_nmap-script.indd 68 13.12.2007 16:54:22 Uhr Nmap Scripting SYSADMIN
NSE is built on the Lua interpreted NSE. The pre-built scripting language [7]. Lua is designed scripts attend to to work with software written in other common tasks Scanned Host languages, such as C or C++. Lua, often associated which is based on ANSI C, owes its flexi- with Nmap. For bility to the fact that most functions are example, if the available in the form of libraries. This scanner discovers approach makes it easy to extend the that TCP port 80 is functional scope by binding required open and if the Nmap modules. In addition, the Lua interpreter pre-built showHT- with Nmap Scripting Engine has a very small footprint; even commer- MLTitle.nse script cial games (including World of Warcraft) is available, a call rely on Lua. to the script is is- A Lua interpreter forms the core of sued. The script NSE (see Figure 1). When Nmap detects queries the web NSE Script Nmap API a host or port, it calls the Lua interpreter server’s front page with a matching script to leverage the and displays the abilities of the Lua language and Nmap- header (see Figure specific functions from the NSE library. 2). Several pre- The NSE library provides additional built scripts are Nmap-related features that Lua doesn’t available in /usr/ Lua Extensions offer such as tools for evaluating and share/nmap/ manipulating IP addresses, using Perl- scripts. compatible regular expressions, or ma- Figure 1: A combination of NSE scripts, Nmap libraries, and new Lua nipulating URLs. Categories calls extends the functional scope of the classic Nmap tool. Lua makes intensive use of data struc- NSE organize tures in tables. Tables contain attribute- scripts in categories to allow for more search fails, Nmap then looks for a direc- value pairs, and they can also contain granular control. The defined script cate- tory with the same name and binds any subtables. For example, NSE uses the host gories are safe and intrusive, malware, scripts ending in .nse. and port tables to access Nmap scan re- version, discovery, and vulnerability. If this search also fails, Nmap searches sults. A registry table that all scripts can Scripts that NSE categorizes as safe are for an individual script of the same read and write to resides above all scripts. very unlikely to cause problems on the name. For example, the command-line This design lets scripts exchange data. scan target. The scripts in the intrusive option --script discovery,malware binds category are unlikely to cause damage, the categories discovery and malware. A NSE in Action but will attempt to use default pass- database file titled script.db in the scripts Nmap and NSE use the Nmap API to ex- words to access systems, and thus they subdirectory maps scripts to individual change information, including the target are more likely to generate log entries categories. Admins can update the file host name, the operating system and IP on the scan target. by entering --script-updatedb after address, the port number, and the port Scripts in the malware and vulnerabil- adding scripts. If you set the -sC flag, status. The API also lets users call ity categories test the scan targets for only scripts from the categories safe and Nmap’s socket functions for network malware and known vulnerabilities. The intrusive are executed. communications. An extension that lets version category plays a special role: users send custom packets is due for re- scripts in this category, which are en- Your Own NSE Scripts lease in the near future. abled by the -sV command-line option, Anexample will make it easier to under- Nmap starts by checking whether it extend Nmap’s version detection capa- stand the structure of an NSE script. can reach ports on a host. In case of a bilities. Their output is no different from Most built-in scripts are kept simple – TCP scan, the tool will ascertain the port standard Nmap output; the script is not normally smaller than 100 lines includ- status, which could be open, closed, or referenced in the report generated by the ing comments. The ripeQuery.nse script filtered. Once Nmap has detected a port, scan. The discovery category includes in Listing 1 should give you some idea and assuming NSE has been enabled scripts that are designed to find out more of how Lua and NSE work. The script with the -sC option, the subsystem will about the target host or network by que- queries an address for an entry in the attempt to locate a matching script for rying various services (including SNMP RIPE network registration system [8]. the test. or LDAP). An NSE script consists of three compo- Scripting rules determine whether The --script argument specifies the nents: the header defines the script Nmap will run an NSE script. These categories, directories, and individual name in the Nmap output, the script scripting rules specify the conditions scripts for NSE to integrate and execute. category, and the run level. The rule under which the script will launch. A comma-separated list of values is pos- specifies the conditions under which the You can build your own scripts or rely sible. Nmap starts by searching for a script is executed. The action component on the pre-built scripts included with category with the specified name; if the calls the functions that handle the actual
FEBRUARY 2008 ISSUE 87 69
068-072_nmap-script.indd 69 13.12.2007 16:54:58 Uhr SYSADMIN Nmap Scripting
socket:connectU ("whois.ripe.net", 43)
tells the script to connect to the RIPE da- tabase, which responds to TCP port 43 requests. The Whois service expects an IP address in quatted dot notation. The code socket:send(host.ip .. "\n") sends the IP address. The query line must be terminated by a newline character. To allow this to happen, the concatenator operator .. adds \n to the IP address for the scanned host parsed from host.ip. In- side an infinite loop beginning in line 23,
local status, lines = U Figure 2: In new script mode (option -sC), Nmap calls external scripts. Nmap has detected socket:receive_lines(1) that port 80 is open, so the tool chips in to query the heading of the website it has discovered and displays the results. The figure also shows NSE detecting the NetBIOS names of a target. reads the response from the RIPE data- base until the data input dries up. The tasks. Authors can implement all three (host, port) function. The function again Boolean variable status shows whether sections of the script in Lua. expects host and port as arguments. the connection is still delivering data. Each NSE script has a header that In line 16, the script first creates a new If this is not so, break in line 27 quits comprises four descriptive details. The TCP/ IP socket then defines the local the loop. The parsed response is avail- id contains the script name and is used variables status, line, and result. Line 20, able in lines, and result accumulates the for Nmap output; description is a short description; and author names the au- Listing 1: ripeQuery.nse thor. The interpreter does not process 01 require “ipOps” 23 while true do the license field. 02 24 local status, lines = socket: A script will only run under specific 03 id = “RIPE query” receive_lines(1) conditions. Port and host rules control 25 its behavior. The example in Listing 1 04 description = “Connects to the defines a function that expects the Lua RIPE database, 26 if not status then host and port tables in lines 11 through 05 extracts and prints the role- 27 break 13. The tables contain the values discov- entry for the IP.” 28 else 06 author = “Diman Torodov ered by Nmap in the course of a scan. 29 result = result .. lines For example, host.ip supplies IP ad-
70 ISSUE 87 FEBRUARY 2008
068-072_nmap-script.indd 70 13.12.2007 16:55:13 Uhr Nmap Scripting SYSADMIN
input. After all response lines are read, Table 1: Host and Port Table Attributes socket:close() closes the socket. Attribute Type Meaning The next few lines are designed to dis- Host Table Attributes cover what the entry for the submitted host.os String Contains a string with the detected operating system name IP address says. The Whois server re- if Nmap is launched with the -O option turns a large number of response lines; host.ip String Contains the IP address for the current scan target host our example simply searches for lines host.name String Contains the host names returned by a reverse lookup that start with role:. To do this, the script host.mac_addr String Contains the MAC address for the target host calls the Lua string.match() function. Port Table Attributes The line port.number Integer The port currently being scanned port.protocol String Either tcp or udp local value = string.matchU (result, "role:(.-)\n") port.service String Contains a service name if Nmap was able to map a name in the scope of version detection port.state String The port status can be open or open|filtered; the script is not searches in the result variable for a line run against closed ports and filtered ports that matches the search key. The key starts with role:; however, the function only returns the remainder of the line in tables provide access to critical values The sample shows how to call these parentheses. To keep this example sim- such as the IP address, port, service, or values and integrate them with an NSE ple, I edited a script that is provided with port status. The Lua host table returns script. To communicate with other sys- the Nmap distribution. To be more ro- the values shown in Table 1, assuming tems, NSE scripts need an interface. An bust, scripts should always perform a that Nmap has collected some values; API makes it possible to use the Nsock plausibility check on returned values. otherwise, it returns an empty string. library (also used internally by Nmap). This also applies to details of active Nmap API Treasures ports, which are stored in the Lua port Socket Programming The Nmap API lets NSE scripts commu- table, along with attributes for the proto- Legacy communications typically follow nicate with Nmap. The Lua host and port col, service, and status. the socket approach. The following code the mathematics of humour
TWELVE Quirky Humans, Over Two Million Geeks around the world can’t be wrong! TWO Lovecraftian Horrors, COME JOIN THE INSANITY! ONE Acerbic A.I., ONE Fluffy Ball of Innocence and TEN Years of Archives EQUALS ONE Daily Cartoon that Covers the Geek Gestalt from zero to infinity!
FEBRUARY 2008 ISSUE 87 71
068-072_nmap-script.indd 71 13.12.2007 16:55:14 Uhr SYSADMIN Nmap Scripting
creates a new socket dubbed sock and Table 2: Lua Extensions for NSE then binds the socket to the host using Extension Function the specified port. bit Supports bit operations; for example, bit.lshift(a, b) performs a bitwise left The protocol is optional and can be shift by b digits for a. tcp, udp, or ssl: pcre Refers to Perl-Compatible Regular Expressions and supports their use. This lets scripts extract search keys from service responses. sock = nmap.new_socket() ipOps Supports operations related to IP addresses. As shown in the sample script, sock:connect(Host, Port U ipOps.isPrivate(address) checks to see whether the specified address is a [, Protocol]) private address in the sense of RFC 1918. shortport Provides standard tests for port rules. Many scripts use this option and run The sock:send(request) code can then shortport.portnumber(port) to check the port. send a prepared request to the target listop Lists processing as used by other programming languages such as Lisp or host. The semantics of the line, Haskell. The developers refer to this extension as experimental. strbuf Supports certain string operations; for example, strbuf.eqbuf(string1, string2) status, value = U compares two strings. sock:receive_lines(lines) url Extends Lua’s URL manipulation abilities, adding functions that create or analyze parameter lists. which reads data from the socket, is fairly unintuitive. The script starts by calling a function design custom Nmap scans. Modules If the requested lines, or a lesser num- whose only purpose in life is to close a from the NSE extensions library are per- ber of newline-terminated lines, arrive socket in case of error. The exception fect for extending the basic functionality before the timeout, they are stored in the handler is created by the line: of the underlying Lua script language. variable value. However, if the network The Nmap API gives programmers the buffer has more lines than anticipated, try = nmap.new_try() ability to reference the host and port ta- the API will send more lines, as de- bles and thus parse scan results. The API scribed in the documentation [2]. The preceding line expects an err_ also adds numerous communications If successful, status contains a value catch() function argument, which closes options for scripts. of true. Finally, sock:close() closes the the socket. If a function returns an error, socket. The Nmap API has far more the interpreter cancels the function with- Conclusions functionality, but most typical cases out a comment. The function passed in NSE scripts are quickly conquering the are covered by the examples provided. as an argument specifies the actions to former domains of security tools such perform in this case. as Nessus. A stable version of NSE is not Exception Handling NSE provides a powerful and flexible available as of this writing. The Nmap API also implements an ex- extension that administrators can use to Nmap 4.21 contained the alpha ver- ception handling mechanism for catch- sions and 4.22 was part of Google’s ing errors and canceling scripts that Listing 2: finger.nse Summer of Code. would provide erroneous results because (Excerpt) The stable Nmap 4.5 version will in- of incorrect conditions. clude NSE, and it is scheduled for com- 01 local err_catch = function() Exceptions are important for robust pletion by the time this issue of Linux 02 socket:close() scripts, in that services that you scan Magazine hits the shelves. NSE defi- across a network will not always re- 03 end nitely has the potential to become an spond as expected. A glance at the fin- 04 indispensable part of Nmap. I ger.nse script shown in Listing 2 illus- 05 local try = nmap.new_try(err_ trates how exception handling works. catch()) INFO If an error occurs on the target system 06 [1] Nmap: or the connection times out, the excep- 07 socket:set_timeout(5000) http:// www. insecure. org/ nmap/ tion handler closes the socket gracefully. 08 try(socket:connect(host.ip, [2] NSE documentation: http:// www. insecure. org/ nmap/ nse/ 09 port.number, Lua Extensions [3] Metasploit Project website: 10 port.protocol)) http:// www. metasploit. org/ Whereas the Nmap API handles com- 11 try(socket:send(“\n\r”)) munications between Nmap and the [4] Nessus project: NSE scripts, the Lua extensions add sig- 12 http:// www. nessus. org nificant functionality to NSE. The mod- 13 status, results = socket: [5] GFI LANguard: ules are located below the nselib subdi- receive_lines(100) http:// www. gfi. com/ languard [6] ISS Internet Scanner: rectory. An NSE script binds the mod- 14 socket:close() ules via require Modulename. Table 2 lists http:// www. iss. net/ products/ Internet_ 15 if not(status) then the available extensions. It is not impor- Scanner/ product_main_page. html tant whether you bind modules before 16 return results [7] Lua: http:// www. lua. org/ or after the header. 17 end [8] RIPE: http:// www. ripe. net/
72 ISSUE 87 FEBRUARY 2008
068-072_nmap-script.indd 72 13.12.2007 16:55:15 Uhr