Rootkit Detection: Finding the Enemy Within

TECH TRACKER Rootkit Detection: Finding the Enemy Within

Tools to spot rootkits are evolving, mon approach is to use a standalone rootkit-detection but rootkit authors have the upper hand tool to probe the infected host. on compromised machines At the same time, rootkits continue to evolve. For BY ANDREW CONRY-MURRAY

NEW PRODUCTS ARE EMERGING to make it easier for >(THE LOWDO WN_ security professionals to unearth rootkits on compromised machines, but identifying those THE PROMISE / Security software vendors are adding >machines and removing the malignant software remains new capabilities to prevent the installation of rootkits. frustratingly difficult. Attackers still have the upper hand Existing standalone rootkit detection products that rely if a machine gets compromised. Malicious software on cross-view differential detection also are being incorporates full rootkits or rootkit-like capabilities to incorporated into security suites, promising to bolster entrench itself on compromised PCs and evade detection. signature and heuristic analysis and provide in-depth diagnosis of potentially compromised computers. The use of stealth techniques by malware has increased 600 percent since 2004, according to McAfee, and the THE PLAYERS / Conventional antivirus players, use of custom rootkits, which are difficult if not impos- including McAfee and Symantec, are enhancing rootkit sible to detect with signatures, is also on the rise. detection. F-Secure offers a standalone rootkit detection product, BlackLight, and will bundle it into its enterprise The security community has responded to these security suite late this year. HIPS vendors, such as CA, developments with standalone rootkit-detection tools Cisco Systems, eEye, ISS and Sana Security, as well as that attempt to find rootkits by examining low-level antispyware vendors, such as Aluria, Tenebril and Web- data, such as the raw file system. Some vendors also are Root, also offer detection and prevention mechanisms. adding enhanced rootkit-detection capabilities to their THE PROSPECTS / Full-blown rootkits make up a security software suites. Anti-rootkit tools generally do tiny percentage of malware, but spyware and Trojans use one of two things: detect and block rootkits before they rootkit techniques to thwart detection and removal, compromise a PC, or attempt to find and remove them which means enterprises need a comprehensive solution after they’ve burrowed into the OS. that emphasizes prevention. Suites from major vendors Toward the goal of prevention, security vendors rec- may be good enough for general user populations, but IT ommend a cocktail of techniques that includes signatures, should consider an antivirus-HIPS combination for high-value computers. IT also should add standalone heuristics, behavioral analysis and generic exploit block- rootkit detection software to its diagnostic toolbox. ing. If a machine has been compromised, the most com-

TIMELINE RUN SILENT, RUN DEEP

> {1986} {1990s} {2001} {2002} {2005} {2006} Brain, the first PC Unix exploits The first Windows HackerDefender, Sony rootkit U. of Michigan virus, uses stealth emerge and the rootkit, NTRootkit, an open-source uncovered; Apropos and Microsoft techniques to hide term “rootkit” is appears; Lion worm Windows rootkit, spyware includes researchers from detection coined includes rootkit in provides cut-and- kernel-mode rootkit announce SubVirt, its payload paste capability a proof-of-concept virtual machine rootkit

Copyright (c) 2006, CMP Media LLC. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribution. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 516.562.7026 or visit www.cmpreprints.com/faxback.jhtml TECH TRACKER machine rootkit called SubVirt. SubVirt installs a virtual machine monitor under the Windows OS of a compromised machine, letting it boot the OS into a virtual envi- instance, rootkit authors are designing their programs so ronment and operate undetected by security software that they don’t modify system information, thereby running inside the virtual environment. thwarting some rootkit-detection tools. Rootkit authors also are exploring new stealth techniques, such as hiding THE SEARCHERS files using ADS (Alternate Data Streams), an NTFS capabil- A variety of security products can prevent a rootkit ity designed to facilitate compatibility. from gaining a foothold on a computer, including antivirus, anti-spyware and HIPS (host intrusion prevention TRACING THE ROOTS system) products. Standard signature detection from Rootkits go back to the Unix OS, in which the “root” antivirus and anti-spyware software still plays a key account provides administrator-level access to all func- role in prevention. The great majority of malware uses tions and facilities. The goal of a rootkit is to hide the binaries or code snippets of known rootkits, which presence of an attacker and malicious tools. Rootkits means signatures and heuristics can spot variants of exist for Unix and its variants, but most rootkits—and known rootkits before they hit the hard disk. anti-rootkit software—focus on the Windows OS HIPS software also can provide a measure of detec- because of its ubiquity. tion. Rootkits are often bundled into the payload of an Windows rootkits can be divided into user-mode exploit, but if the HIPS stops the execution, the rootkit and kernel-mode. User-mode rootkits run as an individ- won’t be installed. For more on HIPS, see “Probing Ques- ual application or may modify an existing program. tions” at nwc.com/channels/security/showArticle.jhtml? arti- Kernel-mode rootkits run in the kernel of the OS, and cleID=193005679. are often loaded as a device driver. Both types hide by McAfee VirusScan 8.5, which is due to ship this month, intercepting and changing system-status information will include a kernel-based scanner that can scan kernel- requested by an application. or user-mode memory for known rootkits. The first Windows rootkit, NTRootkit, emerged in Microsoft also has included a security feature called 2001 as a proof-of-concept by security researcher Greg Kernel Patch Protection, or Patch Guard, in the 64-bit ver- Hoglund. Since then more potent versions that target the sions of its Windows OS. Patch Guard monitors the kernel Windows OS have emerged, including publicly available and detects attempts by other code to intercept and mod- rootkits, such as FU and HackerDefender. And a growing ify kernel code. Microsoft says this feature is designed to number of criminals are buying prepackaged exploit tools. help protect the OS from malware and from legitimate Researchers at the University of Michigan and software that may destabilize the OS. At press time, Microsoft recently described a proof-of-concept virtual Microsoft was meeting with third-party security software vendors about APIs to allow security software to TAINTED EVIDENCE work around Patch Guard.

TAINTED VIEW UPROOTING THE PROBLEM process: pid 2342, program.exe process: pid 2741, system.exe Security vendors are developing methods to uncover process: pid 344, suspicious.exe rootkits on compromised machines. Many standalone file: c:\program files\myprogram\program.exe tools use a technique called cross-view differential detec- TRUSTED VIEW tion. This technique relies on the fact that a rootkit process: pid 154, malware.exe process: pid 2342, program.exe manipulates registries, APIs and system calls. process: pid 2741, system.exe Cross-view detection mechanisms scan system compo- process: pid 344, suspicious.exe file: c:\program files\myprogram\program.exe nents, including files, registry keys and processes, using file: c:\windows\malware.exe the APIs on machines suspected of being rooted. This pro-

WHAT IS HIDDEN? duces a “tainted view” of the system. It then runs a second process: pid 154 malware.exe scan of the computer—the trusted view—without exercis- process: pid 2342, program.exe process: pid 2741, system.exe ing the APIs by examining lower-level data structures, process: pid 344, suspicious.exe such as the raw contents of a file system or the registry file: c:\program files\myprogram\program.exe file: c:\windows\malware.exe hive that aren’t manipulated by the rootkit. It then compares the two scans to identify instances where system information may have been manipulated. Cross-view dierential detection compares tainted and trusted views of a system. If the tainted view has fewer items, such as files Standalone cross-view tools include F-Secure’s Black- or executables, than the trusted view, that’s strong evidence of a light and SysInternals’ Rootkit Revealer. F-Secure includes rootkit or other stealthing activity. Blacklight in its consumer security suite, and plans to

Copyright (c) 2006, CMP Media LLC. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction 24 or public distribution. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 516.562.7026 or visit www.cmpreprints.com/faxback.jhtml TECH TRACKER tools into suites with a management console, this will be less of a problem. Meantime, there are signs that indicate a compro- incorporate it into the next version of its enterprise secu- mise and, if encountered, standalone tools are warrant- rity suite, F-Secure Anti-Virus Client Security. Symantec ed. First, if you find a machine that has been infected also has created a new tool, VxMS, that uses a method sim- with spyware or adware, you should also run a rootkit ilar to the cross-view differential technique. The VxMS scan. Second, some rootkits can cause PCs to freeze up. technology will be included in forthcoming enterprise edi- If you’ve got machines on your hands falling prey to tions of Symantec Client Security and Symantec AntiVirus. Blue Screens of Death for no apparent reason, a rootkit Note that these tools look for generic rootkit activity, scan should be included in your diagnostic analysis. not rootkit signatures. That means an experienced IT Other indications include the typical behavior of a administrator must examine the results to determine if machine infected with malware, such as high volumes the files represent a threat. of e-mail or Web traffic and back-channel communications using unusual ports or protocols. WARNING SIGNS If you can, also track how the machine got infected. Rootkit detection also is complicated by the number Too often it can be tied back to employee behavior. That of desktops under IT administration. It’s simply not must be corrected to prevent future infections. I conceivable to run a standalone tool on every PC— you’d have to touch each PC individually. As security ANDREW CONRY-MURRAY IS NETWORK COMPUTING’S BUSINESS EDITOR. WRITE TO HIM AT [email protected]. POST A COM- vendors integrate cross-view differential detection MENT OR QUESTION ON THIS STORY AT NWC.COM/GO/ASK.HTML. SIP Trunks Find a Niche

Small companies can save big, but implementation can be tricky BY MATT VLASACH

WHEN OUR COMPANY first considered SIP trunking long-distance costs an average of 50 percent less than as a voice connection to the PSTN, we were conventional long-distance service. advised against it by various VAR representatives, >many of whom asked, “Do you want to spend your time running your business or fixing your phones?” >(THE LOWDO WN_ Despite the warnings, we pursued SIP (Session Initi- ation Protocol) trunking in the hope it could offer ben- THE PROMISE / Although not the first VoIP trunk- efits and cost savings to a small company like ours. ing technology available to small-to-midsize companies, Having used Vonage service to run my business out of SIP trunking may be the first to have a widespread my college apartment, I believed SIP-based VoIP service impact on phone communications for these enterprises. offered tremendous value, scalability and flexibility By enabling businesses to place calls over the Internet that could be useful for our expanding business. using any number of SIP-enabled carriers, these enter- There are indeed benefits to be had for small com- prises are no longer limited to their local telco, thereby panies, such as fixed cost and easy scalability. But the increasing market competition and driving down costs. obstacles to SIP trunking can be formidable. The SIP THE PLAYERS / Voice service providers and equip- standard hasn’t been around long enough to be univer- ment manufacturers are striving to roll out products and sally accepted. And even if a hardware company or VSP services that more thoroughly conform to SIP. Asterisk, (voice service provider) claims it supports SIP, out-of- Avaya, Cisco Systems and Nortel Networks are active- the-box performance isn’t guaranteed. Working ly improving hardware or software solutions to be universally SIP trunk-compliant, and VSPs ranging from through implementation problems can be time-con- small companies such as Bandwidth.com to telco giants suming, so it’s essential to educate yourself about the like Verizon are providing corresponding SIP services. potential pitfalls. THE PROSPECTS / Having already taken hold in A GOOD FIT the less technically demanding residential market, SIP trunks have proven their ability to perform to customer SIP trunking, an interoperable SIP-based VoIP connec- needs. At the current rate manufacturers and service tion established between a carrier’s voice equipment providers are improving SIP compliance and QoS levels, and a customer’s IP PBX, held appeal for our company this technology will establish a major small-business because of its price, specifically the fixed communica- presence in the next few years. tion costs available through unlimited voice plans. SIP

Copyright (c) 2006, CMP Media LLC. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction 26 or public distribution. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 516.562.7026 or visit www.cmpreprints.com/faxback.jhtml TECH TRACKER claimed they would support our Cisco Call Manager Express IP PBX, but upon establishing service, many of the advanced call features failed to work. Among And once a SIP trunking system is set up, adding the five SIP VSPs we tried, only one was able to prop- lines is as simple as asking the VSP for more simultane- erly accommodate our IP PBX. ous lines to be enabled. If your Internet connection has There is one major shortcoming with Cisco’s SIP enough bandwidth, no additional hardware or software trunking implementation: As of this writing, it can modifications are necessary. handle SIP transfers and forwards only through a Finally, adding a branch office to the SIP trunk is proxy—in this case, a VSP. All transfers and forwards easy. All you need is an IP PBX for the new office, along using Cisco’s hardware require a customized setup with a broadband Internet connection. Both main and for each customer. branch offices can use the same VSP. And you can “peer” We gather that Cisco implemented call functionality branch offices together using SIP, making interoffice this way with the assumption that two Cisco CMEs communication completely free. would be connected using SIP, with an administrator in Other benefits of SIP trunking include presence control of both boxes. In other words, though Cisco awareness between companies; the potential for supports SIP trunking by definition, it does not yet pro- improved voice quality; and integrated voice, video and vide complete support for SIP as a connection to the IM between offices. PSTN. The company says it plans to support this SIP trunking functionality in CME in early 2007. THE SIP STANDARD DILEMMA Another SIP trunking initiative under way is the SIP Conventional telco service delivery standards are well- Forum’s SIPconnect. Endorsed by companies such as established. For POTS lines, the protocol is very simple, Avaya, Cbeyond and Cisco Systems, SIPconnect aims to and compliance is not difficult. Higher-capacity solu- tie enterprises to VSPs with an end-to-end IP connection. tions such as T1 PRIs, though technically more complicated, have been around long enough that the commu- THE RIGHT CONNECTION nication protocols are observed universally. One of the major caveats of any SIP trunking solution Not so for SIP trunks. SIP as a protocol works great, is that voice packets must arrive in time and in order. but proper functionality is contingent on compatible Losing only a few packets quickly garbles a conversa- implementation on both client and provider sides. To tion. This usually occurs due to overuse of available address potential problems, many SIP VSPs provide cus- bandwidth or too many hops between the IP PBX and tomers with preapproved equipment or a list of com- the VSP. patible equipment to select from. One way to address this is to use a VSP that also Because we didn’t fully understand the serves as your ISP. Such providers can set up appropri- compatibility concerns, we took the reverse ate QOS metrics to ensure that important sales calls approach: We purchased hardware based on func- take precedence over a co-worker’s download of the lat- tional requirements, then looked for a compatible est “The Office” episode. One pitfall of using the same VSP. Learn from our mistake: Many service providers provider for voice and Internet access is that you are restricted to the provider’s voice and data service offer- FULL VOIP SIP DEPLOYMENT ings, and pricing tends to be more expensive for a higher quality of service. We opted for a VSP independent of our ISP, as we SIP-enabled PSTN found our ISP’s voice service unsatisfactory. We so! switch obtained a dedicated 768-Kbps fractional T1 line that connected us to a Level 3 Communications Tier 2 provider. After some searching, we found Bandwidth. Internet com, which provides SIP trunking service through Level IP PBX VoIP phone 3 Tier 1 soft switches. After our voice data leaves our network SIP over ISP’s network, it never has to leave Level 3’s backbone broadband Internet connection before connecting to the PSTN—resulting in superb call quality, with very few QoS problems in our three months of experience. I

A full SIP deployment operates independently of the conventional MATT VLASACH IS CEO OF PACIFIC SWELL NETWORKS. telco infrastructure. All voice signaling and media use the Internet as WRITE TO HIM AT [email protected]. the only connection between the voice provider and the customer. POST A COMMENT OR QUESTION ON THIS STORY AT NWC.COM/GO/ASK.HTML.

Copyright (c) 2006, CMP Media LLC. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction 28 or public distribution. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 516.562.7026 or visit www.cmpreprints.com/faxback.jhtml