March 2006 www.veritest.com • [email protected] Webroot Spy Sweeper Enterprise Spyware Effectiveness Testing Test report prepared under contract from Webroot. Executive summary Webroot, Inc. commissioned VeriTest, a division of Lionbridge Technologies, Inc., to conduct a test Key findings comparing the following Enterprise class anti- spyware applications: Webroot cleaned 94% of all spyware • Webroot Spy Sweeper Enterprise 2.5.1 tested, vs. 53% for McAfee and 26% for Sunbelt • Sunbelt CounterSpy Enterprise 1.5.268 • McAfee AntiVirus Enterprise with AntiSpyware Webroot Spy Sweeper Enterprise Module V8.0 removed 97% of System Monitors tested. Webroot Spy Sweeper Enterprise The testing was designed to focus on spyware identified and removed 96% of Adware detection and removal effectiveness. tested. For the purposes of this test, spyware was intended to include all varietals including system monitors, adware and Trojans. Spyware is software with a wide variety of purposes that varies as designed by spyware creators. This software is often installed on a personal computer without knowledge of the PC user. Spyware, unbeknownst to the PC user may monitor activities on the PC and glean personal information for unscrupulous third parties. Spyware may also present undesired advertising to the PC user, or even provide a means for additional undesired software to be installed. VeriTest began with a CD-ROM containing 200 individual pieces of spyware comprising system monitors, adware and Trojans to be used in this test1. Testing was performed over a period of four months during which time, many of the spies morphed with new variations. Each Enterprise anti-spyware application was installed to its own server, each of which had three client PC’s dedicated as agents. All computers in this test were provided Internet access via a proxy server. A Snapshot was taken which included the File and Operating System configurations on each PC prior to installing spyware. After the Snapshot was taken, five individual spyware programs were installed to each client PC. The PC was then rebooted. Upon reboot, Internet Explorer was opened and a known web page was visited. The anti-spyware application was then instructed to perform an exhaustive scan with subsequent reboots and rescans if required. When the anti-spyware application indicated that there 1 The spyware programs utilized for this test were randomly chosen from a database of over 8000 spyware installation programs that was provided by Webroot. These spies consisted of a random mix of adware, system monitors and Trojans. 250 spies were randomly chosen from the database, 200 of which were used in the test. Webroot Spyware Removal Effectiveness Study were no further traces of spyware, or demonstrated no progress in removing identified spyware, an analysis of changed file and operating system configurations was performed. It is important to note that the proper analysis of spyware scanning results is critical and by its nature lends itself to misinterpretation of what constitutes a “clean” or “not clean” PC system. The single most important tool used in the analysis of the results of this spyware removal testing was the testing methodology document (Appendix A). Without having a pre-defined, concrete definition of how to interpret the scanning results it is extremely likely that the results can fall into a “gray area” where the results can be subject to individual opinion. The testing methodology used in this test goes to great lengths to eliminate this gray area of partially cleaned spies so that the results can only be interpreted as cleaned or not cleaned by whoever views them. In addition, the analysis of a PC after the cleaning process requires an intimate knowledge of Registry and File System components. A spyware program will often install shared applications or components that are common among legitimate software. In analyzing the log files produced during this test, VeriTest engineers took special care in utilizing their experience to identify Registry and File System modifications that are not unique to the spyware program. The result is that you may remove or break a legitimate application when attempting to remove the common component along with the spyware program. Therefore, these shared and benign components were not counted as spyware traces left behind by the anti-spyware application. In testing 200 individual spyware programs, Webroot Spy Sweeper Enterprise performed exceptionally well in detection and thorough removal of spyware traces. Though other anti-spyware applications were competent in their detection capabilities, their ability to completely remove all spyware traces was weak. Webroot Spy Sweeper proved superior to the other applications tested in effectively identifying and removing spyware. Webroot Spy Sweeper Enterprise went beyond removing the spyware infection by also removing the spyware installation file. This is critical to prevent re-infection. Individuals responsible for the security of their Enterprise PC infrastructure should take special care to ensure that the threat of future infection is eliminated by removal of the spyware installation file from the PC. VeriTest Enterprise Spyware Test Scoring: Scores were determined by subtracting from a total of 200 possible points, relative to the number of spyware programs tested. One point was subtracted for each spyware program noted to have not been effectively cleaned. Total Score 200 187 - Webroot Spy Sweeper Enterprise: 187 150 - Sunbelt CounterSpy Enterprise: 52 105 Webroot 100 - McAfee AntiSpyware Enterprise: 106 Sunbelt 52 McAfee 50 Webroot Spy Sweeper Enterprise proved to provide the most effective detection and removal of Spyware 0 applications in this test. Points out of 200 Test Findings Spyware Identification and Removal Effectiveness Testing Results Of the 200 spyware applications tested, Webroot Spy Sweeper Enterprise effectively cleaned 187 spyware applications. Sunbelt CounterSpy Enterprise cleaned 52 and McAfee VirusScan Enterprise with AntiSpyware Module cleaned 106. Effective cleaning of spyware applications is critical to the security of the PC in the enterprise. As demonstrated in the graph below, Webroot Spy Sweeper Enterprise demonstrated the greatest ability to detect and remove spyware. 94% 100% 80% 53% 60% Webroot Sunbelt 40% 26% McAfee 20% 0% Spyware Elimitated Spyware Cleaned by Category The graph below demonstrates detection and cleaning ability based on spyware category. For the purposes of this test, spyware was grouped into adware, system monitors and Trojans. There was a total of 122 adware, 30 system monitor and 53 Trojan programs tested. 100% 96% 97% 85% 80% 64% 60% 53% Webroot Sunbelt 40% 26% 26% McAfee 17% 20% 10% 0% Adware System Monitors Trojans Spyware Retest Sample After all spyware applications had been tested, the VeriTest test engineer selected ten random spyware programs that all three products had failed to clean in the first round for re-testing. All spyware programs to be re-tested were selected from a list of the first 75 spyware programs installed early in the testing process. Webroot and Sunbelt software demonstrated the most significant improvement in cleaning leaving only two spyware application traces remaining. These initial tests took place during 8 the second half of November and first 7 half of December 2005. Not only is a 6 vast database of known spyware 5 applications important to the Webroot 4 Administrator, but aggressive Sunbelt 3 identification of new spyware threats McAfee is equally important. In this test, both 2 Webroot and Sunbelt demonstrated 1 comparable rates of progress in 0 Cleaned Not Cleaned identifying new and morphed spyware programs. McAfee fell short of Webroot and Sunbelt in this area. CONCLUSION: Testing anti-spyware applications for effectiveness is extremely complex. Most businesses conduct rudimentary tests with common spies that produce inconsistent results. In this robust test that spanned four months and included 200 spies, with simultaneous installations of adware, system monitors and Trojans, Webroot Spy Sweeper Enterprise significantly outperformed the Sunbelt and McAfee products by accurately identifying and effectively removing more spyware. Effectively removing 94% of spyware programs demonstrates excellent early detection and cleaning methodology. On a later rescan of spyware programs noted as not effectively cleaned early on in testing, VeriTest found that eight of ten programs were cleaned. This is an excellent improvement as many spyware programs morph or evolve, making detection and removal even more difficult. Administrators must take into account the rate at which their anti-spyware vendor identifies new threats. The aforementioned testing results are evident of a “right tool for the job” scenario. Webroot has proven to provide the greatest protection against spyware at the time of this testing. APPENDIX A: Testing Methodology Each Enterprise anti-spyware product was installed to an individual Windows 2003 Standard Edition server. Each Enterprise anti-spyware product had three client PC’s dedicated as agents of that software. Each agent PC had a Windows XP Professional Operating System. All PC’s and servers were provided unrestricted Internet access via a proxy server. Anti-spyware applications were allowed to update their products via the Internet at will. On each client PC, an enterprise agent was installed along with Install Watch, Regmon, Filemon and HijackThis analysis tools. InstallWatch was used to take