Counting Spyware Detections Perception and Reality
Total Page：16
File Type：pdf, Size：1020Kb
White Paper March 2005 AVERT™ Counting Spyware Detections Perception and Reality www.mcafee.com Counting Spyware Detections Perception and Reality 2 Table of Contents Table of Contents 2 The myth of signature counts 3 Shenanigans! 3 When is a threat counted as unique 3 But wait, there’s more! 5 Clouding the picture 6 So how should products be compared? 6 Next Steps 7 Industry Cooperation 7 Improved Independent Testing 7 Conclusion 7 About AVERT 8 www.mcafee.com Counting Spyware Detections Perception and Reality 3 The myth of signature counts Now imagine a group of anti-virus vendors writing code to detect all of these (detection code executed in order Shenanigans! shown): In the early days of the anti-virus industry, it was common for vendors to brag about their signature counts and Vendor X: withhold their samples from other anti-virus vendors to - Detect A*CD*F as Virus 1 prevent them from catching up. Eventually, everyone realized that this was causing misleading claims and was Vendor Y: performing a disservice to the marketplace. This practice in many ways led to the rise of the independent testing - Detect AACD*F as Virus 1 bodies, such as ICSA Labs, VTC, AV-Test.org and so on. - Detect A*CDEF as Virus 2 Unfortunately, the anti-spyware industry as a whole has not reached this level of maturity. Many vendors count Vendor Z: each executable, data file, registry key, etc. belonging to a - Detect ABCD** as Virus 2 package as a new signature. There is also wide variation in the kinds of threats detected. Some detect backdoors - Detect ACCD** as Virus 3 Trojans and worms, while others do not. Others have numerous detections for cookies, which have no security - Detect AACD** as Virus 4 impact and debatable privacy implications. What you end up with is: When is a threat counted as unique Table 1: Hypothetical anti-virus naming concordance The Anti-virus perspective File Vendor Vendor Vendor The anti-virus industry has long taken heat for confusion X: Y: Z: over virus naming, and there is wide disparity not only in Sig Count Sig Count Sig Count how many detections a product claims to be capable of, 1 2 3 but in how threats are named and counted. This ABCDEF Virus 1 Virus 2 Virus 2 discrepancy occurs in the anti-virus world for a variety of ACCDEF Virus 1 Virus 2 Virus 3 reasons: AACDEF Virus 1 Virus 1 Virus 4 • Historical conventions AACDFF Virus 1 Virus 1 Virus 4 • Lack of time to synchronize names during outbreak events So in this grossly-simplified case, three vendors with a 200% variation in signature count all detect all 4 samples, • Simple stubbornness but with different names. • Differences in generic and heuristic detection The VGrep tool, maintained by McAfee for Virus Bulletin technology (http://www.virusbtn.com/resources/vgrep/index.xml) To understand the last point, imagine the following 4 files attempts to help alleviate some of the confusion by cross- consisting only of 6 letter sequences were all viruses: referencing vendor’s names across a large body of samples. 1. ABCDEF For a more realistic example, in a recent test at AV- 2. ACCDEF Comparatives (http://www.av- comparatives.org/seiten/ergebnisse_2004_08.php), 3. AACDEF products tested claim anywhere from just shy of 53,000 signatures (Dialogue Science) to nearly 123,000 4. AACDFF www.mcafee.com Counting Spyware Detections Perception and Reality 4 signatures (Frisk Software). But at the end of the day, Spycop SpyCop 467 almost all of the products tested detected over 300,000 of Webroot Spyware Sweeper 31104 323,000 unique samples. So a large difference in number Javacool SpywareBlaster 3183 of signatures may relate to only small differences in actual PC Tools.com Ltd Spyware Doctor 10684 detection capabilities. Giant Company GIANT Antispyware > 100,000 Software‡ Now this level of confusion occurs in an industry that: McAfee VirusScan 3175* Enterprise • Has been around for nearly 20 years in some McAfee Antispyware** 384 form or another * counts only Potentially Unwanted Program detections • Has consistent definitions for what is and is not a ** McAfee consumer antispyware product virus or a Trojan (the definition for virus can be ‡ Now Microsoft Antispyware expressed in mathematical terms it is so precise, This shows a difference of 3 orders of magnitude between Adleman, 1988, “An Abstract Theory of Computer the highest and the lowest signature counts in the different Viruses”) products. Now consider the performance of those • Has a variety of professional bodies (AVPD, products when they scanned McAfee’s APP collection CARO, AVED, AVAR, EICAR) to promote (detections that are neither viral nor Trojan): cooperation and consistency Table 3: Spyware product detection over McAfee • Has routine collection-trading occurring between APPS collection a vast majority of vendors Vendor Number of Time Taken Detections • Has a number of well-respected testing bodies McAfee VirusScan 11288 0:19:50 (e.g. ICSA, VTC, VB, AV-Test.org, AV- Enterprise Comparatives.org) with high-quality collections Spyware Doctor 135 0:00:24 SpySweeper 951 0:02:54 In fact, the entire basis for rating the detection capabilities Adwaresafe 151 0:00:57 of anti-virus vendors rests on the fact that there are people Adaware 356 0:03:19 and organizations with canonical, or at least, very complete, collections. In this way, we can make apples- PestPatrol 2601 0:25:00 to-apples comparisons between anti-virus products in a McAfee Antispyware* 270 0:03:53 way that is unambiguous despite all of the inherent Aluria Spyware Eliminator 358 0:12:07 inconsistency in counting and naming. Giant Antispyware 617 0:22:19 The Spyware perspective SpyBot 0 0:03:08 Now let’s switch to the spyware industry, where a recent * McAfee Consumer antispyware product (September, 2004) internal McAfee survey showed the following “raw” signature counts: PLEASE NOTE: Table 2: Spyware Product signature count • This is a terrible test and should not be Competitor Product # detections used for real comparisons or competitive Aluria Spyware Eliminator 18625 evaluation. Lavasoft Ad-Aware 9637 • McAfee Anti-Spyware Enterprise Computer PestPatrol 118060 (corporate product) was not available for Associates inclusion at the time of the tests. Safer Networking Spybot S&D 17679 Ltd www.mcafee.com Counting Spyware Detections Perception and Reality 5 The only thing the above results prove is that it is VERY vendors over the correct family name, they are in many easy to run a meaningless test. Why is it a terrible test? ways, less complex than PUPs. • It includes samples collected by only one vendor Many PUPs are complete software packages. They (McAfee) and is HEAVILY biased in VirusScan include installers, uninstallers, read-me’s, EULA’s, data Enterprise’s favor, as this is the collection used to files, support DLLs, shortcuts and other paraphernalia verify that we detect what we are supposed to in common to Windows applications. In our early experience terms of Potentially Unwanted Programs (PUPs) with the spyware collected by the McAfee Anti-spyware before every DAT release. (MAS) consumer team, we’ve seen the following: • It is heavily biased AGAINST products that rely • There are something like 14,000 files (only about on more than a file to trigger detection. For 5,000 are left after removing data files like txt, example, SpyBot detects PUPs ONLY when the jpg, registry, etc. files) contained in the MAS properly installed package (including registry collection, belonging to the only 400 or so unique entries, et. al) is present on a system. A dumb detections that the product contains. So whereas file collection will not trigger anything. SpyBot the DATs have 3,000 signatures to detect 11,000 actually is a decent anti-spyware product against files (about 3 files per signature), MAS only live PUPs, which cannot be determined from this needs 400 to cover about 14,000 (about 35 per test. signature). • It does not include detection for cookies and • A single MAS detection will often contain files registry entries, which may make up large detected under 5-10 completely different names portions of the signature databases for some in the DATs. products. • There is much more code reuse in PUPs, such • It does not include backdoor Trojans and other that the same exact binary might exist in 15 or “traditional” malware, which often find their way more individual PUP packages. Even worse, the into anti-spyware products. For example, nearly same binaries might exist in packages that do 70% of the files listed in Pest Patrol’s Pest NOT have PUP characteristics at all, and which Encyclopedia we do not want to detect in this context. (http://research.pestpatrol.com/search/browse.as px) of 25,000+ pests are in categories that In a nutshell, there is absolutely no correspondence AVERT generally treats as Trojan horses. In between the number of signatures in a database, and other words, nearly 3/4 of their signature base the effectiveness of that product against any given set may include items already detected by McAfee of threats. VirusScan Enterprise. There is very little consistency about how detections are • It does not include a collection of over 200,000 counted amongst anti-spyware vendors, and no one is in a unique dialer programs that are detected by less position to call them on it, because no one has a than 100 signatures in the McAfee DAT files. comprehensive collection. AVERT estimates that there are approximately 7,000 -10,000 real unique PUPs to But wait, there’s more! detect, counting them more or less the way we do in the But these aren’t the only challenges in trying to make anti-virus space.