Testing Endpoint Security: An Introductory Guide A Sophos technical paper January 2018 Testing Endpoint Security: An Introductory Guide Contents Preface 3 Exploits 14 Introduction 4 Definition 14 Why Conduct Endpoint Security Testing 4 Why Exploits Matter 14 Attack Chain and Modern Attacks 4 How to Test Exploits 15 Disclaimer 6 Active Attacks 17 Prerequisites 6 Definition 17 Testing Overview 6 Why Active Attacks Matter 17 Testing Checklist 7 How to Test Active Attacks 18 Portable Executables (Malware) 7 False Positives 18 Definition 7 Definition 18 Packed/Compressed Files 7 Why False Positives Matter 18 Potentially Unwanted Applications 8 How to Test False Positives 19 Why Portable Executables Matter 8 How to Test Portable Executables 9 Additional Resources 20 Packed Files 9 Testing Lab Environment 20 Potentially Unwanted Applications 10 Overview 20 Ransomware 10 Virtual Testing Lab Architecture 21 Definition 10 Where to Get Samples 22 File Encryptors 10 Conclusion 24 Disk Encryptors/Wipers 10 Why Ransomware Matters 11 How to Test Ransomware 11 Weaponized Documents and Malicious Scripts 12 Definition 12 Weaponized Documents 12 Malicious Scripts 12 Why Weaponized Documents and Malicious Scripts Matter 12 How to Test Weaponized Documents and Malicious Scripts 13 A Sophos technical paper January 2018 2 Testing Endpoint Security: An Introductory Guide Preface The endpoint security market includes numerous vendors with various powerful technologies designed to keep IT assets safe, yet organizations of all sizes are still susceptible to even the most basic attacks, such as a phishing email with a weaponized document or embedded malicious macro. In general, there is a disconnect between the standard processes by which the industry tests security products to validate their efficacy and the broad spectrum of rapidly evolving real-world threats. What’s worse, the endpoint security market has become shrouded in a fog of acronyms, vendor-specific terminology, and forward-looking marketing. The risk to IT organizations is that they place the security of their endpoints and servers in the hands of a product that has not been effectively tested beyond a few basic scenarios. Perhaps due to varying availability of malicious file samples, ill-understood saliency of threat types, or the costs associated with construction of a versatile testing environment, there is a tendency to measure performance of endpoint security across only a single threat vector. As a result, tests are often not holistic, and disproportionally focus on file-based portable execution detection (we'll expand upon this later). As a result, we at Sophos decided to provide a resource for those who wish to become excellent testers of endpoint security, to provide an aid in conducting unbiased, extensive, and effective testing on any endpoint security product. We did not wish to produce yet another exhaustive "How to Test" guide, but rather a guide on how to become a world-class tester. There are ample resources for those that need a simple step-by-step guide on how to download some malicious .exes and double-click on them. There is, however, a distinct lack of resources catering towards helping a tester establish the connections between how attacks upon networked infrastructure are performed in the real-world and the various capabilities of endpoint security products to defend against them (and how to validate their efficacy). We hope you find this resource both enlightening and useful. A Sophos technical paper January 2018 3 Testing Endpoint Security: An Introductory Guide Introduction Why Conduct Endpoint Security Testing No organization can afford ineffective endpoint security that fails to provide protection against the wide array of ever-increasing and mutating real-world threats. Worse still is that many organizations must rely on third-party tests, vendor-biased or influenced tests, or simply no tests at all. In the age where attackers can learn their skills in a matter of hours watching tutorials on YouTube, the importance of knowing, with some degree of relative certainty, how effective your endpoint security product is couldn't be greater. In the past few years, there has been a movement among security professionals and hobbyists to empower themselves and take the matter of testing into their own hands. The old adage "the proof of the pudding is in the eating" comes to mind. While it is commendable that an increasing number of individuals wish to supplement third-party and first-party testing results with their own, the industry has been awash with advice and guidance typically focused on how to test antivirus and antivirus alone, often via a single attack vector (such as double-clicking on a malicious file or a scripted equivalent). Attackers are well aware that the majority of testing of endpoint security products is focused on this single attack vector, typically of portable executables. This has given rise to a number of antivirus evasion techniques, and the prevalence of their use, such as code caving an executable or exploiting vulnerabilities within software, all the while (ineffective) testing would appear to indicate that the endpoint's defenses are strong. Best phrased by Bruce Schneier, "Security is a chain; it's only as secure as the weakest link.”¹ Testing is essential to find out, for ourselves, how strong our defenses are, for if they are weak, we are only a matter of a misplaced click away from making tomorrow’s news headlines. Attack Chain and Modern Attacks With modern attacks, it is rare that a single event or malware component constitutes the entirety of how an adversary attacks an organization to complete an objective. Most attacks involve multiple steps or stages and not all attacks will use the same techniques to achieve their objective. When observing opportunistic, real-world attacks, it is important to understand that breaking any link in the chain of events that the adversary must complete is often sufficient to thwart the attack in its entirety. The more capabilities a security product has to prevent and detect elements across the length of the attack chain, the more opportunities it has to defeat the attack. Often in testing, organizations focus on a single aspect of an attack, like the detection of malicious executables or the prevention of data theft. While this focus is valid in determining a product’s ability to stop that step of the attack, it is also important to realize that not all attacks will be detectable if the protection software only focuses on only some techniques and not others. For example, having software that is great at detecting malicious executables will do an admirable job in detecting attack campaigns that use malicious executables, but will be completely useless if the adversary is able to complete their objective while never delivering an executable file. ¹ https://www.schneier.com/books/secrets_and_lies/pref.html A Sophos technical paper January 2018 4 Testing Endpoint Security: An Introductory Guide Web and email have become common vectors of delivering an attack as they specifically target humans to socially engineer them. While this document focuses on attacks post- delivery, these are two pivotal stages of the attack chain that should be considered when conducting testing. A product that can block access to the malicious URL or IP address, from which the attack is launched or controlled, breaks the entire chain of the attack at its earliest point. Equally, a product that can block an inbound email or provide heightened scrutiny of email clients and web browsers (context-aware behavioral monitoring) will add an invaluable layer of additional security alongside typical anti-malware capabilities. When evaluating a protection product, it is important to understand how it will address attacks from multiple perspectives. Some of this can be done with testing but often it is just as easy to read the product literature to understand if the product is offering a point solution that targets a single technique or is more comprehensive. Some things to consider and test for include: Ì Attack Surface reduction, preventing users from going to known dangerous locations on the internet or plugging in uncontrolled devices to their computer Ì Ability to detect or prevent the deployment of malicious executables Ì Ability to detect or prevent interaction with malicious documents, image files, and other non-executable files Ì Ability to prevent legitimate applications from performing malicious actions Ì Ability to detect and take action on malicious code or scripts loaded directly into memory that do not require a file to be written to disk Ì Ability to prevent the exploitation of known and unknown vulnerabilities in applications already deployed on the device As a real-world example of the nature of a modern attack, let’s look at NotPetya, the ransomware attack from early 2017. It should be noted that NotPetya was a supply chain attack – the attackers compromised the updating system for a popular software package so that, when the users of the package updated to the latest version, they downloaded and installed NotPetya. An interesting part of NotPetya was that the ransomware mechanism appears to have no intention of collecting the ransom, instead seemingly focused on causing nothing but destruction – a stark contrast to the original Petya variant whose orchestrators would provide a decryption key upon payment of the ransom. After NotPetya was installed via the supply chain attack, it overwrote sections of the master boot record and leveraged exploits in system components to spread laterally to infect more machines. At the same time, it harvested user authentication credentials and scanned the local network for other devices to attack and ultimately encrypt files on those devices. Protection software that stopped any one of the aspects of the attack would either completely prevent the attack or significantly reduce the damage. An endpoint security product that could stop all the aspects is more effective than a product that depends upon a single protection method to do all the work.