Android 360° Assessment Programme Q1 2020
Copyright © 2020 MRG Effitas Ltd.
MRG Effitas Ltd.
------
Effitas is a world-leading, independent IT security efficacy testing & assurance company. We are trusted by Contents antimalware vendors across the world. Introduction ...... 3 ------Our Mission ...... 4 Tests Applied ...... 5
Early Stage Detection ...... 5 Detection During Installation ...... 5 False Positive Tests ...... 6 MANAGEMENT TEAM: Samples ...... 6 Malicious In-the-wild Samples ...... 6 - Chris Pickard, Chief Executive Officer - Norbert Biro, Chief Technical Officer Simulator samples ...... 7 - Rehab Frimpong, Chief Finance Officer False positive samples ...... 9 Security Applications Tested ...... 9 Test Results ...... 10 False positive tests ...... 25
Summary ...... 26 Conclusions ...... 26
WEBSITE:
www.mrg-effitas.com
TEL: +44 (0)20 3239 9289
EMAIL: [email protected]
TWITTER: @mrgeffitas
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 2 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Introduction
MRG Effitas is an independent IT security research company, with a heavy focus on applied malware analysis. Besides conventional AV efficacy testing and providing samples to other players in the AV field, we regularly test APT detection appliances and enterprise grade IT security products, simulating realistic attack scenarios.
Android devices are used by around 2.3 Billion people around the globe. As the overall platform philosophy allows an easy-to-opt in platform with no mandated central application distribution platform, Android based malware has been on a constant rise since the early Gingerbread days. As a result, the market for Android AVs is heaving with applications that promise loud taglines with ‘100% security’. A quick search on the Play Store for Antivirus products reveals literally hundreds of results – our test aims to help user decisions with a complex test regime with both in-the-wild and artificially crafted simulator samples and results that reflect a real-life efficacy of our test participants.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 3 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Our Mission
In providing quarterly certifications, the MRG Effitas 360 Programme is the de facto standard by which security vendors, financial institutions and other corporations can attain the most rigorous and accurate determination of a product’s efficacy against current financial malware attacks.
We test over twelve months beginning in Quarter 2 and ending in Quarter 1, at which point (or shortly after) we publish our results. As with all of our certification testing, we work with vendors, offering feedback and helping them to improve their product as we go.
Products that pass all tests during a quarter will receive the MRG Effitas certification for Android efficacy protection.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 4 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
been downloaded, ready to be installed. In our opinion, a properly designed Tests Applied AV suite should detect threats as early as possible and should not allow users to install potentially dangerous applications on their devices. MRG Effitas performed an in-depth test of several Android AV applications. The level of protection provided was measured in real-life scenarios with in- Detailed steps were as follows. the-wild pieces of malware as well as some benign samples to map the shortcomings of the applied detection mechanisms. This report summarises 1. Having prepared the test device, we installed and initialized the AV application (accepted the EULA, downloaded the latest definition files, the results of our efficacy tests. accepted all requested permissions etc.) When asked, we enabled SD Card scanning features1. In cases where we received configuration Testing took place on Android 8.0.0 Genymotion emulator images in guides from the vendor, we followed the steps detailed there. February 2020, covering a significant portion of user devices on the market. 2. We set up the application to include the SD Card in the scan scope. In order to ensure maximum compatibility for samples that contain native 3. We downloaded the sample set to the SD Card and started the scan. 4. We instructed the application to remove all suspicious files. ARM code, the ARM Translation package has also been installed on emulator 5. We ran the scan again, until we saw no warning or suspicious files on images. In cases where ARM native libraries have been extensively used and the device. the AV application could not be installed or properly run on an x86 emulator, 6. We collected the remaining samples. we opted for stock Nexus 5x devices with Android 8.0.0. In order to ensure the cleanliness of testing process, the Play Protect feature has been disabled. Detection During Installation
Our efforts were focused on the following aspects of the products. The second scenario involved individual installation of each sample, aiming to check the level of protection provided by the participants.
1. Using adb, we performed an install operation on the device. Following Early Stage Detection the installation, the AV was informed about the newly installed application, kicking in detection routines. Our first scenario focused on an early stage of detection, when test samples 2. We gave plenty of time for the AV to finish all scanning activities 2,3. have been copied on the SD Card drive of the test device. In the tested 3. We created a screenshot of the resulting screen. Should the AV scenario, the device has not yet been infected, malicious APK files have only display a warning or an alert, the test was counted as a Pass, no
1 Due to performance reasons, this option was disabled for most AVs after an out-of-the box installation before actually starting the newly installed application – in our testing methodology, initialization. a ‘too late’ detection or a detection without a clear notification is also considered a Miss. 2 The timeout threshold is a critical aspect of testing. Should the value be too low, the test 3 During the result discussion stage, we actively cooperate with vendors to eliminate timeout results would not reflect actual results as the AV has no chance of finishing detection. We aim related issues, in order to make sure that the figures presented in the report reflect the results of to choose the threshold to be realistic, as it is unlikely that a user waits for several minutes after a realistic scenario.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 5 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
warning resulted in a Miss. All logcat logs were saved from the device during the process. 4. Using adb, we uninstalled the sample and went on to test the next one. Samples
Note that on Android, installation of a piece of malware does not necessarily mean unwanted consequences for the user, as it is the first launch that kicks Malicious In-the-wild Samples in any actual malicious code within. Having started the sample, however, can have detrimental consequences from a security perspective. After the first Testing used an initial 160-sample malware set. All samples have been launch, a piece of malware requesting SYSTEM_ALERT_WINDOW permission categorized using the following labels. is able to continuously display a Device Administrator or an Accessibility Admin request screen to the user. In such cases, the user is unable to get rid • SMS Payment. The application provides features to send SMS messages to premium rate numbers. Most of the selected samples of the application as they have no access to the launcher, the application were able to ‘auto-send’ messages, as they usually opted for the 4 drawer or the Settings application to perform an uninstall . SEND_SMS permission, resulting in a direct financial loss for the victim. • Trojan. Trojans are applications, which display a certain set of features within their description and their overall appearance False Positive Tests suggests some expectations regarding their functionality. However, the implemented modules require a wider range of permissions, which In order to cover all aspects of the efficacy of the participants, a limited set do not belong to the advertised functionality. A typical example is a of samples has also been selected. The samples have been downloaded flashlight app, which can read the contact list, location information and send them to the Internet. from a well-known 3rd party app store, exhibiting no malicious behaviour but • Spyware. We classified a sample Spyware if it leaks information, requiring a varying range of permissions. which can be used to track the user (as most security-conscious users do not wish to be tracked). Ironically, most ad propelled applications using aggressive frameworks qualify as spyware, as they leak IMEI, phone number, phone vendor and model etc. to the ad provider network. • Financial/banking. This type of malware aims for direct financial abuse. A typical financial piece of malware detects if the user is logged in to a mobile banking session using either a browser or mobile banking application and, for instance, might attempt to display a matching phishing site or to draw an overlay window to fool the user into thinking that the session has ended and that they need to re-
4 Note that in order to mitigate this kind of typical malware behaviour, the Android API design a checkbox that can be used to prevent the OS from displaying the screen again. This feature team reviewed the Device Administrator and the Accessibility Admin Request screens to include however, made its way only to recent revisions of the Android API.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 6 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
authenticate. Typically, such samples use permissions to get the task list, combined with the SYSTEM_ALERT_WINDOW permission. Sample distribution • PUA. 5 The term ‘Potentially Unwanted Applications’ denotes applications, which perform actions that are not in alignment with the security-conscious user’s intentions. For instance, applications provided with aggressive advertisement modules usually make it 20% possible for ad campaigners to track individual users, even to assign Banking PUAs the device with the user’s demographic properties through social 41% network ad services. Effitas claims that security-conscious users are sensitive regarding their privacy and possibly no application feature 8% SMSs Spyware can make it up for the users’ private data and browsing habits to be sold over the Internet and a decent AV should let the user know if such Trojans an application is about to be installed. 15%
Note that most samples implement several kinds of operation, therefore 16% most samples fall into several categories (for instance, consider a typical piece of malware, which serves malicious ads and if possible, it attempts to obtain the SEND_SMS permission to send premium rate messages). FIGURE 1 IN-THE-WILD SAMPLE DISTRIBUTION Figure 1. depicts the distribution of test samples. Simulator samples
Simulators are custom samples, introduced into the testing process to put the sophistication of the detection routines to the test. Our simulators were created to simulate the attack model of a ‘malicious 3rd party app store providing backdoored applications’ type of scenario, which means that counterfeit versions of legitimate applications are provided to the victims (many times pirated application versions can be downloaded for free-of- charge). The counterfeit versions are backdoored versions of popular
5 Android applications with a social network integrated advertising module often fall into a kind the developers include an aggressive advertising module. Hence, we included charts, which of ‘grey zone’ from a detection perspective, as any application can be turned into a PUA, should handle PUA and non-PUA samples separately.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 7 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
applications, which, while retaining the functionality of the original application, also include malicious modules.
The samples have been created using a proof-of-concept engine using static smali byte code injection techniques, making no effort to obscure the malicious actions of the injected modules. Many of the simulator samples have been modified to implement Accessibility features, which is a common trait for several malware families.
For testing, we used 5 custom created samples. It is important to stress that these samples have not been collected or observed in-the-wild. Our custom samples implemented a well-known method exploiting the accessibility features of the Android API, which has been a popular method to read on- screen messages, SMS tokens, banking details and other sensitive information. Our samples were counterfeit versions of legitimate Android applications, sending SMS messages, keystrokes, passwords etc. to our custom HTTP web service endpoint.
FIGURE 2 - COUNTERFEIT APPLICATIONS EXPLOITING THE ACCESSIBILITY FEATURES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 8 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
False positive samples Security Applications Tested For false positive testing, a 10-sample set was used, retrieved from non- The following security suites have been selected for testing. Besides well- malicious 3rd party applications stores. The applications have been selected established vendors with considerable reputation and track history, we select to cover a wide range of permissions and functionality. smaller vendors with less market share. As the Play Store is heaving with Android security applications, we tend to select AV products with a considerable number of downloads.
Product name Caption Play Store URL AVG AVG https://play.google.com/store/apps/details?id=com.antivirus Avira Mobile Antivirus Avira https://play.google.com/store/apps/details?id=com.avira.android Bitdefender Mobile Security & Antivirus Bitdefender https://play.google.com/store/apps/details?id=com.bitdefender.security Comodo Mobile Security Antivirus Comodo https://play.google.com/store/apps/details?id=com.comodo.cisme.antivirus Mobile Security & Antivirus ESET https://play.google.com/store/apps/details?id=com.eset.ems2.gp Kaspersky Mobile Antivirus Kaspersky https://play.google.com/store/apps/details?id=com.kms.free Virus Cleaner, Anti-Malware Malwarebytes https://play.google.com/store/apps/details?id=org.malwarebytes.antimalware McAfee Mobile Security McAfee https://play.google.com/store/apps/details?id=com.wsandroid.suite Norton Security and Antivirus Norton https://play.google.com/store/apps/details?id=com.symantec.mobilesecurity Antivirus & Virus Cleaner, Applock, Clean, Booster TAPI Labs https://play.google.com/store/apps/details?id=com.antivirus.mobilesecurity.viruscleaner.applock Zemana Antivirus 2019: Anti-Malware & Web Security Zemana https://play.google.com/store/apps/details?id=com.zemana.msecurity Zoner Antivirus Zoner https://play.google.com/store/apps/details?id=com.zoner.android.antivirus
TABLE 1 - TEST PARTICIPANTS6,7
6 When testing Malwarebytes, a few samples were classified as Misses. During the dispute batches being not detected reliably when considering Malwarebytes. A re-test confirmed that the phase, following the Vendor’s observations, an internal investigation has been conducted to Missed samples were misclassified. identify the cause. As it was discovered, Genymotion images handle start-up services in a 7 At the time of testing, TAPI Labs did not provide SD card scanning features, therefore Early different fashion than what is expected on real devices, resulting in the first sample in the test testing has not taken place.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 9 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Test Results
The tables and charts below show the results of testing under the MRG Effitas Android AV Testing Program.
Averaged non-PUA Detection Scores8
Summary, non-PUA samples missed or detected
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0% Avira Bitdefender ESET Kaspersky Malwarebytes Norton AVG Zoner McAfee Labs TAPI Comodo Zemana
FIGURE 3 - SUMMARY, NON-PUA SAMPLES
8 The figures were created by averaging Early and Install scores for all non-PUA samples.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 10 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Summary, averaged scores Summary Summary Short name blocked missed Avira 100% 0% Bitdefender 100% 0% ESET 100% 0% Kaspersky 100% 0% Malwarebytes 100% 0% Norton 100% 0% AVG 97% 3% Zoner 85% 15% McAfee 81% 19% TAPI Labs 76% 24% Comodo 52% 48% Zemana 42% 58%
TABLE 2 – SUMMARISED RESULTS, NON-PUA SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 11 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Overall non-PUA Detection
Non-PUA samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton AVG AVG Zoner Zoner McAfee McAfee Labs TAPI Comodo Comodo Zemana Zemana
FIGURE 4 - SUMMARY, NON-PUA SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 12 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Sum summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed Avira 144 100,0% 0 0,0% 144 100,0% 0 0,0% Bitdefender 144 100,0% 0 0,0% 144 100,0% 0 0,0% ESET 144 100,0% 0 0,0% 144 100,0% 0 0,0% Kaspersky 144 100,0% 0 0,0% 144 100,0% 0 0,0% Malwarebytes 144 100,0% 0 0,0% 144 100,0% 0 0,0% Norton 144 100,0% 0 0,0% 144 100,0% 0 0,0% AVG 139 96,5% 5 3,5% 141 97,9% 3 2,1% Zoner 122 84,7% 22 15,3% 123 85,4% 21 14,6% McAfee 109 75,7% 35 24,3% 123 85,4% 21 14,6% TAPI Labs n/a n/a n/a n/a 110 76,4% 34 23,6% Comodo 72 50,0% 72 50,0% 77 53,5% 67 46,5% Zemana 19 13,2% 125 86,8% 101 70,1% 43 29,9%
TABLE 3 – RESULTS, NON-PUA SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 13 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
PUA Detection
PUA samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton AVG AVG Labs TAPI Zoner Zoner McAfee McAfee Comodo Comodo Zemana Zemana
FIGURE 5 - SUMMARY, PUA SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 14 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
PUA summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed Avira 16 100,0% 0 0,0% 16 100,0% 0 0,0% Bitdefender 16 100,0% 0 0,0% 16 100,0% 0 0,0% ESET 16 100,0% 0 0,0% 16 100,0% 0 0,0% Kaspersky 16 100,0% 0 0,0% 16 100,0% 0 0,0% Malwarebytes 16 100,0% 0 0,0% 16 100,0% 0 0,0% Norton 16 100,0% 0 0,0% 16 100,0% 0 0,0% AVG 15 93,8% 1 6,3% 16 100,0% 0 0,0% TAPI Labs n/a n/a n/a n/a 14 87,5% 2 12,5% Zoner 13 81,3% 3 18,8% 13 81,3% 3 18,8% McAfee 9 56,3% 7 43,8% 15 93,8% 1 6,3% Comodo 11 68,8% 5 31,3% 12 75,0% 4 25,0% Zemana 1 6,3% 15 93,8% 14 87,5% 2 12,5%
TABLE 4 - PUA SAMPLES BLOCKED OR MISSED
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 15 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Trojan Detection
Trojan samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% AVG AVG Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton Zoner Zoner McAfee McAfee Labs TAPI Comodo Comodo Zemana Zemana
FIGURE 6 - SUMMARY, TROJAN SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 16 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Trojan summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed AVG 79 100,0% 0 0,0% 79 100,0% 0 0,0% Avira 79 100,0% 0 0,0% 79 100,0% 0 0,0% Bitdefender 79 100,0% 0 0,0% 79 100,0% 0 0,0% ESET 79 100,0% 0 0,0% 79 100,0% 0 0,0% Kaspersky 79 100,0% 0 0,0% 79 100,0% 0 0,0% Malwarebytes 79 100,0% 0 0,0% 79 100,0% 0 0,0% Norton 79 100,0% 0 0,0% 79 100,0% 0 0,0% Zoner 74 93,7% 5 6,3% 74 93,7% 5 6,3% McAfee 65 82,3% 14 17,7% 64 81,0% 15 19,0% TAPI Labs n/a n/a n/a n/a 49 62,0% 30 38,0% Comodo 42 53,2% 37 46,8% 44 55,7% 35 44,3% Zemana 16 20,3% 63 79,7% 46 58,2% 33 41,8%
TABLE 5 – RESULTS, TROJAN SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 17 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Banking Detection
Banking samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% AVG AVG Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton Labs TAPI McAfee McAfee Zoner Zoner Zemana Zemana Comodo Comodo
FIGURE 7 - SUMMARY, BANKING SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 18 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Banking summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed AVG 39 100,0% 0 0,0% 39 100,0% 0 0,0% Avira 39 100,0% 0 0,0% 39 100,0% 0 0,0% Bitdefender 39 100,0% 0 0,0% 39 100,0% 0 0,0% ESET 39 100,0% 0 0,0% 39 100,0% 0 0,0% Kaspersky 39 100,0% 0 0,0% 39 100,0% 0 0,0% Malwarebytes 39 100,0% 0 0,0% 39 100,0% 0 0,0% Norton 39 100,0% 0 0,0% 39 100,0% 0 0,0% TAPI Labs n/a n/a n/a n/a 33 84,6% 6 15,4% McAfee 29 74,4% 10 25,6% 34 87,2% 5 12,8% Zoner 28 71,8% 11 28,2% 29 74,4% 10 25,6% Zemana 7 17,9% 32 82,1% 30 76,9% 9 23,1% Comodo 14 35,9% 25 64,1% 17 43,6% 22 56,4%
TABLE 6 – RESULTS, BANKING SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 19 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
SMS Detection
SMS samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% AVG AVG Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton Zoner Zoner McAfee McAfee Labs TAPI Zemana Zemana Comodo Comodo
FIGURE 8 - SUMMARY, SMS SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 20 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
SMS summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed AVG 29 100,0% 0 0,0% 29 100,0% 0 0,0% Avira 29 100,0% 0 0,0% 29 100,0% 0 0,0% Bitdefender 29 100,0% 0 0,0% 29 100,0% 0 0,0% ESET 29 100,0% 0 0,0% 29 100,0% 0 0,0% Kaspersky 29 100,0% 0 0,0% 29 100,0% 0 0,0% Malwarebytes 29 100,0% 0 0,0% 29 100,0% 0 0,0% Norton 29 100,0% 0 0,0% 29 100,0% 0 0,0% Zoner 26 89,7% 3 10,3% 26 89,7% 3 10,3% McAfee 20 69,0% 9 31,0% 25 86,2% 4 13,8% TAPI Labs n/a n/a n/a n/a 19 65,5% 10 34,5% Zemana 17 58,6% 12 41,4% 21 72,4% 8 27,6% Comodo 18 62,1% 11 37,9% 19 65,5% 10 34,5%
TABLE 7 – RESULTS, SMS SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 21 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Spyware Detection
Spyware samples missed or detected
100%
90%
80%
70%
60%
50%
40% early install - - early install early install
30% - - install - - - early install early install early install early install ------
20% early install - - early install early install early install ------10%
0% Avira Avira Bitdefender Bitdefender ESET ESET Kaspersky Kaspersky Malwarebytes Malwarebytes Norton Norton AVG AVG Labs TAPI McAfee McAfee Comodo Comodo Zoner Zoner Zemana Zemana
FIGURE 9 - SUMMARY, SPYWARE SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 22 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Spyware summary Early Early Early Early Install Install Install Install Short name blocked blocked missed missed blocked blocked missed missed Avira 25 100,0% 0 0,0% 25 100,0% 0 0,0% Bitdefender 25 100,0% 0 0,0% 25 100,0% 0 0,0% ESET 25 100,0% 0 0,0% 25 100,0% 0 0,0% Kaspersky 25 100,0% 0 0,0% 25 100,0% 0 0,0% Malwarebytes 25 100,0% 0 0,0% 25 100,0% 0 0,0% Norton 25 100,0% 0 0,0% 25 100,0% 0 0,0% AVG 23 92,0% 2 8,0% 25 100,0% 0 0,0% TAPI Labs n/a n/a n/a n/a 21 84,0% 4 16,0% McAfee 18 72,0% 7 28,0% 21 84,0% 4 16,0% Comodo 18 72,0% 7 28,0% 20 80,0% 5 20,0% Zoner 18 72,0% 7 28,0% 18 72,0% 7 28,0% Zemana 5 20,0% 20 80,0% 21 84,0% 4 16,0%
TABLE 8– RESULTS, SPYWARE SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 23 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Simulators
Simulator samples blocked or missed 100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0% AVG Bitdefender Malwarebytes Norton ESET McAfee Avira Kaspersky Zoner Labs TAPI Comodo Zemana
FIGURE 10 - SUMMARY, SIMULATOR SAMPLES
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 24 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Simulator summary Full name Blocked Blocked Missed Missed AVG 5 100% 0 0% Bitdefender 5 100% 0 0% Malwarebytes 5 100% 0 0% Norton 5 100% 0 0% ESET 4 80% 1 20% McAfee 4 80% 1 20% Avira 3 60% 2 40% Kaspersky 2 40% 3 60% Zoner 2 40% 3 60% TAPI Labs 1 20% 4 80% Comodo 0 0% 5 100% Zemana 0 0% 5 100%
TABLE 9 – RESULTS, SIMULATOR SAMPLES
False positive tests
All test participants achieved 100% results in this category.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 25 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Summary Conclusions As a result of our testing efforts, a couple of conclusions can be drawn from The following AV engines reached a 100% detection rate in a non-PUA our time with the AV engines and samples in our test lab. sample set, therefore they have been awarded with the MRG Effitas Certificate. Vendor reputation and extra services • Avira As of 2020Q1, the majority of the well-established vendors reached a perfect • Bitdefender or near-perfect score in the in-the-wild categories. From a user perspective, • ESET this is all good news as several viable options are provided, some even • Kaspersky without a subscription fee. In the future, we expect that the extra features • Malwarebytes (VPN, family locator, connections with desktop AV licenses etc.) will have a • Norton significant effect on user choice in the future.
‘AV as another app’
Testing led us to the conclusions that detection for many AVs relies heavily on the metadata of installed packages (hashes, developer certificates etc.), meaning that unlike in a Windows based environment, an AV is unable to get an insight into the actual activity of other applications. This behaviour is in alignment with the basic Android security principles, as “AV is another app”. As a result, having already started the freshly installed sample, it is quite hard to get rid of some samples in the in-the-wild test set, making a timely and properly displayed detection an absolute must for AV engines.
Detection mechanisms
Our tests confirmed that most AVs use different methods for detection before and after installation. This is due to the fact that prior to installation, different set of metadata is available for and AV engine of a file that is stored on the SD card than what is available after its installation.
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 26 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
Furthermore, the general idea of providing Early scan features, is a significant Detection notification feature for the security conscious. As it provides an opportunity to scan an apk file, prior to installation, it makes it possible to detect malicious During the history of 360 Android tests, we noticed that there are significant applications, prior to being installed and sitting only one tap away from differences in terms of user notification. When it comes to a successful causing all sorts of havoc. In our opinion, Early scan features should be more detection, efficient and clear user notification is an essential part of both the prominent in all Android based AVs (in our recent tests, we almost always efficacy of the AV application and the overall user experience. found applications, not providing this feature). The tested AVs choose one of the following approaches.
Simulator detection 1. A separate activity is launched, usually with a bright red background and a couple of lines describing the nature of the threat, presented Most AV engines, having detected our custom simulator samples in past by the freshly installed application. tests, were able to perform detection purely based on the package signature 2. The Android notification subsystem is utilised to issue an important traits. This means that even though a notification has been displayed for notification, usually displayed in the status bar. those samples, the successful detection has been a result of a mechanism, heavily prone to false positives. As a result, in our previous Android 360 Both approaches have their merits, namely a separate activity is harder to engagements many AVs had problems with detecting simulator samples. dismiss or overlook, and the second one being more streamlined with the overall Android experience. However, Android provides a lot of options for Our test lab has been reached out to on several occasions with claims that users to customize notifications, therefore it is possible for the notification to the simulators we utilise, do not present a lifelike challenge for an AV engine. get lost in the clutter, therefore the majority of the tested apps opt for the However, field reports show the presented scenario, when an adversary first approach. patches an existing Android application to perform hidden spying activity, are well known and have been utilised for almost a decade9. The most famous of As for the wording and the overall design of the displayed notification and campaigns with this approach, is still the Dark Caracal APT10, with an the consequent user choice description, there is a significant room for excellent analysis by Lookout11. improvement in many apps. The Android way is to communicate as much information to the user as possible, just enough so that she can make a responsible decision – however, a responsible user has to read and process the text displayed on the screen, which presents a significant mental load,
9 https://threatvector.cylance.com/en_us/home/mobile-malware-and-apt-espionage-prolific- 11 pervasive-and-cross-platform.html https://www.lookout.com/info/ds-dark-caracal-ty
10 https://en.wikipedia.org/wiki/Dark_Caracal
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 27 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
especially for the not tech savvy. As a result, a well-designed UX can make all the difference for the everyday user, when it comes to making users’ choices more responsible (and consequently, their devices more secure).
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 28 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder
MRG Effitas Android 360 Degree Assessment Programme – Q1 2020 29 Copyright © 2020 MRG Effitas Ltd. This article or any part thereof may not be published or reproduced without the consent of the copyright holder