Managing App Testing Device Clouds: Issues and Opportunities Mattia Fazzini Alessandro Orso University of Minnesota Georgia Institute of Technology Minneapolis, MN, USA Atlanta, GA, USA [email protected] [email protected] ABSTRACT Table 1: Android versions, devices, and tests in our study. Because creating and maintaining an in-house test lab is expensive Android Version # Devices CTS Version Tag # Modules # Tests Jelly Bean 4:2 2 4.2.2_r1 50 17268 and time-consuming, companies and app developers often use de- Jelly Bean 4:3 5 4.3_r2.2-cts 57 18013 vice clouds to test their apps. Because quality-assurance activities KitKat 4:4 38 cts-4.4_r4 73 24039 depend on such device clouds, it is important to understand possible Lollipop 5:0 11 cts-5.0_r9 93 33929 Lollipop 5:1 6 cts-5.1_r28 96 34207 issues related to their use. To this end, in this paper we present a Marshmallow 6:0 15 cts-6.0_r32 114 37613 preliminary study that investigates issues and highlights research Nougat 7:0 7 cts-7.0_r32 124 43494 opportunities in the context of managing and maintaining device Nougat 7:1 2 cts-7.1_r28 133 43598 Oreo 8:0:0 11 cts-8.0_r20 195 59459 clouds. In the study, we analyzed over 12 million test executions on Oreo 8:1:0 4 cts-8.1_r16 210 60383 110 devices. We found that the management software of the cloud Pie 9 9 cts-9.0_r9 289 84182 infrastructure we considered affected some test executions, and study, we ran the set of tests from Google’s Android Compatibility almost all the cloud devices had at least one security-related issue. Test Suite [8] on the devices in the AWS Device Farm [1] and used ACM Reference Format: the tests to identify issues in this device cloud. After analyzing Mattia Fazzini and Alessandro Orso. 2020. Managing App Testing Device more than 12 million test executions on 110 devices, we found Clouds: Issues and Opportunities. In 35th IEEE/ACM International Conference that (1) the cloud management software used to run and update on Automated Software Engineering (ASE ’20), September 21–25, 2020, Virtual devices can interfere with test executions and (2) devices tend to Event, Australia. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/ have security-related issues that could be exploited by attackers. 3324884.3418909 1 INTRODUCTION 2 METHODOLOGY As mentioned above, we investigated possible issues with device Thorough in-house testing of Android apps has been particularly clouds by executing the set of tests in Google’s Compatibility Test challenging for companies and developers, due to the fragmentation Suite (CTS) [8] on devices hosted in the AWS Device Farm (DF) [1]. of the Android ecosystem [6, 11, 12, 14, 16, 18, 19, 21, 26, 34–36, 40]. Google’s CTS provides a mechanism for checking whether a Because of the extremely large number of devices and versions of device’s Android operating system (OS) exhibits standard behavior; the Android operating system running in the field, companies and devices that pass all CTS tests are considered Android compatible. developers are required to check that their apps behave as expected This helps ensure that app developers can rely on a consistent execu- on a large set of devices. However, creating and maintaining an tion environment despite the various OS customizations performed in-house test lab with a large set of devices is expensive, difficult, by device vendors [6, 34, 35]. and ultimately impractical [7]. We selected DF as our cloud testing environment because (i) To support companies and developers, many organizations [1, 9, is a popular environment for mobile app testing [13, 29, 30], (ii) 13, 25, 29, 30] built publicly accessible device clouds that developers offers access to the machine driving the test suite execution (which can use to thoroughly test their apps. Specifically, developers can is necessary to run the CTS tests), and (iii) has a wide variety of use these cloud devices to run tests on multiple devices, monitor devices running different versions of the Android OS. test execution progress, and retrieve text execution artifacts. In our study, we considered devices certified as Android com- Understanding possible issues with device-cloud infrastructure patible and checked whether they passed all the CTS tests also in is extremely important, as these issues might affect the quality as- the cloud testing environment. Because these devices passed all surance processes that rely on such infrastructure. In this spirit, we CTS tests when the tests were run locally, test failures on the cloud performed a preliminary study that aims to investigate issues, as devices would likely indicate issues in the cloud infrastructure. well as highlight research opportunities, in the context of manag- Table 1 shows relevant information for the devices we considered ing and maintaining cloud-based app-testing infrastructure. In the and their corresponding Android OS versions. For each version of Permission to make digital or hard copies of all or part of this work for personal or the Android OS (Android Version), the table shows the number of classroom use is granted without fee provided that copies are not made or distributed devices running that version (# Devices), the CTS version used (CTS for profit or commercial advantage and that copies bear this notice and the full citation Version Tag),1 the number of test modules (# Modules), where a test on the first page. Copyrights for components of this work owned by others than the 2 author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or module contains tests that exercises a specific functionality, and republish, to post on servers or to redistribute to lists, requires prior specific permission the overall number of tests. and/or a fee. Request permissions from [email protected]. ASE ’20, September 21–25, 2020, Virtual Event, Australia 1For each OS version, we used the latest release of the CTS available for that version. © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. 2We disregarded some test modules because they either required a specific device ACM ISBN 978-1-4503-6768-4/20/09...$15.00 configuration that we could not set programmatically, or their execution time exceeded https://doi.org/10.1145/3324884.3418909 the time limit enforced by the DF. ASE ’20, September 21–25, 2020, Virtual Event, Australia Mattia Fazzini and Alessandro Orso 175 Android 4.2 Android 4.3 Android 4.4 Android 5.0 Android 5.1 Android 6.0 Android 7.0 Android 7.1 Android 8.0 Android 8.1 Android 9.0 150 125 100 75 50 25 0 G6 Pixel G2 ATT Pixel XL Nexus 5 Nexus 5 Nexus 6 Nexus 5 Sunset 2 Moto G 4 Galaxy J2 Pixel 2 XL Galaxy E7 G Flex ATT Galaxy S10 Moto G ATT G2 T-Mobile G5 T-Mobile Rainbow 4G Aqua Y2 Pro One M8 ATT One M8 ATT One M9 ATT Galaxy J5 4G Nexus 4 4-4-3 G Pad 7.0 ATT Google Pixel 2 Google Pixel 2 Google Pixel 2 Google Pixel 3 Galaxy S3 ATT Galaxy S5 ATT Galaxy S4 ATT Galaxy S5 ATT Galaxy S4 ATT Galaxy S4 ATT Galaxy S7 ATT Galaxy S5 ATT Galaxy S4 Mini Galaxy Note 10 Galaxy Grand 2 Xperia Z4 Tablet Sony Xperia XZ2 Sony Xperia XZ1 Sony Xperia XZ2 Sony Xperia XZ3 Moto E - 2nd Gen Google Pixel 3 XL Galaxy Note5 ATT Galaxy S3 Verizon Galaxy S5 Verizon Galaxy S4 Verizon Galaxy S3 Verizon Galaxy S4 Verizon Galaxy S5 Verizon Galaxy S6 Verizon Galaxy Tab S2 9.7 Galaxy Note 4 ATT Galaxy Note 3 ATT Galaxy Note 3 ATT Galaxy Note 4 ATT Galaxy S3 T-Mobile Galaxy S5 T-Mobile Galaxy S4 T-Mobile Galaxy S6 T-Mobile Galaxy S7 T-Mobile Galaxy S8 T-Mobile Galaxy S6 T-Mobile Galaxy S8 Unlocked Galaxy S8 Unlocked Galaxy S9 Unlocked Galaxy S9 Unlocked Samsung Galaxy A7 Samsung Galaxy A6 DROID Ultra Verizon Galaxy Note 3 Sprint Galaxy S6 SM-G920F Galaxy S7 SM-G930F Galaxy S5 Active ATT Galaxy S4 Active ATT Galaxy S9+ Unlocked Galaxy S9+ Unlocked Galaxy S4 US Cellular Galaxy Note 3 Verizon Galaxy Note 4 Verizon Galaxy Note 2 Verizon Galaxy Note 4 Verizon Galaxy Note5 T-Mobile Galaxy Grand Neo Plus Optimus L70 MetroPCS Galaxy Tab S2 8.0 WiFi Galaxy Tab S2 8.0 WiFi Galaxy S4 mini Verizon Galaxy Note8 Unlocked Galaxy S3 LTE T-Mobile Nexus 7 - 2nd Gen WiFi Nexus 7 - 2nd Gen WiFi Nexus 7 - 2nd Gen WiFi Samsung Galaxy Note 9 Samsung Galaxy Tab S4 Galaxy Note5 SM-N920C Galaxy Note 4 SM-N910H Moto X - 2nd Gen Verizon Galaxy Grand Prime Duos Galaxy Tab 3 7.0 T-Mobile Galaxy Tab 3 Lite 7.0 WiFi Galaxy S6 Edge SM-G925F Galaxy S6 Edge SM-G925F Samsung Galaxy Tab A 10.1 Galaxy S4 Unlocked GT-I9500 Figure 1: Test failures for the different devices considered. Note that, to account for test flakiness5 [ , 22, 33], we executed of failing tests. Across all devices and versions, we found 1,307 each test three times and considered a test as passing if it passed security-related test failures (13% of all failures) caused by 153 dif- at least once. This configuration led to the execution of over 12 ferent tests. All devices but one had at least one security-related test million tests, which took 136 days of machine time to complete. failure. For example, test testStagefright_cve_2016_2507 revealed that seven of the considered devices are affected the vulnerabil- 3 RESULTS AND ANALYSIS ity described in CVE-2016-2507 [4] (an arbitrary code execution vulnerability).