Record and Replay for Android: Are We There Yet in Industrial Cases? Wing Lam Haibing Zheng Tao Xie Zhengkai Wu Hui Luo [email protected] Dengfeng Li Peng Yan University of Illinois at Wenyu Wang Yuetang Deng Urbana-Champaign, USA {winglam2,zw3,dli46,wenyu2}@ {mattzheng,huiluo,peteryan, illinois.edu yuetangdeng}@tencent.com University of Illinois at Tencent, Inc., China Urbana-Champaign, USA ABSTRACT ACM Reference format: Mobile applications, or apps for short, are gaining popularity. The Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and Replay for input sources (e.g., touchscreen, sensors, transmitters) of the smart Android: Are We There Yet in Industrial Cases?. In Proceedings of 2017 11th devices that host these apps enable the apps to offer a rich experi- Joint Meeting of the European Software Engineering Conference and the ACM ence to the users, but these input sources pose testing complications SIGSOFT Symposium on the Foundations of Software Engineering, Paderborn, to the developers (e.g., writing tests to accurately utilize multiple in- Germany, September 4–8, 2017 (ESEC/FSE’17), 6 pages. put sources together and be able to replay such tests at a later time). https://doi.org/10.1145/3106237.3117769 To alleviate these complications, researchers and practitioners in recent years have developed a variety of record-and-replay tools to support the testing expressiveness of smart devices. These tools 1 INTRODUCTION allow developers to easily record and automate the replay of com- Smart devices such as mobile phones are becoming increasingly plicated usage scenarios of their app. Due to Android’s large share widespread and these devices rely on mobile applications, or apps of the smart-device market, numerous record-and-replay tools have for short, in order to perform a variety of activities. These apps been developed using a variety of techniques to test Android apps. make use of the many input sources included in a smart device, To better understand the strengths and weaknesses of these tools, such as the touchscreen, sensors (e.g., GPS, accelerometer, compass, we present a comparison of popular record-and-replay tools from gyroscope), and transmitters (e.g., WiFi, Bluetooth) to perform researchers and practitioners, by applying these tools to test three their activities. The numerous input sources on a smart device popular industrial apps downloaded from the Google Play store. pose challenges for app developers to create tests that accurately Our comparison is based on three main metrics: (1) ability to repro- utilize these input sources and to replay an input generated from duce common usage scenarios, (2) space overhead of traces created them at a later time. Record-and-replay tools developed by both by the tools, and (3) robustness of traces created by the tools (when practitioners and researchers have aimed to address such problems. being replayed on devices with different resolutions). The results In particular, these tools allow developers to easily record and from our comparison show which record-and-replay tools may be replay complicated usage scenarios of their app. By automating the the best for developers and identify future directions for improving replay of the complicated usage scenarios of their app, developers these tools to better address testing complications of smart devices. no longer have to manually control each input source every time CCS CONCEPTS they want to test their app. The automation also allows developers to test their apps at scale (e.g., testing their app on multiple devices • Software and its engineering → Software testing and de- at once). bugging; Various record-and-replay tools have been developed by prac- titioners and researchers for the Android platform for three main KEYWORDS reasons. First, majority of the smart devices in the world are using Android, GUI testing, Record-and-replay the Android platform and consequently, tools compatible with the platform are thereby compatible with most apps and devices out there. Second, the Android platform is used in the majority of smart Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed devices partly because the Android platform is compatible with for profit or commercial advantage and that copies bear this notice and the full citation a variety of devices. These devices (often varying in screen sizes on the first page. Copyrights for components of this work owned by others than ACM and available input sources) make app developers’ testing process must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a expensive. Record-and-replay tools can automate what otherwise fee. Request permissions from [email protected]. would be a manual process for developers to confirm that their app ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany works as expected on a variety of devices. Third, with the Android © 2017 Association for Computing Machinery. ACM ISBN 978-1-4503-5105-8/17/09...$15.00 platform being open-sourced, record-and-replay tools can make https://doi.org/10.1145/3106237.3117769 use of the Android platform to gain complete access to record and Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie replay the interactions between the app under test and the Android tools, on the other hand, are tools that ought to be installed directly platform or other apps. onto a mobile device, and the recording and replaying of events To investigate the practicality of these different record-and- are controlled through the installed app. These tools commonly replay tools for industrial practices, we present the first compari- require the user’s device to be rooted and often lack functionality son of popular record-and-replay tools from practitioners and re- compared to desktop tools. The tools used in our study are found searchers by applying these tools on three popular industrial apps by examining the research literature in record-and-replay tools or from the Google Play store. In particular, we study the strengths are recommended to us by the WeChat developers [18], a highly and weaknesses of these tools, how effective they are compared to popular messenger app with over 900 million monthly active users. each other, and how they could be improved. Our comparison is To understand desirable characteristics of record-and-replay based on three main metrics: (1) ability to reproduce common usage tools from industry practitioners, we consult with over 30 develop- scenarios, (2) space overhead of traces created by the tools, and (3) ers from WeChat’s Android Development and Engineering Tools robustness of traces created by the tools (when being replayed on team. These WeChat developers have over 100 years of experience devices with different resolutions). We evaluate the tools’ ability to combined in testing mobile apps. The WeChat developers inform reproduce common usage scenario and their traces’ robustness to us of five highly desirable characteristics that a record-and-replay different devices, since prior to a new version’s release, developers tool should have and four secondary characteristics that are also of commonly have to test their app on a variety of devices to ensure interest. The five highly desirable characteristics that a record-and- that their changes do not unexpectedly affect the common usage replay tool should have are whether the tool is coordinate sensitive scenarios of their app on such devices. In addition, we evaluate the (recording based on coordinates), widget sensitive (recording based space overhead of the traces generated by the tools to show which on a GUI widget’s unique identifier, e.g., R.id), state sensitive (the tools utilize space efficiently. tool can restore the state of the app when the trace was recorded), The results from our comparison show which record-and-replay timing sensitive (whether the tool can record the time difference tools may be the best for developers and identify future directions between the events when they were recorded) and lastly, whether for improving these tools to better address testing complications of the tool requires the developers to install a custom operation sys- smart devices. In our experiment, we find that the tools under study tem (OS) onto their device. The four secondary characteristics that have different problems and they do not meet the expectation of are also of interest for a record-and-replay tool are access to the developers. On our project website [13], readers can see and replay tool’s source code, and not having to require access to the source the recorded traces of common usage scenarios for each of the apps code of the app under test, instrumenting the app, and rooting the under test. device in order to use the tool. This paper makes the following main contributions: Table 1 provides an overview of the record-and-replay tools for • A survey of popular record-and-replay tools for Android Android that researchers and practitioners commonly use. The table apps. reports all of the tools and classifies them according to what type • A comparative study of applying such tools on three popu- of tool it is (desktop or app). Additionally, the table presents other lar industrial apps. useful information for each of the tools, such as whether the tool • An analysis of the results for highlighting the strengths and is sensitive to coordinates, widgets, state or timing changes, and weaknesses of the different tools under study and possible whether the tool is publicly available, instruments the app, requires future directions for record-and-replay tools.