Arxiv:2104.11271V1 [Cs.HC] 22 Apr 2021 Currently, Many People Regard Smartphones As an Extension of Their Bodies

ASYSTEMATIC SURVEY ON ANDROID APIUSAGEFOR DATA-DRIVEN ANALYTICS WITH SMARTPHONES

APREPRINT

Hansoo Lee Joonyoung Park Uichin Lee∗ School of Computing Graduate School of Knowledge Service Engineering School of Computing KAIST KAIST KAIST Daejeon, Republic of Korea Daejeon, Republic of Korea Daejeon, Republic of Korea [email protected] [email protected] [email protected]

April 26, 2021

ABSTRACT

Recently, there has been an increase in industrial and academic research on data-driven analytics with smartphones based on the collection of app usage patterns and surrounding context data. The Android mobile operating system utilizes Usage Statistics API (US API) and Accessibility Service API (AS API) as representative APIs to passively collect app usage data. These APIs are used for various research purposes as they can collect app usage patterns (e.g., app status, usage time, app name, user interaction state, and smartphone use state) and ﬁne-grained data (e.g., user interface elements & hierarchy and user interaction type & target & time) of each application. In addition, other sensing APIs help to collect the user’s surroundings context (location, network, ambient environment) and device state data, along with AS/US API. In this review, we provide insights on the types of mobile usage and sensor data that can be collected for each research purpose by considering Android built-in APIs and sensors (AS/US API, and other sensing APIs). Moreover, we classify the research purposes of the surveyed papers into four categories and 17 sub-categories, and create a hierarchical structure for data classiﬁcation, comprising three layers. We present the important trends in the usage of Android’s built-in APIs and sensors, including AS/US API, the types of data collected using the presented APIs, and discuss the utilization of mobile usage and sensor data in future research.

Keywords Smartphone sensing, User interaction data, data analysis, android accessibility, Android API, data collection, sensor, mobile device, smartphone, data driven analytics

1 Introduction arXiv:2104.11271v1 [cs.HC] 22 Apr 2021 Currently, many people regard smartphones as an extension of their bodies. According to research of smartphone usage statistics, 79% of adults carry their mobile phone for 22 hours a day [1], and smartphone users spend time with their devices for an average of 3 hours and 15 minutes a day [2]. During smartphone usage, ﬁne-grained contextual information (e.g., ambient environment, physical activity, app usage pattern) can be collected from built-in sensors and application programming interfaces (API). Thus, smartphones provide new opportunities to collect and analyze a user’s everyday lifelog data (i.e., smartphone/mobile usage and sensor data) for enabling digital health services and user experience research. For example, mobile usage and sensor data serves as a valuable resource for the diagnosis and prognosis of mental and physical health [3, 4]. As the number of smartphone users worldwide in 2021 is expected to increase to more than 3.8 billion people [5], it has become increasingly important to explore novel opportunities for data-driven analytics using mobile usage and sensor data. This review focuses on the Android mobile operating system (OS) that dominates 85% of the global smartphone market [6]. Android OS allows developers to customize these built-in APIs to build applications for smartphone data collection.

∗The corresponding author A PREPRINT -APRIL 26, 2021

For this reason, Android has been actively used in the research community. As shown in Figure 1, Android APIs1 can identify a user’s life patterns by collecting various mobile usage and sensor data through human and physical related context aware. In addition, users touch their phone an average of 2,617 times a day [7]. Thus, we can collect app usage pattern data (e.g., application status/name/time) on frequent digital interactions between users and smartphones, including fine-grained interaction details (e.g., user interaction type and time, user interface (UI) hierarchy and elements, notification) for each application. In addition, we can collect the user’s external contextual data, such as physical activities and environmental states. Among Android’s built-in APIs, researchers commonly use the Accessibility Service API (AS API) which is also called Accessibility API. The AS API includes the classes such as AccessibilityService2, AccessibilityEvent3, Accessibili- tyRecord4, AccessibilityManager5 [8]. The AS API is operated to generate various AccessibilityEvent classes (e.g., transition type, view type, exploration type, and notification type) for app usage, by recognizing the state change of a smartphone which is the event based system [9]. AccessibilityEvent can be used to collect fine-grained datasets for app usage pattern data as depicted in Figure 1. Another way to collect app usage pattern data is using the Android Usage Statistics APIs (US APIs) such as UsageS- tatManager6, UsageStats7, UsageEvents8, UsageEvents.Event9. US API can be used to collect the app and device usage history and statistics as shown in Figure 1 [10]. US API can access device usage history and statistics to obtain the currently running app information in third-party applications (as of Android API level 21 in 2014). The US API provides functions to query app usage logs by specifying time units of day, month, and year, thereby retrieving the results specified by the interval which is a more granular and explicit query system [11]. Since 2014, the US API has been mainly used to collect app usage pattern data in many studies (e.g., [12, 13, 14, 15]). However, the AS API is still exclusively used to track the user interface (UI) status information (e.g., UI changed status, interaction type, UI properties & hierarchy, and notification) in the present studies (e.g., [16, 17, 18, 19]). Along with the AS/US APIs, other Android built-in sensors and APIs are also used to collect mobile usage and sensor data for the research. These data can be collected using Android built-in hardware (HW) sensors (e.g., accelerometer, gyro, GPS, Wi-Fi, Bluetooth, proximity, light, compass, pressure) and APIs (e.g., Sensor APIs11, Wi-Fi APIs12, Google Location Services APIs13, Bluetooth APIs14). In other words, we can use these Android built-in sensors and APIs to track a user’s physical activity, physiological state, and surrounding context information (e.g., location, ambient environment) for the research as shown in Figure 1. Furthermore, Android system APIs (e.g., BroadcastReceiver15, BatteryManager16) can be used to collect device status information (e.g., battery information, screen status, device information setting, ringer mode) as depicted in Figure 1. Leveraging data-driven analytics with smartphone data is widely performed in various research fields (e.g., sensors, human-computer interaction, ubiquitous computing, and mobile computing). These include:

• Understanding a user’s smartphone usage pattern [17, 20, 21, 22, 18] • Identifying a user’s notification checking factors, how the user is affected by the notification, and message management and scheduling [23, 24, 16, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]. • Understanding a user’s psychological & health states and personal traits in digital phenotype and digital health research fields [37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48]. • Validating permission, improve authentication & authorization, and improve protection from malware in privacy & security research fields [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].

1https://developer.android.com/reference 2https://developer.android.com/reference/android/accessibilityservice/AccessibilityService 3https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent 4https://developer.android.com/reference/android/view/accessibility/AccessibilityRecord 5https://developer.android.com/reference/android/view/accessibility/AccessibilityManager 6https://developer.android.com/reference/android/app/usage/UsageStatsManager 7https://developer.android.com/reference/android/app/usage/UsageStats 8https://developer.android.com/reference/android/app/usage/UsageEvents 9https://developer.android.com/reference/android/app/usage/UsageEvents.Event 11https://developer.android.com/reference/android/hardware/package-summary 12https://developer.android.com/reference/android/net/wiﬁ/package-summary 13https://developers.google.com/android/reference/com/google/android/gms/location/package-summary 14https://developer.android.com/reference/android/bluetooth/package-summary 15https://developer.android.com/reference/android/content/BroadcastReceiver 16https://developer.android.com/reference/android/os/BatteryManager

2 A PREPRINT -APRIL 26, 2021

Figure 1: Mobile usage and sensor data collection through Android APIs

• Improving user’s interaction, accessibility, and optimization [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]. • Testing and programming support ﬁelds, such as graphical user interface (GUI) automated testing, crowdsourced system testing, programming support for developer, and app performance measurement [73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86]. • Development of frameworks and platforms to collect smartphone data [87, 88].

Therefore, it is important to understand the nature of the collected smartphone data and its research purpose. Existing studies have reviewed mobile usage and sensor data collection, but they have mainly focused on one specific purpose or type of data related to their research. For example, Rooksby et al. [38] investigated the type of collected mobile data and the duration of the data collection for monitoring student mental health. In addition, Kourtis et al. [3] investigated the types of mobile and wearable devices data collection, metrics, and sense-domain for each data to identify digital biomarkers for Alzheimer’s disease. Berkel et al. [89] investigated the types of collected sensor data, experimental period, personal device, trigger type of the experiment sampling method (ESM), the response rate of ESM, and the ESM’s compaction for the research used by collecting and utilizing ESM data and other sensor data. However, to the best of our knowledge, no prior studies performed a systematic survey of the existing research papers based on using the Android’s AS/US APIs, built-in sensors and other APIs for data-driven analytics. Furthermore, existing papers often described what data were used, but did not describe which APIs were used for the collection of each data type. The existing review papers do not give a holistic view about the collection of mobile usage and sensor data, and the research purposes that the data could be used for. Therefore, the main objectives of this research were as follows: We first identified the major purposes of using AS/US APIs data in existing research. We then reviewed the type of mobile usage and sensor data that can be used for each research purpose. For the research purpose classification, we systematically reviewed the existing studies and categorized them through affinity diagramming. For data classification, we considered three major categories: i.e., physical context sensing, system sensing, and interactive sensing data by referring to Android developer documentation [90] and existing data-driven analytics research through

3 A PREPRINT -APRIL 26, 2021

Android’s AS/US APIs, built-in sensors, and the other APIs. In addition, mobile usage and sensor data were categorized into various subcategories in a hierarchical structure. This study aims to provide the guidelines for using and reporting Android’s AS/US APIs, and other APIs data for research purposes. The main contributions of our work can be summarized as follows:

• We provide an overview of the trends in existing research over the last ten years by categorizing the papers on the basis of the research purposes. Next, we report on how to use and collect the data, and summarize the type of data collected according to research purposes and AS/US APIs used. • We present and elucidate the following: (1) A summary of the results; (2) Classiﬁcation of papers based on the ease of identifying data terms; (3) Practical issues about Google Android policy when using AS and US APIs for data-driven analytics; (4) Discussion of the privacy and security issues associated with mobile usage and sensor data collection and use;

The rest of this article is organized as follows. Section 2 provides background information on AS/US API, built-in sensors, and the other APIs, including a description of the historical overview, purpose of APIs, main functionalities of the APIs, and types of data that can be collected. The previous review studies on mobile usage and sensor data-driven analytics research is also explained in Section 2. In Section 3, we explain the overall methods of finding, classifying, and selecting the literature covered for the systematic survey. In Section 4, we identify and classify the research purpose in surveyed studies. In Section 5, we classify the mobile usage and sensor data used in surveyed studies. In Section 6, we present the resuls of data categorization in surveyed studies. In Section 7, we summarize and discuss the key findings from classification results in each section, summarize the comparative analysis of existing studies, discuss the open issues (i.e., related analysis of data term issues, privacy & security issues, and practical issues in AS/US API use), and list the limitations of this study and scope for future research. Finally, we conclude this article in Section 8.

2 Background

2.1 Existing reviews on smartphone data utilization research

As presented in Table 1, we list the existing review studies that investigated prior works that used various mobile usage and sensor data for specific research purposes or specific data. We examine the reviewed items, type of investigated data, review purpose, and number of surveyed studies of the existing review studies in Table 1. Rooksby et al. [38] surveyed the three reviewed items corresponding to 19 studies which used sensor data to monitor student mental health. Kourtis et al. [3] investigated the three reviewed items for sensor data corresponding to 62 studies that used mobile and wearable devices to determine a digital biomarkers for Alzheimer’s disease. Liang et al. [4] reviewed 92 studies that used various types of data (e.g., online activities, blood pressure, ECG, EEG), including mobile and wearable device data, in a digital phenotyping study of mental health. There are also review studies that investigated research that used various mobile usage and sensor data under different experimental conditions, or focused on specific mobile usage and sensor data as depicted in Table 1. Church et al. [20] analyzed the differences in data collected in 22 papers according to the number of different experimenters, experiment period, deployment, recruitment, incentives, and methods. Berkel et al. [89] surveyed 110 studies using the experience sampling method (ESM) in terms of experiment duration, types of triggers, response rate, compensation, personal device, and utilized data. Because these review studies investigated highly specific content (i.e., specific research purposes, data type, and experimental conditions) related to mobile usage and sensor data, it is difficult to provide a wide range of information about which mobile usage and sensor data is used for what research purposes. Furthermore, mobile usage and sensor data terminology used in each field of research and each study are named differently, making it difficult to specify the terms of mobile usage and sensor data as search keywords. Therefore, it is difficult to determine which mobile usage and sensor data terms to use as literature search terms.

2.2 Basic Information on Android APIs to Access Mobile Usage and Sensor Data

We describe the basic components and API framework to understand the Android APIs for mobile usage and sensor data collection. The Android Operating System (OS) consists of five layers which are system apps, API framework, Native C/C++ library, Android runtime, Linux kernel, and hardware abstraction layer (HAL) [91]. Third-party apps can access and utilize mobile usage and sensor data through the Android API framework’s manager objects such as AccessibilityManager, UsageStatsManager, NotificationManager, and SensorManager, which are system level services provided by Android. An instance of each object can be obtained through the getSystemService() function [92], and specific data can be accessed via each instance. Table 2 shows constant parameter values input in the process of

4 A PREPRINT -APRIL 26, 2021

Table 1: List of existing studies that investigated prior work that employed mobile usage and sensor data Number of Surveyed Ref. Purpose of Review Reviewed Items Type of Investigated Data Studies Review the papers to gain a more generalizable Number of participants, experiment duration, Time, communication, mobility, battery, screen, Church et al. [20] 22 understanding of mobile device use across deployment, recruitment, incentives, method, notiﬁcations, multimedia, applications, locations, touch different user populations data collected, user input Acceleration, activity, app usage, battery, charge, Review the papers using sensor data for Utilized data, experiment duration, browser history, Bluetooth, call logs, camera events, Rooksby et al. [38] 19 monitoring student mental health number of participants screen, keyboard, UI, location, light sensor, microphone, SMS/email Camera, microphone, acceleration/gyro, barometer, Review the papers using mobile and touchscreen, geoposition, device use, ECG, PPG, Kortis et al. [3] 62 wearable device data for tracking Utilized data, sense domain, metrics IR thermometer, ballistocardiography, galvanic skin response, the digital biomarker for Alzheimer’s disease ambient light sensor, UV sensor, electromyogram (EMG) Review the papers using experience Experiment duration, trigger, response rate, ESM, location, phone calls, network related, bio-signal (s), Berkel et al. [89] 110 sampling method (ESM) on mobile devices compensate, personal device, utilized data accelerometer, app usage, other (s) • Data collection strategies (data, strategy, pros, cons) • Physical data: facial expressions, behavioral data, • List of affect lexicons vocal data, context data (e.g., location, timestamp) (category, lexicon, affect type, vocabulary) • Cyber data: online behaviors, social media data Review the papers using various data approximately • Applications for mental health • Biological data: blood oxygen saturation, blood pressure, Liang et al. [4] including smartphone data 92 (type, name, symptom, data) heartbeat rate, respiration frequency, brain waves in digital phenotyping for mental health • Affect recognition • Social data: call logs, text messages, GPS location, (modality, category, work, algorithm) hot spots, email, friendship list, online activities • Behavior anomaly detection (e.g., like, comment, sharing) (category, sensor, work, algorithm)

Table 2: Macro constants for service request according to Android Manager type [92] Constants: Requested Service Managers: Returned Object Description of Managers ACCESSIBILITY_SERVICE AccessibilityManager Provides user feedback on UI events through registered event listeners. USAGE_STATS_SERVICE UsageStatsManager Provides to the information of app and system usage history and statistics. NOTIFICATION_SERVICE NotiﬁcationManager Access to information for the notiﬁcation events to users that happen in the system and app. SENSOR_SERVICE SensorManager Access to the sensors related information in Android device. TELEPHONY_SERVICE TelephonyManager Access to information about telephone communication services. LOCATION_SERVICE LocationManager Access to information about location based services (LBS). AUDIO_SERVICE AudioManager Access to information about volume and ringer mode control. BLUETOOTH_SERVICE BluetoothManager Obtain the BluetoothAdapter resources and manage overall Bluetooth. WIFI_SERVICE WIFIMananager Manage the overall Wi-Fi network information.

obtaining an instance of each object and describes the Android API framework’s manager objects that can be accessed through each instance. Furthermore, there is another way to collect and use mobile usage and sensor data besides accessing the manager in third-party apps. First, BroadcastReceiver can be used to collect and track system status events and information (e.g., screen status, power status, battery status) occurring in Android OS. In addition, Bluetooth status information can be collected using only Bluetooth APIs17 such as BluetoothAdapter18 and BluetoothDevice19 without going through BluetoothManager20 [93]. In the case of location information, Google recommends that the use of the Google Location Services APIs21 is a simpler method for higher accuracy than the Location APIs22 that provide existing Android location-based and related services such as LocationManager23 and LocationProvider24 [94]. In some cases of UI event collection or third-party app development through AS API, AccessibilityService and AccessibilityEvent are only used except for AccessabilityManager (detailed explanation in Section 2.3).

2.3 Application Usage and Interaction Patterns API Framework

As shown in Figure 1, we present an overview of the main functions, history, and collectible data of AS/US APIs. The purpose of this overview is to clarify the Android APIs used for app usage pattern data collection and derive the appropriate API’s terms for the literature search terms.

17https://developer.android.com/reference/android/location/package-summary 18https://developer.android.com/reference/android/bluetooth/BluetoothAdapter 19https://developer.android.com/reference/android/bluetooth/BluetoothDevice 20https://developer.android.com/reference/android/bluetooth/BluetoothManager 21https://developers.google.com/android/reference/com/google/android/gms/location/package-summary 22https://developer.android.com/reference/android/location/package-summary 23https://developer.android.com/reference/android/location/LocationManager 24https://developer.android.com/reference/android/location/LocationProvider

5 A PREPRINT -APRIL 26, 2021

2.3.1 Accessibility APIs AS API is mainly used in existing research to collect fine-grained data related to the user interaction information (e.g., time/types of interaction, app execution status) according to user interface changes. “The ability for you to build and deploy accessibility services was introduced with Android 1.6 (API Level 4) and received significant improvements with Android 4.0 (API Level 14)” according to Android developers documentation [95]. Further, “Accessibility services should only be used to assist users with disabilities (e.g., visually impaired people) in using Android devices and apps” as per Android developers documentation [9]. In addition, Google Android provides an application made with standard AS API (e.g., TalkBack), which is an assistive technology with functions such as screen touch, voice, Braille-based screen reader, speaker-based voice, and vibration [96]. The collection of fine-grained data (i.e., UI information via user interaction) is possible by grasping the Accessibili- tyEvent delivered to the AS API through the onAccessibilityEvent() callback method when the state of UI changes. The AccessibilityEvent mainly appears when a remarkable event occurs in the UI owing to a touch manipulation, notification, or system change. First, to use the AS API in the Android third-party app, the service element should be included in the application element of the manifest, and the intent filter of the AS API must be also included in the service element. Next, the BIND_ACCESSIBILITY_SERVICE permission must be added to protect this service through the manifest. The AS API can inform the system how and when to execute the AS API through a configuration variable setting. For example, the developer can designate the AccessibilityEvent type and package name to be processed by the AS API. Therefore, when UI events occur in Android system and app, the fine-grained data can be collected through AS API are as follows: Types of touch and gesture interaction (e.g., click, long click, scroll, typing), time of interaction (e.g., interaction start/end time), view hierarchy construct, view elements, notification state change, and windows state change. Since the release of Android 4.0 (API level 14-17), functions of AS API have been significantly enhanced in terms of increasing types of UI information collection and enhancing UI interaction [95]. When the AS API was first provided to develop third-party applications in Android 1.6 (API level 4), most of AccessibilityEvent was related to types of view event (i.e., type of interaction, window state changed) based on user interaction as depicted in 3. Since Android 4.0 (API level 14-17), touch exploration mode has been added in AS API. For example, third-party apps can provide feedback (e.g., voice) on what touch contents are by detecting AccessibilityEvent (e.g., gesture detection start/end, view hover enter) generated according to all the gesture/touch interactions of users using the touch exploration mode of AS API. Further, AccessibilityEvent such as window content change, view scroll, and view text change that occur according to user interaction have been added as shown in Table 3 [97]. As the types of AccessibilityEvent increases according to the API level, the AS API can provide more useful feedback and collect UI information by user interaction. As presented in Table 3, the types of AccessibilityEvent that can be collected in Android 11 (API level 30) are largely categorized into VIEW TYPES, TRANSITION TYPES, NOTIFICATION TYPES, EXPLORATION TYPES, and MISCELLANEOUS TYPES, and there are a total of 47 types of AccessibilityEvent [97].

2.3.2 Usage Statistics APIs US API has been used to identify the application and device usage patterns along with AS API in many studies. Prior to Android 5.0 (API level 21), ActivityManager was used to obtain information about the currently running foreground application. However, as the method of getting the currently running application information using ActivityManager’s methods (e.g., getRunningTask(), getRecentTasks(), getRunningServices()) have been deprecated since Android 5.0 (API level 21) [98]. Instead of ActivityManager, the US API is currently used mainly to obtain device and application usage history and statistics information [10]. The US API includes classes as following:

• Classes to access the app and device usage history and statistics information: UsageStatsManager, Us- ageEvents, UsageEvents.Event, UsageStats • Classes to access the network usage history and statistics: NetworkStatsManager25, NetworkStats26 • Classes to access the storage statistics information such as UID, package, UserHandle on the single storage volume: ExternalStorageStats27, StorageStats28, StorageStatsManager29 • Classes to access the device conﬁguration usage statistics information: ConﬁgurationStats30

25https://developer.android.com/reference/android/app/usage/NetworkStatsManager 26https://developer.android.com/reference/android/app/usage/NetworkStats 27https://developer.android.com/reference/android/app/usage/ExternalStorageStats 28https://developer.android.com/reference/android/app/usage/StorageStats 29https://developer.android.com/reference/android/app/usage/StorageStatsManager 30https://developer.android.com/reference/android/app/usage/ConﬁgurationStats

6 A PREPRINT -APRIL 26, 2021

Table 3: Types of AccessibilityEvent [97] Added Event Types Event Event Description API level TYPE_VIEW_CLICKED “The events of clicking on a View like Button, CompoundButton, etc.” 4 TYPE_VIEW_LONG_CLICKED “The events of long clicking on a View like Button, CompoundButton, etc” 4 TYPE_VIEW_SELECTED “The event of selecting an item in context of an AdapterView” 4 VIEW TYPE_VIEW_FOCUSED “The event of setting input focus of a View” 4 TYPE_VIEW_TEXT_CHANGED “The event of changing the text of an EditText” 4 TYPES TYPE_VIEW_TEXT_SELECTION_CHANGED “The event of changing the selection in an EditText” 14 TYPE_VIEW_TEXT_TRAVERSED_AT_MOVEMENT_GRANULARITY “The event of traversing the text of a view at a given movement granularity” 16 TYPE_VIEW_SCROLLED “The event of scrolling a view” 14 TYPE_VIEW_CONTEXT_CLICKED “The event of a context click on a view” 23

TYPE_WINDOW_STATE_CHANGED “The event of change to a visually distinct section of the user interface” 4 “The event of changing content of a window and Event: 14 TRANSITION TYPE_WINDOW_CONTENT_CHANGED Sub Event: 19, 28, 30 TYPES more speciﬁcally the sub-tree rooted at the event’s source” Event: 21 TYPE_WINDOW_CHANGED “The event change in the system windows shown on the screen” Sub Event: 28 NOTIFICATION TYPE_NOTIFICATION_STATE_CHANGED “The event showing a notiﬁcation” 4 TYPES

TYPE_VIEW_HOVER_ENTER “The event of a hover enter over a view” 14 TYPE_VIEW_HOVER_EXIT “The event of a hover exit over a view” 14 TYPE_TOUCH_INTERACTION_START “The event of the user starting to touch the screen” 17 EXPLORATION TYPE_TOUCH_INTERACTION_END “The event of the user ending to touch the screen” 17 TYPES TYPE_TOUCH_EXPLORATION_GESTURE_START “The event of starting a touch exploration gesture” 14 TYPE_TOUCH_EXPLORATION_GESTURE_END “The event of ending a touch exploration gesture” 14 TYPE_GESTURE_DETECTION_START “The event of beginning gesture detection” 17 TYPE_GESTURE_DETECTION_END “The event of ending gesture detection” 17 MICELLANEOUS TYPE_ANNOUNCEMENT “The event of an application making an announcement” 16 TYPES

TYPE_VIEW_ACCESSIBILITY_FOCUS “The event of gaining accessibility focus” 16 TYPE_VIEW_ACCESSIBILITY_FOCUS_CLEARED “The event of clearing accessibility focus” 16 ETC TYPE_ALL_MASK “Mask for AccessibilityEvent all types” 4 TYPE_ASSIST_READING_CONTEXT “The event of the assistant currently reading the users screen context” 23 INVALID_POSITION “Invalid selection/focus position” 4

• Classes to event type usage statistics information: EventStats

Among the many classes in the US API above, we focus the classes that can access app/device usage history and statistics information for our review research. UsageStatsManager can query application records. For example, by using the usage statistics constants (i.e., INTERVAL DAILY, INTERVAL WEEKLY, INTERVAL MONTHLY, and INTERVAL YEARLY), we can query the records to receive the classified results according to different time periods (i.e., by the day, the week, the month, and the year). In addition, app usage time (e.g., last time used, app foreground/background used time), app usage status history (e.g., status of foreground/background/user interaction), and package name can be returned as the UsageStats object provided in the UsageEvents.Event class through methods such as getPackageName(), getLastTimeUsed(), and getTotalTimeInForegroud(). Furthermore, five types of bucket information (active, working set, frequent, rare, never) which defines the usage status for each app according to how often the app was used recently can be obtained through UsageStatsManager since Android 10 (API level 29) [11]. Various Android runtime information can be collected along with various application pattern information in US API. In addition to the information that can be accessed via using the US API mentioned above, Table 4 shows the detailed information that can be collected through UsageStatsManager and UsageEvents.Event.

2.3.3 Notiﬁcation API framework

TYPE_NOTIFICATION_STATE_CHANGED of AccessibilityEvent in AS API can detect the notification information (e.g., time, app package name, displayed texts of the toasts such as small popup, and text information) notified through getEventType(), getPackageName(), getEventTime(), getParcelableData(), and getText() methods. However, there is a limit to the information that can be accessed through AS API. NotificationListenerService31 (launched Android API level 21) and NotificationManager32 (launched Android API level 1) can collect the more specific notification information (e.g., notification app name, time, status, priority, text, category, sound, visibility). Currently, notification related Android APIs such as NotificationListenerService and NotificationManager are widely used along with AS API to study notification usage behaviors in existing studies (e.g., [17, 15, 100, 25, 14, 32]).

31https://developer.android.com/reference/android/service/notification/NotificationListenerService 32https://developer.android.com/reference/android/app/NotificationManager

7 A PREPRINT -APRIL 26, 2021

Table 4: Constants of UsageStatsManager [11] and Events of UsageEvents.Events [99] Android Class Constants and Events Description API Level INTERVAL_BEST "An interval type that will use the best ﬁt interval for the given time range." 21 INTERVAL_DAILY "An interval type that spans a day." 21 INTERVAL_MONTHLY "An interval type that spans a month." 21 UsageStatsManager INTERVAL_WEEKLY "An interval type that spans a week." 21 INTERVAL_YEARLY "An interval type that spans a year." 21 STANDBY_BUCKET_ACTIVE "The app was used very recently, currently in use or likely to be used very soon." 28 STANDBY_BUCKET_FREQUENT "The app was used in the last few days and/ or likely to be used in the next few days." 28 STANDBY_BUCKET_RARE "The app has not be used for several days and/or is unlikely to be used for several days." 28 STANDBY_BUCKET_RESTRICTED "The app has not be used for several days, is unlikely to be used for several days, and has been misbehaving in some manner." 30 STANDBY_BUCKET_WORKING_SET "The app was used recently and/or likely to be used in the next few hours." 28 ACTIVITY_PAUSED "An event type denoting that an Activity moved to the background" 29 ACTIVITY_RESUMED "An event type denoting that an Activity moved to the foreground" 29 ACTIVITY_STOPPED "An activity becomes invisible on the UI, corresponding to Activity.onStop() of the activity’s lifecycle" 29 CONFIGURATION_CHANGE "An event type denoting that the device conﬁguration has changed." 21 DEVICE_SHUTDOWN "An event type denoting that the Android runtime underwent a shutdown process." 29 DEVICE_STARTUP "An event type denoting that the Android runtime started up." 29 UsageEvents.Event FOREGROUND_SERVICE_START "An event type denoting start of a foreground service." 29 FOREGROUND_SERVICE_STOP "An event type denoting stop of a foreground service." 29 KEYGUARD_HIDDEN "An event type denoting that the screen’s keyguard has been hidden." 28 KEYGUARD_SHOWN "An event type denoting that the screen’s keyguard has been shown, whether or not the screen is off." 28 MOVE_TO_BACKGROUND "This constant was deprecated in API level 29. by ACTIVITY_PAUSED" 21 MOVE_TO_FOREGROUND "This constant was deprecated in API level 29. by ACTIVITY_RESUMED" 21 NONE "No event type." 21 SCREEN_INTERACTIVE "An event type denoting that the screen has gone into an interactive state." 28 SCREEN_NON_INTERACTIVE "An event type denoting that the screen has gone into a non-interactive state." 28 SHORTCUT_INVOCATION An event type denoting that an action equivalent to a ShortcutInfo is taken by the user. 25 STANDBY_BUCKET_CHANGED An event type denoting a change in App Standby Bucket. 28 USER_INTERACTION An event type denoting that a package was interacted with in some way by the user. 23

2.3.4 Call and Message API framework

Existing studies have collected the call, short message service (SMS), multimedia message service (MMS) related app usage history/status and sending/receiving notification information through AS/US API (e.g., [24, 34, 16, 21, 101, 102, 103, 104, 36]). However, AS/US API can access only the call/message related app usage history and status information. Therefore, telephony APIs33 (e.g., TelephonyManager34, PhoneStateListener35, SmsManager36) or CallLog37 can be mainly used to access relatively fine-grained and accurate information compared to AS/US API through monitoring of call and message status information. TelephonyManager can check the access and status of information about the device’s telephony service (e.g., SMS, MMS, and call). In addition, third-party apps can use the methods of TelephonyManager to check telephony services/status, access some types of subscriber information, and register listeners to receive phone status change notifications. CallLog can collect call-related data, and through this, incoming/outcoming history information can be also collected. Therefore, telephony APIs (e.g., TelephonyManager, CallLog) can be widely used when collecting relatively fine-grained call and message related information (e.g., type of phone, incoming/outcoming call/SMS/MMS, call/SMS/MMS response state such as missed, rejected, blocked, answered, type of contact, phone service change, cell location change, mobile signal, phone state change) compared to AS/US API in many studies.

2.4 Context-Aware Android API Framework

Android APIs mainly used for context-aware data collection are classiﬁed according to the surrounding context circumstances as follows: motion, position, environmental context APIs, location context APIs, and network context APIs.

33https://developer.android.com/reference/android/telephony/package-summary 34https://developer.android.com/reference/android/telephony/TelephonyManager 35https://developer.android.com/reference/android/telephony/PhoneStateListener 36https://developer.android.com/reference/android/telephony/SmsManager 37https://developer.android.com/reference/android/provider/CallLog

8 A PREPRINT -APRIL 26, 2021

2.4.1 Motion, Position, and Ambient Environment Context APIs Representative Android API frameworks used to collect motion, position, and ambient environment context information include HW sensor-based sensor APIs such as SensorManager38, Sensor39, SensorEvent40, and SensorEventListener41 [105]. HW sensors are speciﬁcally classiﬁed into motion detection sensors (e.g., accelerometers, magnetometers, gyroscope), physical position sensors (e.g., accelerometers, magnetometer, proximity), and ambient environment sensors (e.g., light, pressure, humidity, temperature) [106]. Based on three types of sensors and sensor APIs, motion, position, and ambient environment context information that can be collected in smartphones as follows:

• Motion context information by motion detection sensors: Acceleration, gravity, rate of rotation, rotation vector, signiﬁcant motion, step counter, and step detector data using motion detection • Position context information by physical position sensors: Physical position of a device, the distance between the device and an object • Ambient environment context information by environment sensors: Device/battery temperature, light, pressure, ambient air pressure, and ambient relative humidity/temperature

The context information based on mobile sensors and sensor APIs mentioned above are collected and utilized in many existing studies (e.g., [32, 25, 29, 87, 36, 107]). In addition, the Google Activity Recognition API42 identifies the activity (i.e., walking, running, driving, standing still, cycling) is being performed by the user at each time with the sensors in the smartphone device through detecting a user’s specific activity type constants (i.e., IN_VEHICLE, ON_BYCYLE, ON_FOOT, RUNNING, STILL, TILTING, WALKING) [108]. Furthermore, the Google Activity Recognition Transition API43 can detect a user’s specific activity type constants (i.e., IN_VEHICLE, ON_BYCYLE, RUNNING, STILL, WALKING) to identify when a user starts or stops a specific activity [109]. Therefore, through mobile sensors and Google Activity Recognition APIs mentioned above, the physical activity state information are currently collected and utilized in many existing studies (e.g., [34, 16, 103, 104]).

2.4.2 Location Context APIs Location APIs (e.g., LocationManager, LocationProvider) and Google Location Services API are representative API frameworks mainly used to collect location context information in many existing studies (e.g., [20, 17, 101, 110, 102, 37, 103, 104, 23, 16, 15, 27, 68]). In terms of efﬁciency and accuracy, Google Location Services APIs are superior to the Location APIs. Through Google Location Services APIs, location information such as timestamp, latitude, longitude, altitude, accuracy, and speed of location data can be collected. If the Google Location Services API is not available, the location information can be collected via the Location APIs (e.g., LocationProvider and LocationManager) in the traditional way as follows: (1) GPS location provider (GPS, AGPS), network location provider (GPS, Cell ID, Wi-Fi MAC ID), and the authority (android.permission.ACCESS_FINE_LOCATION or android.permission.ACCESS_COARSE_LOCATION) receive information from the network location provider (GPS, Cell ID, Wi-Fi MAC ID) and passive location provider (Cell ID, Wi-Fi MAC ID) are set in AndroidManifest xml. (2) After setting the authority, set the provider to be used by LocationManager, and get updated location information from the GPS and network through the requestLocationUpdates() method in the LocationManager. It is also possible for a third-party app to collect the location provider and information (e.g., from the LocationManager in real time through LocationListener) [111].

2.4.3 Personal & Local Area Network Context APIs Smartphone’s network context information can be largely divided into personal area network (PAN) and local area network (LAN) information. In the current Android environment, Android’s built-in Bluetooth sensor and APIs (including Bluetooth low energy; BLE) such as BluetoothManager, BluetoothAdapter and BluetoothDevice are mainly used to collect PAN usage data (e.g., device name, type, address, and connection status) [112]. The Android PAN usage data can be collected in real-time using these Android built-in Bluetooth sensors and APIs typically including the connected information with nearby devices. Bluetooth scanning classes [113] such as BluetoothLeScanner44 can be used to scan nearby BLE devices that can be connected. LAN information can be collected using Android built-in

38https://developer.android.com/reference/android/hardware/SensorManager 39https://developer.android.com/reference/android/hardware/Sensor 40https://developer.android.com/reference/android/hardware/SensorEvent 41https://developer.android.com/reference/android/hardware/SensorEventListener 42https://developers.google.com/android/reference/com/google/android/gms/location/ActivityRecognitionClient 43https://developers.google.com/android/reference/com/google/android/gms/location/ActivityTransition 44https://developer.android.com/reference/android/bluetooth/le/BluetoothLeScanner

9 A PREPRINT -APRIL 26, 2021

network sensors (e.g., Wi-Fi) and network APIs such as WifiManager45 and ConnectivityManager46; e.g., Service Set Identifier (SSID), Basic Service Set Identifiers (BSSID), Received Signal Strength Indicator (RSSI), frequency, and the presence of connectable Wi-Fi networks can be detected in real-time (known as Wi-Fi scanning). ConnectivityManager can be utilized to collect the network connection state information. In addition, NetworkStatsManager and TrafficStats can be used to obtain the network traffic usage statistics and history information (e.g., transmitted and received information of the network packets and bytes from all interfaces, mobile, and UID) [114, 115].

2.5 Device and System Status APIs

Android device and system status information can be obtained by tracking events that occur according to real time operation states (e.g., battery states, power on/off, network connection states) by using device and system status APIs. Currently, there are two main methods used to obtain device and system status information in Android OS. The ﬁrst method is to register BroadcastReceiver and selectively access device/system events using IntentFilter47 to receive device/system status information and resources occurring in Android [116]. When a speciﬁc device system status change occurs in the Android system, event information (e.g., ACTION_AIRPLANE_MODE_CHANGED, ACTION_BATTERY_CHANGED/LOW/OKAY, ACTION_BOOT_COMPLETED) is delivered in the form of an intent through the BroadcastReceiver [117]. At this time, third-party app can access the detailed device system information (e.g., battery status change, battery charge status such as low, high, system booting completion, bug reporting, phone connection, call button press, date change, system reboot, and device system status information). The second method is to access the manager (e.g., BatteryManager, AlarmManager48, PowerManager49, AudioManager50) related to the device system by using the getSystemService() method. For example, the BatteryManager can broadcast all battery and charge details to a sticky Intent containing the state of charge without registering the BroadcastReceiver [118]. Through BatteryManager, third-party apps can access the device battery states information such as battery health (cold, dead, good, overheat, over voltage), battery plugged (AC, USB, wireless), battery property (e.g., capacity, charge counter), battery status (charging, discharging, full, not charging).

2.6 Discussion of Android APIs

Among the Android APIs that can collect mobile usage and sensor data through a smartphone, as discussed above, AS/US APIs are the representative APIs that can elucidate the smartphone usage patterns well. In particular, AS API collects information on the current app usage states, screen states, interaction type/time/target (UI hierarchy & elements), and notification states through four major accessibility events (VIEW TYPES, TRANSITION TYPES, NOTIFICATION TYPES, EXPLORATION TYPES), presented in Table 3. Using the US API, it is possible to grasp app and device usage patterns of not only third-party apps, but also device system-related applications. For example, it is possible to check records such as call/SMS usage time by using US API without using other APIs such as TelephonyManager or CallLog. In addition, using the US API allows to collect device system states such as configuration change, user interaction, device shutdown/startup, and screen interactive/non-interactive states. For this reason, AS/US APIs are often used more frequently than other Android APIs in most studies. Therefore, this article aims to review the studies using AS/US APIs. As a result, we describe the search conditions, combined keywords results, filtering methods, and search results for extracting and reviewing the studies that used AS/US APIs in Section 3.

3 Methodology

3.1 Literature Search

We created a keyword list and searched using those terms, to ﬁnd studies in which AS/US APIs were used, on four scholarly literature search engines: Google Scholar51, Web of Science52, Scopus53, and ScienceDirect54. Because AS/US APIs are used in various research ﬁelds, we searched for papers in the general literature search engines, in which

45https://developer.android.com/reference/android/net/wiﬁ/WiﬁManager 46https://developer.android.com/reference/android/net/ConnectivityManager 47https://developer.android.com/reference/android/content/IntentFilter 48https://developer.android.com/reference/android/app/AlarmManager 49https://developer.android.com/reference/android/os/PowerManager?hl=en 50https://developer.android.com/reference/android/media/AudioManager 51https://scholar.google.co.kr/ 52http://www.webofknowledge.com/ 53https://www.scopus.com/ 54https://www.sciencedirect.com/

10 A PREPRINT -APRIL 26, 2021

Table 5: List of Android, Accessibility APIs, Usage Statistics API, and combined keywords List Keywords Android-related “android" “accessibility"; “accessibility service"; “accessibilityservice"; Accessibility APIs-related “accessibility API"; “accessibility framework"; “accessibility event"; “accessibilityevent” Usage Statistic API-related “usagestats"; “usagestatsmanager"; “usageevent” “android" and "accessibility"; “android” and "accessibility service"; “android” and "accessibilityservice"; “android” and “accessibilityevent"; Combined “android” and “accessibility event"; “android” and “accessibility API"; “android” and “accessibility framework"; “android” and “usagestats”

papers from different fields can be found, instead of the web library databases of specific journals and proceedings. These scholarly literature search engines contain relevance-based ranking systems and citation counts, which helped us to consider the importance of each paper. In the case of Web of Science, Scopus, and ScienceDirect, we could extract a list of specific paper information (e.g., title, abstract, publication site, publication date, download link) in different formats (e.g., *.txt, *.csv, *.BibTeX). In addition, we could download the literatures in the PDF format from the Web of Science, Scopus, and ScienceDirect. However, it was not possible to extract the above information as a list from Google Scholar, and patents were also included in the Google Scholar list. Therefore, we obtained a list of papers (excluding patents) from Google Scholar through web crawling under the conditions mentioned above in this study. We divided the keywords to search papers into four lists: Android related, Accessibility APIs related, Usage Statistic API related, and combined keywords. This keyword list was composed of terms that are frequently used in existing studies. Table 5 presents the list of the keywords used to search the papers. We selected the start year of the literature search period to be the year when the AS/US APIs were first made available in the Android Framework. AS API became available in Android 1.6 (API level 4) in September 2009, and US API became available in the Android Framework from Android 5.0 (API level 21) in October 2014. Therefore, we searched published papers using AS/US APIs for the period from 2009 to 2020 (present year). The literature search keywords, period, language, and sites are summarized in Figure 2.

3.2 Literature Filtering and Classiﬁcation

The studies were classified into two main categories: (1) studies regarding the development of applications or frameworks for collecting a fine-grained dataset using the AS/US APIs, and (2) studies that collected fine-grained datasets using the AS/US APIs. As shown in Figure 2, we used the customized Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [119] for our review to visualize the filtering and classification process for finding target papers. In addition, the classification results according to the four filtering criteria were divided into four categories of included papers (IP) to be reviewed and four categories of excluded papers (EP) to be excluded in our survey scope. We developed a classification system for the type of research and defined the type of research to filter out only the target papers using the customized flow diagram by PRISMA technique. Furthermore, we defined the most important aspects to be considered while filtering the studies as four filtering criteria: (Filtering criteria 1) studies corresponding to journals or proceedings, (Filtering criteria 2) studies that mentions the Android AS API or US API in the mobile environment, (Filtering criteria 3) studies using the AS API to enhance interface accessibility for people with disabilities, (Filtering criteria 4) studies using the AS or US API for research purposes other than improving accessibility of interfaces for people with disabilities Filtering criteria 1. Journals or Proceedings: For this criterion, we focused only on the studies from journals and proceedings (i.e., we excluded other types of sources such as patents, e-magazines, books, and dissertations). As a result of searching with the search keywords, a total of 1,724 papers were found in Google Scholar (1,513 papers), Web of Science (16 papers), Scopus (141 papers), and ScienceDirect (54 papers). 880 papers were extracted, excluding the papers written by non-English and duplicate results by each search engine. 475 papers were categorized as EP1, as they did not belong to the category of journals and proceedings papers.

11 A PREPRINT -APRIL 26, 2021

Select databases: Google Scholar, Scopus, ScienceDirect, Web of Science

Initial query keyword: (“android accessibility”) or (android and “accessibility service”) or (android and “accessibilityservice”) or (android and “accessibilityevent”) or (android and “accessibility event”) or (android and “accessibility API”) or (android and “accessibility framework”) (“usagestats”) or ("usagestatsmanager") or ("usageevents")

Identification Search area: All fields, Search language: English Search year: 2010 ~ 2020

Records through database Search (n = 1,724) Google Scholar: 1513, Scopus: 141, ScienceDirect: 54, Web of Science: 16)

Records after duplicates and duplicate and non- English removed (n = 880)

No Record after not conference paper or journal removed EP1 (n = 405 ) n = 475

Yes

No Record after not Mention about Android Accessibility API EP2 or Usage Statistics API screening (n = 258) n = 147

Screening Yes

Record after not purpose of enhancement interface for the users with disabilities (n = 182)

Yes No

No No EP4 Used Accessibility API EP3 n = 92 or Used Accessibility API n = 56 Usage Statistics API

Yes Yes

IP1 IP2 n = 90 n = 20

User studies User studies Included Yes No Yes No

IP1-1 IP1-2 IP2-1 IP2-2 n = 60 n = 30 n = 17 n = 3

Figure 2: Flow diagram for ﬁltering and classiﬁcation12 of target papers based on PRISMA technique A PREPRINT -APRIL 26, 2021

Table 6: Descriptions of meaning of IP Types of IP Descriptions Number of Studies IP1-1-AS Research that perform user studies on AS API for improving the interface for the people with disability 44* IP1-1-US Research that perform user studies on US API 17* Research that did not perform user studies to improve the interface IP1-2-AS 28* for the people with disability among the studies using AS API IP1-2-US Research that did not perform user study among studies using US API 3* Among the studies using the AS API, the studies where user studies were conducted IP2-1 17 for the purpose of research to improve the interface for the people with disability Among the studies using the AS API, the studies where user study was not conducted IP2-2 3 for the purpose of research to improve the interface for the people with disability Overall 112-2*=110 * Bonné et al. [17] and Ryu et al. [131], which is used AS/US APIs both, were included in IP1-1-AS/US and IP1-2-AS/US twice, respectively. Therefore, the number of surveyed studies in this paper is 110, excluding the two out of a total of 112 overalls.

Filtering criteria 2. Mentions the Android AS or US API in the mobile environment: We focused only on the studies using AS/US APIs in the Android OS of the mobile environment. Despite the search keywords "Android”, “Accessibility Service”, and “Accessibility API”, there were many cases where related to “Accessibility” was mentioned in the PC or iOS environment. Therefore, studies that do not mention AS API or US API in the Android mobile environment were defined as EP2 (147 papers). Filtering criteria 3. Studies using AS API to enhance interface accessibility for people with disabilities: We distin- guished the existing studies using the AS to improve the interface accessibility of the users with disabilities (the original purpose of the Google Android policy) from papers which focused on AS API use for other research purposes. Studies that were conducted to enhance interactions for the users with disabilities using the AS API were classified as IP2, and 56 papers that did not use the AS API were classified as EP3. Among the studies corresponding to IP2, studies that collected smartphone data or conducted a questionnaire through user study was classified as IP2-1, and studies that did not perform user studies were classified as IP2-2. Filtering criteria 4. Studies using AS or US API for research purposes other than improving accessibility of interfaces for people with disabilities: Finally, existing studies that mention terms related to Android AS/US APIs (i.e., AccessibilityEvent, AccessibilityManager, Accessibility API, AccessibilityService, UsageStats, UsageStatsManager, UsageEvent) in the paper, or use AS/US APIs for research purposes rather than for interface accessibility enhancement of the users with disabilities were classified as IP1 or EP4, respectively. Among the studies, we classified existing studies that actually used smartphone data or developed an application to collect smartphone data using AS/US APIs as IP1, and other existing studies (e.g., review papers, technical papers) were defined as EP4 (92 papers). The IP1 studies that collected data or conducted a questionnaire through a user study were classified as IP1-1. The studies were classified as IP1-2 if they collected data through other methods (e.g., developer testing, crawling, and crowdsourcing) than user studies or if only a framework/application was developed by AS/US APIs for the smartphone data collection purpose and no smartphone data was collected. Furthermore, among the studies belonging to IP1-1 and IP1-2, studies using AS API were classified as IP1-1-AS and IP1-2-AS, while studies utilizing US API were classified as IP1-1-US and IP1-2-US. Bonné et al. [56] and Ryu et al. [62] used AS/US APIs both. Hence, they were included in IP1-1-AS/US and IP1-2-AS/US twice, respectively. The studies that were classified as IP1 and IP2 were considered to lie within the scope of the review. Through a review of these studies, we identified the research purpose of the studies, the types of AS/US APIs data used, and other smartphone data that used by other Android APIs. Table 6 shows the description and number of studies according to types of IP (IP1-1-AS, IP1-1-US, IP1-2-AS, IP1-2-US, IP2-1, IP2-2) which are the studies included in the scope of review. In this review, a total of 110 studies were investigated, as two studies [56, 62] were counted twice.

4 Research Purpose Categorization

We categorized the major research purposes of the included studies as IP1 (i.e., studies that used AS or US API other than accessibility enhancement purposes for those with disabilities) in Section 4.1 and IP2 (i.e., studies that used AS or US API of accessibility enhancement purposes for those with disabilities) in Section 4.2 (See Table 6 for the detailed types of included paper). For categorization, we used afﬁnity diagramming, a grouping technique that discovers meaningful rules among numerous data for classiﬁcation according to the research purpose [120]. The process of

13 A PREPRINT -APRIL 26, 2021

categorizing the research objectives of several papers using the afﬁnity diagramming technique is as follows. First, we extracted the words of related research purposes in the order of title, abstract, keyword, main text, and published site name of each paper. Among the words extracted from the reviewed papers, highly relevant words were grouped together, and the words that were duplicated or unrelated to the purpose of the research were removed. Then, the research purposes representing the characteristics of the grouped words were found. The groups were then adjusted (e.g., groups of the same concept were merged, large groups were divided, small groups were merged with other groups) and named according to the research theme, and the themes were then divided into sub-themes. While reviewing the completed diagram from the beginning, we corrected the themes.

4.1 Categorization of Research Purposes for People without Disabilities

We first identify research purposes of IP1 studies; a total of 90 papers belong to IP1. Table 7 shows the research purposes of each paper, which were classified into five major themes via affinity diagramming: (1) usage pattern (UP), (2) notification (NT), (3) enhancement of user interaction and experience (EUI/EUX), (4) privacy and security (PS), and (5) programming and testing support (PTS). In addition, the five major research themes were classified into 18 sub-themes. The list of papers were then classified and organized into one of the 18 sub themes depending on whether they belonged to IP1-1 or IP1-2. The description of each theme and sub-theme is as follows: Usage Pattern (UP)

• Generic Usage Pattern Analysis (GUPA): Research on the analysis of generic mobile usage patterns based on mobile data-driven analytics according to specific conditions (e.g., society, culture, education, health status, age group) of users • Machine Learning-based Model and System (MLMS): Research on machine learning models and system development to understand user characteristics based on the mobile usage pattern • Framework and System Development (FSD): Research on developing a framework and system using AS/US APIs, and other Android APIs for enabling fine-grained mobile data collection Notification (NT)

• Identifying Factors responding to Notifications (IFN): Research to understand or predict how users react to and interact with which notifications in what context • Notification Timing and Notification Management (NT/NM): Screening research to reduce interruption by notification, just in time notification scheduling research, and research on notification management according to the importance • How Notification Affects a User (HNAU): Research to identify how notifications affect users (e.g., attentiveness, emotion, action, responsiveness, interruptible, learning) Enhancement of UI and UX (EUI/UX)

• Computational Enhancement (CE): Research for user interaction and experience improvement (e.g., upgrad- ing system performance, lip recognition) through computational enhancement (e.g., automation, recognition, offloading, microservice) • User Interface Enhancement (UIE): Research for improving UI (e.g., personalized conversation interface, conversation-aware interface, verification by password-less access, multi-modal interface) • User Interface Profiling (UIP): Research on UI profiling such as complexity measurement of the UI or power management of CPU/GPU Privacy and Security (PS)

• Authentication System/Scheme (ASS): Research on the suggestion of systems/schemes for user authentication in the Android mobile environment • Access and Permission Control (APC): Research to identify user intention and context for user access and permission control or to suggest new methods • Privacy-Preserving System (PPS): Research to propose a system for preserving user privacy in Android mobile environment • Monitoring and Detection for Privacy and Security (MDPS): Research to detect and monitor spying behavior and hidden attacks in terms of personal privacy and security protection

14 A PREPRINT -APRIL 26, 2021

Table 7: Results of research purpose categorization in IP1 Theme Sub Theme Ref: IP1-1-AS Ref: IP1-1-US Ref: IP1-2-AS Ref: IP1-2-US Generic Usage Pattern Analysis (GUPA) [22, 20, 17, 21, 121] [122, 123, 124, 125, 13, 101] Machine Learning-based [18] [110, 12, 126] [19] Usage pattern (UP) Model and the system (MLMS) Framework and System [102, 127] [37, 87, 88, 103, 104, 107] Development (FSD) Identifying Factors responding [24, 23, 16] [15] to notifications (IFN) Notification (NT) Notification Timing/ [27, 30, 28, 29, 26, 25] [14, 100] Notification Management (NT/NM) How Notification Affects User (HNAU) [34, 32, 36, 33, 35, 31] Computational Enhancement (CE) [71] [70, 69, 72] Enhancement of UI/UX (EUI/UX) User Interface Enhancement (UIE) [61, 63, 64] [65, 62] [62] User Interface Profiling (UIP) [68, 66] [67] Authentication system/scheme (ASS) [52, 53, 54] [128, 129, 130] Access and permission control (APC) [56, 55] Privacy and Security (PS) Privacy-preserving system (PPS) [50, 49, 51] [131] [132] Monitor and detection for [57] [133, 60, 58, 59, 134] [135, 136] privacy and security (MDPS) GUI Automated Testing (GAT) [73] [75, 76, 74] Crowdsourced Testing (CT) [78, 77] Programming and Automated Programming Support (APS) [85, 137] [84, 86] Testing Support (PTS) Performance Measurement [79] [138] and Management (PMM) Test Recorder and [82, 83] [81, 80] Script Generator Technique (TRSGT)

Programming and Testing Support (PTS)

• GUI Automated Testing (GAT): Research to propose an automated model and system for GUI testing of Android applications • Crowdsourced Testing (CT): Research to test android applications through crowdsourcing • Automated Programming Support (APS): Research on supporting the automation of programming tasks in system development • Performance Measurement and Management (PMM): Research to measure or manage the performance of Android devices and systems • Test Recorder and Script Generator Technique (TRSGT): Research on technology to automatically generate test scripts by converting and encoding test actions into test script format via user interactions for automated testing

4.2 Categorization of Research Purposes for People with Disabilities

We identify and classify the research objectives of the IP2 studies that use the AS API for the original purpose (i.e., to improve the user interface to support the users with disabilities) [9]. IP2 can be included in the enhancement of UI and UX (EUI/UX) among the themes of the research purpose of surveyed studies corresponding to IP1. However, in the case of IP2, the sub-themes have to be classified differently than the IP1 sub-themes to provide a more differentiated insight. Accordingly, a total of 20 papers using the AS API for its original purpose have been largely classified into three sub-themes: (1) improvement of the functionality of accessibility service for the users with disabilities (IFASD), (2) development or evaluation of an application using accessibility service for the users with disabilities (DEASD), and (3) comparison of functions of accessibility service for the users with disabilities (CFASD). IFASD (Improvement of the Functionality of Accessibility Service for the users with Disabilities): The purpose of these papers is to improve the functions of the accessibility aspect of the existing AS API, and they are classified further into two sub research purposes. The first is to develop the accessibility service with improved functions in terms of accessibility enhancement by customized AS API [95]. The second is to develop improved functions to enhance the accessibility for users with disabilities by using the AS API and physical devices (e.g., braille, external sensor) together. In addition, the studies can be divided into three groups depending on the evaluation methods: (1) evaluating through user study by comparing and analyzing the existing AS API and the newly developed AS API in the paper, (2) evaluating only the developed AS API through the user study, and (3) not evaluating through the user study. Table 8

15 A PREPRINT -APRIL 26, 2021

Table 8: Analysis of research in IP2: (1) what is proposed, (2) Accessibility Service function used or improved, (3) usability test method according to there research purpose (IFASD: Improvement of the Functionality of Accessibility Service for the users with Disabilities, DEASD: Development or Evaluation of an application using Accessibility Service for the users with Disabilities, and CFASD: Comparison of Functions of Accessibility Service for the users with Disabilities) Methods of Usability Tests Implemented Functions or Utilized Theme Ref. What is Proposed Number of Disabilities of Functions in Accessibility Service Evaluation Metrics Participants Participants A low-cost auxiliary development device AI-based image recognition, collision Chinchole et al. [141] with economical efficiency, high efficiency, mobility, detection, and obstacle detection Uncounted Non-disabled - and ease of use via AI and detection sensors functions to navigate the environment System Usability Scale [143], setup time duration, An improvement of voice command to improve Voice commands improvement and Oliveira et al. [142] 6 Visually impaired qualitative questions autonomy, satisfaction, the interaction for visually impaired people used Talkback to evaluation daily use, fatigue, and feeling of being lost Manual macro creation, automatic macro Visually impaired, Macros to enhance time spent Rodriques et al. [140] generation, and macro production to perform 1 speech disorder, Interviewed user experience searching and inaccessible content with visually impaired multiple steps of interaction with a single tab motor impaired Interactive non-visual tutorials that Rodriques et al. [144] A tutorial guide for the Accessibility Service 11 Visually impaired Success rate of tasks support audio recording and launch tutorials A Hybrid-Brailler that integrate physical Input solution with combining physical Words per min, typing error rate, editing ratio, IFASD Trindade et al. [145] and touch screen devices to provide fast, accurate, 11 Visually impaired Braille keyboard and touch screen devices task completion time and flexible nonvisual input for visually impaired users Interaction proxies to runtime repair Low vision, Accessibility barriers, user experience, usefulness Zhang et al. [64] Interaction proxies 14 and accessibility improvement of app. visually impaired and potential of proposed method based on interview Capture tool, template screen tool, Robust annotations for app interface element - Screen equivalence heuristics: runtime library, alternative collection Developer, Zhang et al. [146] to solve the access problem of 5, 1 the error rate of misjudgment of two screens and annotation tools to get properties visually impaired the users with disabilities in Accessibility Service - Runtime repair based interview of each element from Accessibility Node A voice control to use with other Android apps Hands-free voice control Zhong et al. [147] -- and talkback to support the system quickly based on Accessibility Service The system-wide assistance service Touch area enhancement in Zhong et al. [139] that can solve the accessibility problem of 8 Motor-impaired Acquisition time via types of touch and trial interfaces Accessibility Service people with hand tremors Alnfiai et al. [148] The BrailleTap calculator using tap gestures Talkback’s screen reader 2 Visually impaired Average time spent for calculation the equation The voice assistant for fall detection, safety care, The volunteers "Computing time of object detection in Chen et al. [149] phone’s accessibility, daily broadcasting information, Used the Accessibility Service with "wearing a different internet environment" view description via AI and mobile computing blinder" Different types of touch input Correa et al. [150] A configurable game for education 6 Visually impaired Efficiency and satisfaction (Tap, Double tap, swipe, tow finger swipe) A learning app incorporating Talkback’s screen reader and Samonte et al. [151] haptic and a voice feedback for braille recognition Uncounted Visually impaired Success rate of tasks execution user feedback (haptic, vibration) and 3D printing for visually impaired students DEASD A electronic calendar by voice, keyboard, Interviewed user experience when participant perform Ghidini et al. [152] and touch command interaction, and identify Talkback’s screen reader 4 Visually impaired the proposed tasks what interaction is best for visually impaired A system of digital accessible information Hann et al. [153] Used and Customized Talkback - - for print disabled people A low cost and a friendly indoor Used Accessibility Service’s functions: Correctness of the navigation instructions and Ganz et al. [154] Uncounted Not visually impaired navigation system for visually impaired "explore by touch" and "hovering" the user interface by performing certain functions Survey of X-Ray’s learnability, comfort, usefulness, A system that captures and embeds the Used Accessibility Service Pareddy et al. [155] 5 Visually impaired perceived speed, perceived accuracy, meaning of basic contents as images for visually impaired. to collect UI hierarchy satisfaction with Likert scale of 1 to 7 Finding the best method among existing user interface Interface reaction time, commands correctness, Dobosz et al. [156] Talkback’s screen reader 11 Visually impaired approaches for playing mobile games screen size influence when using mobile game Interviewed the main concerns, observations, challenges, A method to investigate smartphone adoption Used Talkback to understand device usage CFASD Rodriques et al. [157] 5 Visually impaired barriers, and experiences about smartphones, and usage by beginner users with visual impairments and interaction observation device usage, and interaction observation Tools to evaluate Android accessibility’s Used the Accessibility Service to monitor the Conduct survey of current practices, challenges, Alshyban et al. [158] 66 Practitioners capabilities in terms of apps, developers, and users AccessibilityEvent when each app is crawled accessibility tools, and guidelines related to accessibility

summarizes studies in IFASD: what is proposed, what system or app is implemented compared to the basic Android AS API in terms of accessibility enhancement, and how usability tests were performed. DEASD (Development or Evaluation of an application using Accessibility Service for the users with Disabilities): These studies were conducted for the purpose of developing an application program (e.g., calculator, configurable game, 3D printing) using the AS API rather than improving the limitations of the existing AS API. The research purpose of these papers is further divided into two subcategories. The first sub research purpose is to enable TalkBack for verification of interactions with developed applications. In the second sub research purpose, AS API, such as TalkBack, is enabled and used not only for interaction purposes for the developed application programs, but also for application development as its own AS API. Table 8 summarizes studies in DEASD: what is the research purpose, what functions of the AS API were used, and how usability tests were performed about studies in DEASD. CFASD (Comparison of Functions of Accessibility Service for the users with Disabilities): In studies with this research purpose the existing AS API (e.g., TalkBack) is used to find the best interaction method for users with disabilities. The ultimate purpose of these papers, which corresponds to the CFASD, is to enhance smartphone use for users with disabilities. Most studies were conducted to solve the accessibility issues in smartphone utilization for visually impaired people. By contrast, Zhong et al. [139] and Rodriques et al. [140] aimed to improve the accessibility for users with disabilities who have hand tremors or suffer from speech impairment. Table 8 summarizes studies in CFASD: what is the research purpose, what functions of the AS API were used, and how usability tests were performed.

16 A PREPRINT -APRIL 26, 2021

Figure 3: Android mobile usage and sensor data categorization based on a hierarchical structure used in the surveyed studies

5 Categorization of Android Mobile Usage and Sensor Data

We investigate the mobile usage and sensor data that can be collected by the frameworks and applications developed in existing papers using the AS/US APIs, and other Androids APIs. Moreover, we present the type of mobile usage and sensor data that was used for each of the research purposes of the surveyed papers classiﬁed in Section 4.1. In addition, various types of contextual data that can be collected using the Android Framework APIs are presented. Therefore, this section provides an insight into the utilization of mobile usage and sensor data in the surveyed papers.

5.1 Excluded Studies in Data Categorization

The data utilization in surveyed studies corresponding to IP1 (i.e., studies that used AS or US API other than accessibility enhancement purposes for those with disabilities) is divided into three cases as follows: (1) mobile usage and sensor data collected through user studies, (2) mobile usage and sensor data collected through other methods such as developer test, crawling, crowdsourcing other than user study, (3) types of mobile usage and sensor data that can be collected in a framework developed for data collection purposes. We included all studies in the data categorization, except for one study [76] for which it was difﬁcult to determine the scope of the data used. Next, among the surveyed studies corresponding to IP2 (i.e., studies that used AS or US API of accessibility enhancement purposes for those with disabilities), only eight surveyed studies [158, 149, 157, 140, 144, 155, 146, 139] that performed user studies and collected data are included in the data categorization. Therefore, 97 studies out of the total number of surveyed studies (n=110) are included in data categorization.

17 A PREPRINT -APRIL 26, 2021

Table 9: Mobile usage and sensor data categories of surveyed studies Ref. Category Sub-Category "Application session" "when, for how long and which applications were active, provided by the Accessibility Service API screen usage" Ferreira et al. [121] "Screen usage" "current screen status (on/off) and for how long it was on/off" "Hardware sensors" "Accelerometer, magnetometer, and photometer" Ferreira et al. [107], Church et al. [20] "Software sensors" "User’s calendar, email, social activity, and other logs (e.g., calls, messages)" "Human-based sensors" "mobile questionnaires (i.e., for ESM), voice, or gesture input" "Contextual information" "location, physical activity, headphone jack, ringer mode, screen state, battery information, network information, Foreground application" Visuri et al. [23] "Notification information" "source application contents notification outcome" "Sender application, arrival time, tag, ticker text, sound mode, vibrate mode, category, title, text, subtext, priority, visibility, "Notification" action, audio mode, clearable, ongoing of a notification-flag" chang et al. [32] "Lastest used, application and time, ringer mode, audio mode, volume, network and wifi status, location, physical activity, acceleration, "Context" rotation, gravity, orientation, proximity, ambient light level, battery level, charging state" "Accessibility" "Triggered time, event type, event text, screen status" "Application events" "Active/inactive apps, touch and text input events, web browsing URLs, and notification events" Lee et al. [21] "System events" "Power on/off and screen on/off/unlock" "Phone events" "Call and SMS" "Phone state" "Ringer mode, battery level" Pielot et al. [26] "Usage" "Screen use, app launches, and access to the notification center" "Event listeners" "Interface accessibility events (e.g., button click, text field focus, view update, app screen switch, device app switch)" Zhang et al. [64] "Content introspections" "Element (e.g., content, size, state, possible actions)" "Accessibility service representation (e.g., click, long press, "Automation" select, scroll, or text input directed to a node in the tree)" Schweizer et al. [103], "Physical sensor" "Sound pressure, ring mode, acceleration, activity, illuminance, network connectivity" Schweizer et al. [104] "Software sensor" "Browsing history, contents, calendar, location, call log, foreground app" "App usage data" "Visited pages ("Screens" within an app), dwell times (per page), number of interaction (per page), device orientation" "Interaction data" "Type of interaction (touch, scroll, long touch), interaction target" Holzmann et al. [87] "Network data" "Network type, network subtype, roaming status" "Device data" "Device description, country of origin, language, network operator, operating system, screen resolution" "Battery data" "Battery level, charging status (charging/discharging), temperature, voltage" "Context data" "Light condition" "Orientation data" "Device orientation" "Ringer mode (Mute, vibration, ringer), charging mode (unplugged, charging), battery status, display orientation (portrait or landscape mode, "Phone context" Dingler et al. [36] orientation changes), light sensor, proximity sensor, location, motion" "Calls (incoming, outgoing), SMS (incoming, outgoing), notifications (received, dismissed, ignored, interacted with), screen (on/off events), "Phone usage" unlocks (phone unlocks), data usage (upload/download), applications (applications in foreground, switches, usage duration)" "Notification" "Application (name, package), Notification time (issued, interacted, posted), Notification ongoing (true, false)" Anderson et al. [34] "Contacts" "Contact (hash), Relation (family, friend, work, none)" "WiFi (SSID, BSSID), Location (latitude, longitude), Application (name, package), Time (date, issued, interacted, duration), ESM-Role (private, work), "Events" ESM-Interruptibility (private, work, both, not interruptible), Physical Activity (Google’s Activity Recognition API), Power (connected, disconnected), Screen (on, off), Ringer mode (silent, vibrate, normal") Location, activity recognition (Google activity recognition service API), sensors, network, calendar, phone status (ringer mode, screen on/off), Chang et al. [16] "Contexual information" currently running application "sender application, arrival time, tag, ticker text, sound mode, vibrate mode, category, title, text, subtext, priority, visibility, action, audio mode, "Notification" clearable, ongoing of a notification-flag" Chang et al. [32] "latest used, application and time, ringer mode, audio mode, volume, network and wifi status, location, physical activity, acceleration, rotation, "Context" gravity, orientation, proximity, ambient light level, battery level, charging state" "Accessibility" "triggered time, event type, event text, screen status" Lee et al. [35] "Contextual information" "notifications, user’s location and physical activity, activity type, the phone’s status (e.g., ringer mode, battery level), sensor data, user actions"

5.2 Method of Data Item Categorization

As shown in Figure 3, we create a hierarchical structure of the mobile usage and sensor data item terms using data categorization to identify and classify data collected from the Android thrid-party apps developed through Android APIs (e.g., AS/US APIs) in the surveyed studies. The hierarchical structure of the mobile usage and sensor data terms is created in four phases: (1) ﬁnd and extract the data term list from papers, (2) create a prototype of data categorization, (3) rearrange the data categorization, and (4) reﬂect on issues in the categories.

5.2.1 Find & Extract Data term

A data term list of mobile usage and sensor data that can be collected through the Android framework/application developed in the papers is obtained. Data terms described in studies were individual words or words included in categories and sentences. Examples of each description type for the screen on/off data are as follows:

• Word form: “Screen (on, off)” [34], “Screen on/off” [16], “Screen on/off event” [20], “Screen state” [33] • Sentence form: “Time when the phone’s screen was turned on or off” [24]. “Determine whether the user is currently using the phone” [29]

18 A PREPRINT -APRIL 26, 2021

5.2.2 Designing of Prototype of Data Categorization We create a typology of data by conducting data grouping and data term integration, and make the hierarchical structure for the prototype of data categorization. For data grouping, the data term lists derived in 5.2.1 are grouped based on similarities. As shown in Appendix A, uniﬁed data terms are deﬁned as the representative data term for each grouped data by referring to the data term of the existing research (Table 9) and Android developer documentation [159]. In addition, a hierarchical prototype of data categorization with three layers was created by referring to the data categorization of the existing research (Table 9) and the Android developer documentation [159].

5.2.3 Rearranging of Data Categorization We arranged the hierarchical structure by data categories when carefully reviewing the overall prototype of data categorization. The steps are as follows: (1) The data are grouped into categories or the existing categories are divided into multiple categories, reﬂecting on issues derived from Section 5.2.4; (2) the categories of data terms not described in the papers are excluded; (3) if the papers state that data are collected or can be collected, but no categories of data in the prototype of data categorization are available, a new category is created and added in the prototype of data categorization.

5.2.4 Reflected Issues in Data Categorization we reflected on the cases from Section 5.2.3 in which it was difficult to grasp the collected mobile usage and sensor data or show all the data types in the chart due to there being a large number of data types in one category. For example, we originally intended to present a four-layer hierarchical structure for the category of all data terms. Figure 3 shows the data categorization-based four-layer hierarchical structure results according to the unification of the mobile usage and sensor data terms described in the studies. The meaning of the first layer is as follows:

• Interaction sensing: Mobile usage and sensor data that users generate by interacting with or initiating an app or system service (excluding system-speciﬁc data) • Context sensing: Mobile usage and sensor data to understand the context of the system and the user • System sensing: Mobile usage and sensor data on the main state of the system (e.g., resources, screen status, device settings)

However, the data collected through the AS API (i.e., which is main Android API in this study) can be further classified below the four-layer. For example, among the user interaction data corresponding to the third-layer, the touch interaction data of the four-layer can be classified as click, long click, scroll, and typing once more. However, for the visible uniformity of the hierarchical structure of data categorization, the surveyed studies is mainly classified with only data categorization of four-layer hierarchical structure in the Section 6. Instead, the study using the data types classified from the user interaction data was further explained in Section 6.4. In the process of creating the hierarchical structure for mobile usage and sensor data terms, we identified some papers in which only a few examples of collected data were described or vaguely described. For example, there were data terms in some papers, such as “sensors,” for which it was difficult to identify what data were specifically collected. In addition, there were papers that described only a few examples of data using “such as,” “e.g.” and “ex”. In the discussions part of Section 7.3, we will cover the issues in the naming of data terms in more detail by dividing the surveyed studies on the basis of the ease of understanding the data terms presented in the papers.

6 Results of Classiﬁcation for Used Data and Research Purpose

According to goal of this study, we explain what data were used in each surveyed studies based on the data categorization result derived in Section 5, and described the trend of data used for each research purpose (i.e., research theme, sub- research theme) derived in Section 4. Through the results described in this section, our purpose is to understand the trend of data categories used via research themes and sub-themes. The results of data categorization and the analysis derived from each result consist of three sections. In Section 6.1, we begin with Table 10’s overall results of data categorization via surveyed studies including information on which research themes, sub-research themes, and type of inclusion survey for each of the surveyed studies. Further, we found the different data patterns of collection and utilization for each research theme through Table 10. In Section 6.2, we show the results of how many data categories were used for each research theme calculated through the results of Table 10 in Figure 4 and explain the analysis results from Figure 4. In Section 6.3, we describe the results of how many data categories were used for each research theme and sub-research theme calculated through the Table 10’s results in

19 A PREPRINT -APRIL 26, 2021

Table 11. Further, we analyze the results to identify type, number, and average number of data categories that used data categories via each research theme and sub-research theme. Further, we interpret the results of data categorization and analysis of the results described in the three sections (i.e., Sections 6.1, 6.2, and 6.3). Through this process, we provide the results and interpretations of how the types of data collected are different according to the research themes and sub-research themes classiﬁed in Section 4. Further, we classiﬁed and analyzed in depth the utilized user interaction data based on the AS API in the surveyed studies in Section 6.4.

6.1 Results of Classiﬁcation via Surveyed Studies

We show mobile usage and sensor data usage patterns according to the research theme, sub-research theme, and type of included papers (IP) corresponding to the surveyed studies based on Figure 3’s mobile usage and sensor data categorization in Table 10. For all mobile usage and sensor data, the columns in Table 10 are organized based on the second layer in Figure 3. However, user interaction data are based on the third layer, because they are collected by using the main API (AS API) in this review.

Table 10: Result of mobile usage and sensor data categorization

Interaction Sensing System Sensing Context Sensing

User Interaction

Type of Ref. Theme Sub-Theme Included Papers Type of interaction UI changed Time of interaction Target of interaction App usage patterns Notiﬁcations Contacts information Calendar and Standard Time Device setting Device sound states Device Battery states Device Operation states Device screen states Microphone Light Motion Proximity Network information GPS Camera

Blanke et al. [22] UP GUPA IP1-1-AS ••• Church et al. [20] UP GUPA IP1-1-AS •••• Kim et al. [17] UP GUPA IP1-1-AS •••••••• Lee et al. [21] UP GUPA IP1-1-AS •••••• Ferreira et al. [121] UP GUPA IP1-1-AS •• Yuan et al. [122] UP GUPA IP1-1-US •• Qin et al. [123] UP GUPA IP1-1-US • Qin et al. [124] UP GUPA IP1-1-US • Radesky et al. [125] UP GUPA IP1-1-US • Singh et al. [13] UP GUPA IP1-1-US • Welke et al. [101] UP GUPA IP1-1-US •••• Lee et al. [18] UP MLMS IP1-1-AS ••••• Khan et al. [110] UP MLMS IP1-1-US ••• Dutta et al. [12] UP MLMS IP1-1-US • Yan et al. [126] UP MLMS IP1-1-US • Yu et al. [19] UP MLMS IP1-2-AS • Andone et al. [102] UP FSD IP1-1-AS ••••••• Zheng et al. [127] UP FSD IP1-1-AS • Ferreira et al. [107] UP FSD IP1-2-AS •••••• Bitsch et al. [37] UP FSD IP1-2-AS ••• Holzmann et al. [87] UP FSD IP1-2-AS •••••••••• Montague et al. [88] UP FSD IP1-2-AS • Schweizer et al. [103] UP FSD IP1-2-AS •••••••••• Schweizer et al. [104] UP FSD IP1-2-AS •••••••••• Pielot et al. [24] NT IFN IP1-1-AS ••••• Visuri et al. [23] NT IFN IP1-1-AS •••••••• Chang et al. [16] NT IFN IP1-1-AS ••••••••• Mehrotra et al. [15] NT IFN IP1-1-US •••••• Lee et al. [27] NT IFN IP1-1-US •••••• Okoshi et al. [30] NT NT/NM IP1-1-AS •••• Okoshi et al. [28] NT NT/NM IP1-1-AS •••• Park et al. [29] NT NT/NM IP1-1-AS •••••• Pielot et al. [26] NT NT/NM IP1-1-AS •••••••••• Pradhan et al. [25] NT NT/NM IP1-1-AS ••••••••• Lee et al. [14] NT NT/NM IP1-1-US ••••• Lee et al. [100] NT NT/NM IP1-1-US •••••• Anderson et al. [34] NT HNAU IP1-1-AS •••••••••• Chang et al. [32] NT HNAU IP1-1-AS •••••••••••••• Dingler et al. [36] NT HNAU IP1-1-AS ••••••••••• Komuro et al. [33] NT HNAU IP1-1-AS ••• Lee et al. [35] NT HNAU IP1-1-AS •••••• Pielot et al. [31] NT HNAU IP1-1-AS • Sun et al. [71] EUI/UX CE IP1-1-AS •• Elazhary et al. [70] EUI/UX CE IP1-2-AS • Tarakji et al. [69] EUI/UX CE IP1-2-AS •• Wang et al. [72] EUI/UX CE IP1-2-AS • Chen et al. [61] EUI/UX UIE IP1-1-AS •• Li et al. [63] EUI/UX UIE IP1-1-AS •• Rauen et al. [65] EUI/UX UIE IP1-2-AS •••• Ryu et al. [62] EUI/UX UIE IP1-2-AS/US • Lin et al. [68] EUI/UX UIP IP1-1-AS •••••••••

20 A PREPRINT -APRIL 26, 2021

Table 10. Continued from previous page Interaction Sensing System Sensing Context Sensing

User Interaction

Riegler et al. [67] EUI/UX UIP IP1-2-AS ••• Riegler et al. [66] EUI/UX UIP IP1-1-AS ••• Zhang et al. [146] EUI/UX IFASD IP2-1 • Zhong et al. [139] EUI/UX IFASD IP2-1 ••• Rodrigues et al. [140] EUI/UX IFASD IP2-1 ••• Rodrigues et al. [144] EUI/UX IFASD IP2-1 • Chen et al. [149] EUI/UX DEASD IP2-1 ••• Pareddy et al. [155] EUI/UX DEASD IP2-1 • Rodrigues et al. [157] EUI/UX CFASD IP2-1 ••• Alshayban et al. [158] EUI/UX CFASD IP2-1 •••• Aras et al. [52] PS ASS IP1-1-AS • • Kraus et al. [53] PS ASS IP1-1-AS • Canfora et al. [54] PS ASS IP1-1-AS ••• Zhu et al. [128] PS ASS IP1-1-US ••• Zhu et al. [129] PS ASS IP1-1-US ••• Torres at al. [130] PS ASS IP1-1-US • Bonné et al [56] PS APC IP1-1-AS/US •• Rahman et al. [55] PS APC IP1-1-AS ••• Fawaz et al. [50] PS PPS IP1-1-AS •• Lau et al. [49] PS PPS IP1-1-AS ••• Ozcan et al. [51] PS PPS IP1-1-AS • Andriotis et al. [131] PS PPS IP1-1-US • Fernandes et al. [132] PS PPS IP1-2-AS ••• Naseri et al. [57] PS MDPS IP1-1-AS •• Rastogi et al. [133] PS MDPS IP1-2-AS • Shao et al. [60] PS MDPS IP1-2-AS • Li et al. [58] PS MDPS IP1-2-AS •••• Vishwamitra et al. [59] PS MDPS IP1-2-AS • Leguesse et al. [134] PS MDPS IP1-2-AS • Wang et al. [135] PS MDPS IP1-2-US • Wang et al. [136] PS MDPS IP1-2-US • Arruda et al. [73] PTS GAT IP1-1-AS • Gu et al. [74] PTS GAT IP1-2-AS •• Muangsiri et al. [75] PTS GAT IP1-2-AS ••• Lian et al. [78] PTS CT IP1-2-AS •• Wu et al. [77] PTS CT IP1-2-AS •• Li et al. [85] PTS APS IP1-1-AS •••• Li et al. [137] PTS APS IP1-1-AS •• Li et al. [84] PTS APS IP1-2-AS ••• Li et al. [86] PTS APS IP1-2-AS •••• Bissig et al. [79] PTS PMM IP1-1-AS •••• Xiang et al. [138] PTS PMM IP1-1-US ••• Fazzini et al. [82] PTS TRSGT IP1-1-AS ••• Fazzini et al. [83] PTS TRSGT IP1-1-AS ••• Liu et al. [81] PTS TRSGT IP1-2-AS •• Negara et al. [80] PTS TRSGT IP1-2-AS ••

We analyzed different data patterns in the collection and use for each research theme in Table 10. System sensing data (i.e., calendar and standard time, device sensing, device sound states, device battery states, device screen states), context sensing data (i.e., microphone, light, motion, proximity, network information, GPS, camera), and interaction sensing data (i.e., app usage pattern, user interaction, notification, contacts information) are collected for 14 out of 24 (58.3%) and 14 out of 18 (78%) surveyed studies involving UP (i.e., Usage Pattern) and NT (i.e., Notification), respectively. In addition, there is a tendency to use mobile usage and sensor data that corresponds to the classification criteria of both the system and context sensing. Out of 19, 21, and 15 total studies corresponding to the research themes of EUI/UX (i.e., Enhancement of UI and UX), PS (i.e., Privacy and Security), and PTS (i.e., Programming and Testing Support), respectively, only 5 (26.3%), 4 (19.0%), and 3 (20.0%) number of surveyed studies, respectively, collect and utilize system sensing data and context sensing data along with interaction sensing data. Therefore, we can infer that more varied types of mobile usage and sensor data are collected for UP and NT papers compared to EUI/UX, PS, and PTS studies.

6.2 Results of Classiﬁcation via Research Theme

We identiﬁed the number of surveyed studies that are collected and used for each research theme according to mobile usage and sensor data categories as depicted in Figure 4. The highest amount of data categories was observed for app

21 A PREPRINT -APRIL 26, 2021

Figure 4: Amount of each mobile usage and sensor data categories used for various research purposes in the surveyed studies

usage patterns (50 studies), and the least amount data categories for device setting (one study). The following types of data are collected and used in more than 15 of the 97 surveyed studies: app usage patterns (50 studies), type of interaction (46 studies), target of interaction (41 studies), motion (24 studies), notifications. (24 studies), device screen stats (21 studies), GPS (20 studies), UI changed (18 studies), and network information (17 studies). As shown in Figure 4, we derive the proportions of how data are used differently depending on the research themes. In case of the UP and NT research theme, almost all data types were used in at least more than one surveyed study. In the case of UP, app usage patterns data among all data types were used in a total of 20 number of studies, and were used more than twice as much as the second most used data which is type of interaction data. Because of the 22 studies using the US API, nine of them belong to the UP research theme, and five of them only collected app usage pattern data. In addition, the device screen states, GPS, and network information data were also widely used after app usage pattern data and type of interaction data with eight, eight, and seven, respectively. Accordingly, it can be seen that in the case of surveyed studies belonging to the UP research theme, context and system sensing data were also widely used with interaction sensing data. In the case of a total of 18 surveyed studies corresponding to the NT research theme, all data types except device setting data were used in more than one study. The most used data was notification data, which was used in a total of 18 studies. In addition, motion, device screen status, app usage patterns, device sound states, GPS, types of interaction, network information data were used in the order of 13, 10, 10, nine, eight, seven, and seven respectively. Therefore, we can infer that NT research theme also used various types of data in surveyed studies. In the contrast, most of the studies in EUI/UX, PS, and PTS research themes mainly utilize data types, which correspond to interaction sensing data (i.e., user interaction, app usage pattern data). Among a total of 19 studies corresponding to the EUI/UX research theme, target/type of interaction data was used the most in 12 and 11, respectively. App usage patterns data and target of interaction data were used in approximately 50% of studies (11 and 10 respectively) out of a total of 21 studies corresponding to the PS research theme.

6.3 Results and Interpretation of Classiﬁcation via Sub-Research Themes

We describe how many surveyed studies were used in each data category for each sub-research theme and analyzed the results with three measures as following: the number of types of data categories, the sum of utilized data categories, and the average number of data categories used per surveyed study (i.e., the sum of utilized data categories / total number of surveyed studies) by each sub-research theme as depicted in Table 11. In order to easily grasp the number of data categories used by each sub-theme in Table 11, we represented cells with a darker green color as the number of papers increased. Moreover, Table 11 is displayed with dark red cells as the number increases to easily distinguish the number of papers for each sub-theme and the trend calculated according to the three measures. Accordingly, we synthesized the

22 A PREPRINT -APRIL 26, 2021

Table 11: Number of surveyed papers used according to types of data in research theme and sub-research theme Interaction sensing System sensing Context sensing User Interaction

Theme Sub-Theme App usage patterns Notiﬁcations Contacts information Device setting Device sound states Device Battery states Device Operation states Device screen states Microphone Light Motion Proximity Network information GPS Camera Type of Interaction UI changed Time of Interaction Target of Interaction used per surveyed study Sum of utilized data categories Calendar and Standard Time # of types of data categories # of surveyed studies Average of data categories UP GUPA 11 2 3 2 1 1 6 1 3 3 10 33 11 3.0 UP MLMS 4 1 1 1 2 1 1 7 11 5 2.2 UP FSD 5 6 1 4 1 4 3 1 2 1 2 2 3 4 4 4 1 17 48 8 6.0 NT IFN 3 1 4 2 1 3 1 4 3 1 2 3 12 28 4 7.0 NT NT/NM 4 4 2 3 8 1 2 2 2 3 2 1 5 2 2 1 1 17 45 8 5.6 NT HNAU 3 2 1 1 1 6 2 1 4 3 2 3 2 5 2 3 4 17 45 6 7.5 EUI/UX CE 3 1 2 3 6 4 1.5 EUI/UX UIE 2 2 3 1 1 5 9 4 2.3 EUI/UX UIP 3 2 2 1 1 1 1 1 1 1 1 11 15 3 5.0 EUI/UX IFASD 2 2 4 3 8 4 2.0 EUI/UX DEASD 1 1 1 1 4 4 2 2.0 EUI/UX CFASD 2 1 1 1 2 5 7 2 3.5 PS ASS 5 1 2 3 1 1 6 13 6 2.2 PS APC 1 1 1 2 4 5 2 2.5 PS PPS 2 2 2 1 2 1 6 10 5 2.0 PS MDPS 3 2 1 6 4 12 8 1.5 PTS GAT 3 1 2 3 6 3 2.0 PTS CT 2 1 1 3 4 2 2.0 PTS APS 1 3 2 1 4 1 1 7 13 4 3.3 PTS PMM 2 1 1 2 1 5 7 2 3.5 PTS TRSGT 2 4 2 2 4 10 4 2.5 Sum of UP 20 9 1 2 6 4 6 3 1 3 1 3 8 0 3 6 0 7 8 1 34 92 24 3.8 Sum of NT 10 7 3 1 4 18 5 4 0 9 6 2 10 2 3 13 5 7 8 1 46 118 18 6.6 Sum of EUI/UX 4 11 3 5 12 0 0 1 0 0 2 0 1 2 1 2 0 2 3 0 31 49 19 2.6 Sum of PS 11 6 4 1 10 1 0 0 0 0 0 0 2 0 0 3 0 1 0 1 20 40 21 1.9 Sum of PTS 5 13 7 1 9 1 0 0 0 0 0 2 0 1 0 0 0 0 1 0 22 40 15 2.7 Total 50 46 18 10 41 24 11 8 1 12 9 7 21 5 7 24 5 17 20 3 153 339 97 3.5

results of Section 6.1 and 6.2, and interpreted the trends in utilization of data categorization for each research theme and sub-research themes.

6.3.1 Usage Pattern (UP) Research Theme Most studies made third-party apps to collect various types of mobile usage and sensor data with smartphones in FSD (i.e., Framework and System Development) sub-research theme. For these reasons, the number of data categories used of FSD sub-research themes was highest compared with other sub-research themes. By contrast, GUPA (i.e., Generic Usage Pattern Analysis) and MLMS (i.e., Machine Learning-based Model and System) sub research themes mainly used interaction sensing data, and the average used data categories was 2.6, which did not use various data categories compared to FSD. When the AS API was used among surveyed studies of GUPA and MLMS sub-research themes, various mobile usage and sensor data were collected by utilizing other Android APIs as well. However,using the US API, the average of data categories used per surveyed study was relatively low because it only needs app usage patterns about the current usage status of device system or third-party apps rather than detailed contextual information. Therefore, as most of the studies using the US API in GUPA and MLMS sub-research themes, relatively fewer number of data categories were used than FSD sub-research themes.

6.3.2 Notification (NT) Research Theme NT research themes had relatively high analysis results corresponding to three measures (i.e., sum of utilized data categories and types of data categories, number of types of data categories, and average number of data categories used per surveyed study) compared to other research themes in Table 11. In addition, the proportion of using context sensing and system sensing data along with interaction sensing data was the highest compared to other research themes according to the analysis in Section 6.3. The reason that surveyed studies belonging to the NT research theme used various data categories on average compared to other research themes is as follows for each sub research theme. In the case of surveyed studies corresponding to the IFN (i.e., Identifying Factors responding to Notifications) sub-research theme in NT, as those studies predict which notifications interacts and in what context, various types of the mobile usage and sensor data are used to identify various contextual information [23, 24, 16]. In the case of Pielot et al. [26],

23 A PREPRINT -APRIL 26, 2021

Pradhan et al. [25], and Park et al. [29] among surveyed studies corresponding to HNAU (i.e., How Notification Affect a User) sub-research themes in NT, various types of mobile usage and sensor data were collected to understand how notifications corresponding to various contextual information affect users. In the case of surveyed studies corresponding to NT/NM (i.e., Notification Timing and Notification Management) sub-research themes in NT, various types of mobile usage and sensor data were collected to find the right timing of notification or notification management corresponding to various contextual information [32, 33, 34, 35, 36].

6.3.3 Enhancement of UI/UX (EUI/UX) Research Themes IFASD (i.e., Improvement of the Functionality of Accessibility Service for the users with Disabilities) and CFASD (i.e., Comparison of Functions of Accessibility Service for the users with Disabilities) sub-research themes collected 100% of interaction sensing data, whereas studies in DEASD (Development or Evaluation of an Application using Accessibility Service for the Users with Disabilities), UIP (i.e., User Interface Profiling), UIE (i.e., User Interface Enhancement), and CE (i.e., Computational Enhancement) used context sensing data along with interaction sensing data as described in Section 6.3. However, it is difficult to generalize because there is a tendency to collect context sensing data in one or two of the surveyed studies corresponding to the actual DEASD, UIP, UIE, and CE sub research themes [71, 69, 65, 68, 149]. Furthermore, the average number of data types used per surveyed study is 2.72, which is hard to say that various data types were used in the Table 11’s results through investigating all surveyed studies corresponding to EUI/UX research themes. Therefore, most of the studies belong to EUI/UX research theme in practice are mainly for the purpose of improving the user interface, and in fact, it can be seen that the user interaction data based on the AS API was mainly used. This can be seen from the fact that among the interaction sensing data, notification and contacts information data was rarely used.

6.3.4 Privacy and Security (PS) Research Themes The average number of data categories used per surveyed study of PS research theme was 2.05, which utilized relatively less types of data than other research themes and sub-research themes. Further, as described in Section 6.3, surveyed studies belonging to APC (i.e., Access and Permission Control), PPS (i.e., Privacy-Preserving System), and MDPS (i.e., Monitoring and Detection for Privacy and Security) sub-research themes used only interaction sensing data. A total of 15 studies related to privacy and security corresponding to APC, PPS, and MDPS sub-research themes mainly used the user interaction data (i.e., type/target of interaction, UI changed) [56, 55, 50, 49, 51, 131, 132, 57, 133, 60, 59, 134, 58, 135, 136]. By contrast, surveyed studies belong to ASS (i.e., Authentication System/Scheme) sub-research themes used the context and system sensing data along with user interaction data for the research related to the user authentication system and scheme to understand the current device hold situation and daily habits of the user or to develop a motion sensor-based authentication system [52, 53, 54, 128, 129, 130].

6.3.5 Programming and Testing Support (PTS) Research Themes The average number of data categories used per surveyed study is 2.66. Furthermore, among the 15 studies on the PTS research theme, 12 studies excluding three studies [79, 138, 86] used only types of data corresponding to interaction sensing data. In the three studies, Bissi et al. [79] and Xiang et al. [138] commonly collected device operation status data for performance measurement and management research, in addition to data types corresponding to interaction sensing data. In surveyed studies corresponding to the GAT (i.e., GUI Automated Testing), CT (i.e., Crowdsourced Testing), APS (i.e., Automated Programming Support), and TRSGT (i.e., Test Recorder and Script Generator Technique) sub-research themes, except for the microphone data used in Li et al. [86], app usage pattern, user interaction, and notiﬁcation data, which are data types that belong to interaction sensing data, are used in more than one paper. This is because, in the case of CT, GAT, and TRSGT sub-research themes, interaction data was mainly used through AS API for crowd sourced testing, GUI automated testing, test recording and script generator technique related research [73, 74, 75, 78, 77, 82, 83, 81, 80]. In addition, the APS sub-research theme mainly used AS API-based interaction sensing data for automated programming support [85, 137, 84, 86].

6.4 Categorization for Type of User Interaction Data

We figured out which data types were used for each surveyed study among the five-layer data types (e.g., click, long click, typing, scroll) not described in Figure 3 that can be classified in user interaction data. First, among the types of interactions (i.e., type of touch interaction) of user interaction data in Table 10, we investigated the 45 surveyed studies that described the use of click, long click, typing, and scroll that can be collected by the AS API as shown in 12. In addition, we provided insights by analyzing in-depth surveyed studies describing which accessibility events were used to detect user interaction data in Table 13. In addition, for studies that did not describe the accessibility events used,

24 A PREPRINT -APRIL 26, 2021

Table 12: Collected touch interaction type information from the surveyed papers Types of Touch Interaction Ref. Num General Expression [55, 35, 32, 107, 29, 71, 65, 85, 158, 140] 10 [17, 18, 87, 88, 30, 28, 70, 69, 68, 67, 66, 132, 73, 74, 75, 78, 77, 84, 86, 79, 82, 83, 81, 80, Click (including tapping) 26 139, 15] Long click [17, 18, 87, 30, 28, 66, 75, 78, 77, 84, 86, 80, 15] 13 Typing [21, 37, 103, 104, 30, 28, 70, 69, 61, 68, 54, 57, 58, 75, 78, 77, 84, 86, 82, 83, 80] 21 Scroll (Including Pinch and Swipe) [17, 87, 30, 28, 69, 68, 67, 66, 49, 58, 74, 75, 78, 77, 139, 15, 80, 88] 18 we deduced the accessibility events that were used to detect app usage patterns, notifications, and types/time/target of interaction data that can be collected by the AS API among interaction sensing data. Among 45 studies using interaction data, 18 studies use terms related to type of touch interaction (i.e., click, long click, typing, scroll, pinch, swipe) to describe the type of touch interaction used in References [17, 21, 37, 87, 88, 103, 104, 15, 70, 69, 61, 68, 67, 139, 73, 74, 77, 80]. The 16 studies which are the names of Accessibili- tyEvent related to the type of touch interaction (TYPES_VIEW_CLICKED, TYPES_VIEW_LONG_CLICKED, TYPES_VIEW_TEXT_CHANGED, TYPES_VIEW_SCOLLED) were used to describe the touch interaction used in References [18, 30, 28, 66, 49, 132, 57, 58, 75, 84, 86, 79, 82, 83, 81, 78]. In the remaining 11 studies, general term expressions were described such as “user action”, “touch event”, “all UI events”, “touch event”, “input action”, and “gesture action” were used in References [107, 29, 32, 35, 71, 65, 140, 158, 54, 55, 85]. To this end, the variously described terms related to type of touch interaction are grouped and classified into five categories: general expression, click (include tapping), long click, type, and scroll (include punch and sweep) as described in Table 12. Therefore, excluding the general expression category, the most frequently used type of touch interaction among the four categories was redundantly used in 26, 21, 18, and 13 studies in the order of click, typing, scroll, and long click, respectively. The reason that click and typing occur most frequently is that the touch interaction gestures that users use most in their smartphones are click and typing. Therefore, it is believed that there are many studies using interaction gesture data of click and typing. Among the studies using interaction sensing data in Table 10, there are a total of 18 studies [18, 28, 30, 66, 144, 52, 49, 132, 57, 58, 75, 78, 84, 86, 79, 82, 83, 81] describing the names of accessibility events used for detection as depicated in Table 13. For each research theme, the names of accessibility events were described in the surveyed studies in the order of eight, five, two, two, and one number of studies in PTS, PS, EUI/UX, NT, and UP, respectively. When the types of accessibility events described as captured in the studies in Ta- ble 13 are largely divided into four types: view types (e.g., TYPE_VIEW_CLICKED), transition types (e.g., TYPE_WINDOW_STATE_CHANGED), notification types (e.g., TYPE_NOTIFICATION_STATE_CHANGED), and exploration types (e.g., TYPE_TOUCH_INTERACTION_START/END), the number of studies used for each accessibility event type are in the order of 55, 23, two, and two, respectively. Among the studies describing all detected accessibility events, the most frequently used accessibility event was TYPES_VIEW_CLICKED, which was used in a total of 13 studies, and the least described accessibility event name was TYPE_TOUCH_INTERACTION_START/END event, which belongs to exploration types, and was used only in one study each. In addition to 18 studies describing accessibility events used in Table 13, we deduced which accessibility service was used for 24 surveyed studies that used the AS API-based interaction sensing data, but did not describe the which types of accessibility events used to detect the interaction sensing data. The list of surveyed studies according to accessibility events that are inferred to have been used to capture the utilized data is as follows.

• Notiﬁcation types (i.e., TYPE_NOTIFICATION_STATE_CHANGED): 18 studies which were used noti- ﬁcation data [22, 21, 102, 24, 23, 16, 27, 30, 28, 29, 26, 34, 36, 33, 35, 31, 50, 85] • Exploration types (i.e., TOUCH_GESTURE_DETECTION_START/END): Rodrgues et al [140] and Alshayban et al [158] which were used gesture interaction data for enhancing the user interface of the people with disabilities in exploration mode. • View Types (i.e., TYPE_VIEW_CLICKED, TYPE_VIEW_TEXT_CHANGED): 42 studies in Table 12 (except [158, 140, 15]) • Transition types (i.e., TYPE_WINDOW_STATE_CHANGED): 18 studies that used UI changed data [18, 30, 28, 32, 68, 66, 158, 55, 49, 132, 58, 75, 78, 137, 86, 79, 82, 83] and 30 studies that used app usage pattern data [22, 20, 17, 21, 121, 18, 102, 107, 87, 103, 104, 23, 16, 26, 25, 34, 32, 36, 63, 65, 157, 158, 52, 53, 49, 58, 85, 79, 82, 83]

25 A PREPRINT -APRIL 26, 2021

Table 13: Accessibility events described in surveyed studies VIEW TYPES TRANSITION TYPES NOTIFICATION TYPES EXPLORATION TYPES

Type of Ref. Theme Sub-Theme Included NOTIFICATION STATE Papers CHANGED CHANGED AT MOVEMENT GRANULARITY CLICKED LONG CLICKED SELECTED FOCUSED TEXT CHANGED TEXT TRAVERSED TEXT SELECTION SCROLLED WINDOW STATE WINDOW CONTENT WINDOW CHANGED INTERACTION START INTERACTION END CHANGED CHANGED TYPE TOUCH TYPE TOUCH

Lee et al. [18] UP MLMS IP1-1-AS ••• Okoshi et al. [30] NT NT/NM IP1-1-AS ••••••••••• Okoshi et al. [28] NT NT/NM IP1-1-AS ••••••••••• Riegler et al. [66] EUI/UX UIP IP1-1-AS •••••••••• Rodrigues et al. [144] EUI/UX IFASD IP2-1 •• Aras et al. [52] PS ASS IP1-1-AS • Lau et al. [49] PS PPS IP1-1-AS ••• Fernandes et al. [132] PS PPS IP1-2-AS ••• Naseri et al. [57] PS MDPS IP1-1-AS •• Li et al. [58] PS MDPS IP1-2-AS •• Muangsiri et al. [75] PTS GAT IP1-2-AS ••••• Lian et al. [78] PTS CT IP1-2-AS •••••• Li et al. [84] PTS APS IP1-2-AS ••• Li et al. [86] PTS APS IP1-2-AS ••••••• Bissig et al. [79] PTS PMM IP1-1-AS ••• Fazzini et al. [82] PTS TRSGT IP1-1-AS •••• Fazzini et al. [83] PTS TRSGT IP1-1-AS •••• Liu et al. [81] PTS TRSGT IP1-2-AS ••

So far, we analyzed and inferred the type of touch interaction used in surveyed studies and the types of accessibility events used in each study to collect AS API-based interaction sensing data as described. Through this, we provided the insights on accessibility events for using AS API-based interaction data to subsequent researchers. However, it is possible that other APIs or accessibility events were used along with the above mentioned accessibility events because the studies of Table 10 using app usage pattern and interaction data except 18 studies of Table 13 did not describe what accessibility events were used speciﬁcally.

7 Discussion

We discuss the several issues and implications found while investigating existing studies. In Section 7.1, we begin with summary of classification results. In Section 7.2, we further discussed what data types and research purposes were used in each surveyed studies according to types of included papers (IP). In Section 7.3, we discuss the cases where it was difficult to grasp the smartphone data terms described by each study. In Section 7.4, the differences due to changes in API policy are then discussed. In Section 7.5, we deal with the privacy and security issues in terms of data collection. In Section 7.6, we discuss limitations and future research through this review. According to these issues, we intend to contribute to the vitalization of research using Android APIs to collect mobile context data by providing researchers with a review of the research up to now in this field and guidelines through it.

7.1 Summary of Classiﬁcation Results

We categorized major research themes of the prior studies and their data types collected using Android AS/US API and other sensor APIs. In Section 3, out of 880 pieces of literature searched through AS/US API related keywords, 110 pieces of literature to be finally reviewed were filtered by the prizma technique. Afterward, the surveyed studies are classified six types of included paper-based on (1) which API among AS/US APIs was used, (2) whether user studies were performed, and (3) the purpose of the AS API, which is the purpose of improved UI/UX for the people with disabilities as shown in Table 6. In Section 4, research themes were classified as UP (i.e., usage pattern), NT (i.e., notification), EUI/UX (i.e., enhancement of user interaction and user experience), PS (i.e., privacy and security), and PTS (i.e., programming testing and supporting) into 16, 18, 13, 21, and 16, respectively, using the affinity diagramming method. Furthermore, it was further classified as sub-research themes according to each research theme. As a result, 18 sub-research themes were derived from a total of non-disabled studies in all research themes, and three sub-research themes were classified from the studies for the people with disabilities in one research theme (EUI/UX). In Section 5, we categorized data types because there was a lack of standardized terms that describe the types of data collected for research purposes. Most of the prior studies used both “AS/US API” for usage data collection and sensor data collection as well. By thoroughly reviewing existing research and Google android documentation’s categories and surveyed studies, we developed the categorization based on a four-layer hierarchical structure as shown in Figure 3. As a hierarchical structure of data categorization, the first layer is grouped into android mobile usage and sensor data. The second layer is classified as interaction sensing data, which corresponds to usage data collection, which is collected mainly through AS/US APIs, context sensing data which is collected through smartphone built-in hardware sensor and

26 A PREPRINT -APRIL 26, 2021

APIs to detect the surrounding context-aware, and system sensing data which is collected by device and system APIs. In the third layer, interaction sensing is classified into app usage pattern, user interaction, notification, and contacts (call and SMS) information. Next, context sensing is categorized into the environment, motion, proximity, network information, GPS, camera, microphone. Lastly, system sensing is composed of device setting and device sound states, device battery states, device operation states, and calendar & times. In Section 6, we analyzed and interpreted what data was used for what research purpose based on the classification results from Section 3 to 5. First, we identified which data categories were used for each surveyed study according to the research theme, sub-research theme, and type of included papers (IP) as shown in Table 10. However, it was difficult to grasp at a glance what data was used for each research purpose through Table 10. Accordingly, we analyzed which data categories were used for each research themes as shown in Figure 4. From the results, we figured out that the trends of the used data types are different according to each research theme. First, 83% of the studies belonging to the NT research theme were used not only interaction sensing data (i.e., user interaction, app usage pattern data, notification, contacts information) but also various kinds of data corresponding to system sensing and context sensing data. In addition, 58% of the surveyed studies corresponding to the UP research theme utilized context sensing and system sensing data as well as interaction sensing data. In contrast, most of the surveyed studies corresponding to 74%, 81%, and 80% of the EUI/UX, PS, and PTS research themes, respectively, mainly used interaction sensing data that can be collected by AS/US API. Further, we analyzed the data types used by sub-research themes classified in each research themes to understand the trend of the data types used for each research purpose in more detail, as shown in Table 11. As a result, different trends were shown for the data types used according to the sub-research themes for each research theme. In NT and UP research themes, data types corresponding to context sensing or system sensing in all sub-research themes except one sub-research theme (i.e., MLMS) were utilized together with data types belonging to interaction sensing. In the case of EUI/UX research themes, there has been at least one research using data types belonging to context sensing or system sensing in addition to interaction sensing in four sub-research themes (i.e., CE, UIE, UIP, DEASD) out of six. In surveyed studies of six sub-research themes (i.e., APC, PPS, MDPS, GAT, CT, TRSGT) among nine sub-research themes belonging to PS and PTS research themes, only the types of data corresponding to interaction sensing was used. Through this result, this study identified the tendency of data types utilized according to the various research purpose and derived insights by interpreting them.

7.2 Comparative Analysis of Existing Studies

In this section, we further discuss the research purpose (theme, sub-theme) and the data utilization that was identified according to types of IP (i.e., included papers) classified in Table 6. Through this, we present different patterns on the categories of research purposes and the utilization data according to the IP criteria (i.e., whether AS API was used and user study was performed), and finally we provide the deep insightful trends of survey results. To discuss this, we describe the number of surveyed studies in each research themes and sub-research themes via the types of IP criteria in Table 14. Second, we show the number of surveyed studies for each data category via the type of IP as shown in Table 15. As shown in Table 14, IP1-1-AS studies (i.e., studies that perform user studies on AS API for improving the interface for the people with disabilities) which are user studies using AS API for improving the interface for the people with disability are used for more diverse research themes than those in other types of IP. More than five belong to each theme, and more than one belong to all sub-research themes except CT, IFASD, DEASD, and CFASD. In addition, studies belong to IP1-1-AS are mostly used for NT research theme. Studies belong to NT research theme are used together to collect and utilize notification data by utilizing TYPE_NOTIFCATION_STATE_CHANGED of AS API or by using notification-related APIs (e.g., NotificationListenerService, NotificationManager). Nine of the IP1-1-US (i.e., studies that perform user studies on US API) studies include UP as a research theme, and one or more involved NT, PS, and PTS as research themes, which shows that the US API can be used for various research purposes in addition to an analysis of app usage patterns. In the IP1-2-AS (i.e., studies that did not perform user studies to improve the interface for the people with disability among the studies using AS API), more than four surveyed studies belong to the research theme of UP, PS, PTS, EUI/UX, and studies related to the sub-research theme of FSD are the most common. This shows that IP1-2-AS is heavily utilized in the FSD sub-research theme. In the IP1-2-US (i.e., studies that did not perform user studies among studies using US API), two studies focused on PS and one on EUI/UX, which shows that even if app usage pattern data are utilized using the US API in surveyed studies, the US API is not used for the collection of user data in research theme in the UP or NT research themes. In the IP2-1 (i.e., studies where user studies were performed to improve the interface for the people with disabilities) and IP2-2 (i.e., studies where user studies were not performed to improve the interface for the people with disabilities), all the studies belonged to EUI/UX. In addition, of the studies in IP2 (i.e,

27 A PREPRINT -APRIL 26, 2021

Table 14: Number of surveyed studies in each research themes and sub-themes via the types of IP criteria IP1: Studies for People without Disabilities IP2: Studies for People with Disabilities Theme Sub-Theme Overall IP1-1-AS IP1-1-US IP1-2-AS IP1-2-US IP2-1 IP2-2 UP GUPA 5 6 11 UP MLMS 1 3 1 5 UP FSD 2 6 8 NT IFN 3 1 4 NT NT/NM 6 2 8 NT HNAU 6 6 EUI/UX CE 1 3 4 EUI/UX UIE 2 2* 1* 5 EUI/UX UIP 2 1 3 EUI/UX IFASD 7 2 9 EUI/UX DEASD 7 1 8 EUI/UX CFASD 3 3 PS ASS 3 3 6 PS APC 2* 1* 3 PS PPS 3 1 1 5 PS MDPS 1 5 2 8 PTS GAT 1 3 4 PTS CT 2 2 PTS APS 2 2 4 PTS PMM 1 1 2 PTS TRSGT 2 2 4 Overall 43 18 28 3 17 3 112-2*= 100 * Bonné et al. [17] and Ryu et al. [131], which is used AS/US APIs both, were included in IP1-1-AS/US and IP1-2-AS/US twice, respectively. Therefore, the number of surveyed studies in this study is 110, excluding the count of two out of a total of 112 overalls.

IP2-1 and IP2-2), the studies of IFASD are more common than DEASD and CFASD. This shows that there are more “studies on improving the functionality of AS API for the users with disabilities” than “studies on the development of application programs for the users with disabilities” and “studies on the comparison of AS functions for the users with disabilities.” Table 15 shows the number of surveyed papers for each data category according to the type of IP. The studies belonging to the IP1-1-AS used 20 data categories, the most among all the IP categories, followed by 18, 10, 7, and 1 data categories for IP1-2-AS, IP1-1-US, IP2-1, IP1-2-US, respectively. The average number of data categories used per study is 4.4, 3.1, 2.5, 2.1, and 1.0 for IP1-1-AS, IP1-2-AS, IP1-1-US, IP2-1, IP1-2-US, respectively. Therefore, on average, the highest number of data categories per paper are used in IP1-1-AS. When only the number of studies is identiﬁed in relation to interaction sensing data, the studies belonging to IP1-1-AS use the more interaction sensing data than those corresponding to other IP categories. By contrast, we found that notiﬁcation data are not collected in studies of IP1-2-AS, IP1-2-US, and IP2-1. We derive the number of surveyed papers for each data category according to the type of IP used in terms of ratios from Table 15. In the studies belong to IP1-1-US, the app usage pattern data are used the most when considering the functions of the US API. IP1-1-US studies used the system and context sensing data at a 46.5% rate, which is the same as that for IP1-1-AS studies. However, IP1-1-US studies employed half of the number of data categories employed in IP1-1-AS studies. It can be seen that when using AS API to use data through user study, there is a higher tendency to use various Android built-in sensors and other Android APIs together for research than when using US API. In addition, IP1-2-US and IP2-1 rarely utilized the system and context sensing data, which shows that other Android built-in sensors and APIs are not used well in studies that employ AS API for the purpose of improving the accessibility for the users with disabilities.

7.3 Data Term Issues Regarding Reproducibility

While investigating the mobile usage and sensor data collected and used in the studies, the identification of mobile usage and sensor data terms in the studies can be divided into (1) the detailed explanation of data terms and (2) the vague description of data terms. As shown in Table 9, for studies of (1) in which they are easy to identify the used events and data terms are clear, and the data are categorized with the description of each data and displayed in a table. In contrast, for studies of (2) in which the API and data names are not specifically described, or the scope of the data is unclear because only a few examples are described. For example, it is difficult to identify specifically what data are collected by certain papers that used vague terms (e.g., "UI Interaction Collection", "sensors", "sensor data") rather than

28 A PREPRINT -APRIL 26, 2021

Table 15: Types of data used in surveyed papers according to types of IP Interaction sensing System sensing Context sensing User Interaction

Type of Included Papers used per surveyed study # of types of data categories Sum of utilized data categories # of surveyed studies Average of data categories App usage patterns Notifications Contacts information Device setting Device sound states Device Battery states Device Operation states Device screen states Microphone Light Motion Proximity Network information GPS Camera Type of Interaction UI changed Time of Interaction Target of Interaction Calendar and Standard Time IP1-1-AS 23 21 12 4 14 21 7 4 10 7 5 14 3 4 13 3 13 11 2 20 191 43 4.4 IP1-1-US 17 2 3 1 1 1 6 6 2 4 10 43 17 2.5 IP1-2-AS 6 20 5 3 20 3 3 1 2 2 1 1 2 3 4 3 4 1 18 84 27 3.1 IP1-2-US 2 1 2 2 1.0 IP2-1 2 3 1 3 7 1 1 1 8 19 8 2.4 sum 50 46 18 10 41 24 11 8 1 12 9 7 21 5 7 24 5 17 20 3 20 339 97 3.5 specifying what UI interaction data or sensor data are utilized using the AS API. Next, there are some papers where it is difficult to understand the entire data used as these papers only described a few examples using “such as,” “e.g.” and “ex”. In the case of (2), it is not difficult to grasp the flow of the paper. However, if the data (e.g., mobile data, events, and APIs) used in the paper are described more specifically as in (1), it can be communicated more clearly to the readers in identifying the data which are described in each paper. In the following, we classify the studies into two types according to the degree of the detailed description in the papers in terms of data categorization based-on hierarchical structure in Figure 4:

• Detailed described studies as the data term level of the third layer in Figure 4 (e.g., "click", "scroll", "text changed", "notiﬁcation time", "accelerometer", "gyro", "battery level"):[23, 24, 34, 33, 25, 31, 20, 21, 17, 30, 28, 27, 102, 87, 103, 104, 18, 22, 36, 121, 68, 66, 73, 79, 82, 83, 80, 49, 57] • Vaguely described studies as the data term level of the ﬁrst or second layer in Figure 4 (e.g., "all UI events", "sensors", "sensor data", "event type", "user actions", "gesture action", "context events", "context logs"): [35, 16, 32, 140, 29, 127, 72, 76]

Furthermore, depending on the method of describing the data terms, there can be a difference in the degree of ease in terms of reproducibility for other researchers. The method of describing data terms in 110 number of papers can be divided into two types of studies: First type of studies are described by words or words with categorization, second type of studies are described by words in sentences. As shown in Table 9, the data term descriptions in first type of studies are clearer to grasp than those of second type of studies. In addition, while there are surveyed studies that described the Android-related component terms (e.g., API, class, event, smartphone data) as described in the Android or Google developer documentation, there are also surveyed studies that did not refer to the terms in the Android or Google developer documentation. Researchers who are new to data-driven research, as opposed to experienced researchers who have conducted smartphone-related research, would find it easier to grasp and reproduce papers with data terms written according to Android or Google developer documentation. Therefore, if the terms used to describe events and data match with that in the Google developer documentation, future researchers can understand the related research more easily in this field.

7.4 Practical Issues about AS and US API

Based on the level of the Android API, there is a difference in the API mainly used to collect app usage data and the types of app usage data that can be collected. Prior to 2014, studies mainly used AS API to determine the changes in window transitions and touch interactions, and to track the view information to collect the app usage and view information data. However, from 2014 onwards, starting with Android version 5.0 (API level 21), app usage patterns can be grasped through the US API, making it more possible to intuitively understand various status information about app usage. In addition, the AS API is regulated by Google Android to be used only to support the use of Android devices and apps by the users with disabilities. However, we are not able to collect the information regarding interaction types (e.g., click, long click, scroll, focused, and typing), and interaction targets (UI elements and hierarchy) from the US API. In particular, studies corresponding to speciﬁc research purposes (PTS, PS) that require view information (interaction type, interaction target) inside the app

29 A PREPRINT -APRIL 26, 2021

had no choice but to rely on the AS API, which has a strong function. While AS API can broaden the functionality of applications, it can potentially generate security risks. Once granted the user’s permissions, the API can be used to read data from other apps. Therefore, Google currently regulates the use of the AS API and it can be used only for the purpose of improving the accessibility for users with disabilities for apps registered in the Google Play Store58. If the developer does not follow the policy, the policy support team of the Google Play Console59 will inform the developer to explain how the AS API is used in the developed application to help users with disabilities. An app that does not meet the requirements within 30 days is removed automatically from Google Play, the ability to AS API within the app is removed, or the app is regulated and unpublished [160]. So far, for apps that use AS API other than those intended to assist the users with disabilities, there are no restrictions on publishing apps to other app stores or web hosting services (e.g., GitHub). In addition, even when user data using the AS API is collected in research through a user study, there are no further speciﬁc prohibition regulations by Google. However, since the AS API has powerful functions that can fetch almost all information (e.g., interaction type/target and app status) about the app being used, the regulation of the Google Android API policy may become more severe due to personal privacy and security issues. Therefore, even when using AS API for future research purposes, other alternatives are needed in preparation for future restrictions.

7.5 Privacy and Security Issues in Personal Data Research

In the surveyed studies of this article, mobile usage and sensor data collected and utilized through Android AS/US API and other APIs may cause privacy and security issues in the process of collection, storage, and utilization. For example, those studies that analyze smartphone data in the wild to understand users’ smartphone usage patterns and their surrounding contexts (i.e., interaction, context, and system sensing data) can be used to reconstruct a user’s everyday life patterns. Specially, AS API can be used to hijack sensitive personal data such as credit information [161]. Furthermore, among the surveyed studies of this article, studies that collect large-scale data over a long period of time may further increase the risk of the privacy and security. Specially, among the UP and NT research theme, if the goal is to identify the predictive features or build machine learning models, it would be beneﬁcial to collect various types of smartphone data, possibly with many participants for longer duration to improve the external validity of the research as investigated in in Section 4, 6. Accordingly, when conducting data-driven research based on mobile usage and sensor data which covered in this article, it is required to minimize privacy and security risks, and this should be carefully addressed in the user consent such as IRB documents and the app’s request for consent. While doing so, researchers can follow well-known data protection guidelines principles that must be followed when using personal and sensitive user data (i.e., information from or about users and devices) with Android smartphone app deﬁned by the Google Play Console’s Policy Center [162], U.S. Federal Trade Commision’s fair information practices principles (FIPPs)55, EU General Data Protection Regulation (GDPR)56, and EU-U.S. Privacy Shield Framework57. Although there are many principles for the personal data protection, we have discussed studies using mobile usage and sensor data through smartphone third-party apps in the Android OS. Therefore, we discuss the core smartphone data protection principles for smartphone data collection, storage, and analysis in the study by referring to other data protection principles centered on data protection principles of Google Play Console’s Policy Center.

• Openness and Transparency: Details regarding smartphone data and app access, collection, use, and sharing processes, including the purposes of data collection and information regarding which data will be collected, should be transparently shared with and explained to the participants. • Individual Participation Rights: Participants should have the rights to access, modify, and delete their data, or may refuse to provide data, or prohibit the use of their data at any time. A careful consideration of participants’ duties and rights should be clearly illustrated in the app and IRB documents as well as the consent form. • Limited Data Collection: Researchers should collect only the data necessary to meet their research purposes. As in transparency requirements, researchers should explain what data items will be collected, and receive explicit consent from users. • Limited Data Pre-processing: The pre-processing of data (including real-time data processing) must be performed to the extent necessary for the research purposes notiﬁed and conveyed to the users in advance.

58https://play.google.com/store 59https://support.google.com/googleplay/android-developer/answer/7218994?hl=ko 55https://web.archive.org/web/20090331134113/ 56https://gdpr.eu/tag/gdpr/ 57https://www.privacyshield.gov/EU-US-Framework

30 A PREPRINT -APRIL 26, 2021

• Limited Data Usage: The collected data should be used only for the purposes agreed by the users (and approved by the IRB and app), which should be announced prior to data collection, unless the purpose of the research is truly for open data collection. • Limited Data Retention: The collected data should be retained only for a period agreed by the participants as speciﬁed in the app and IRB document. • Data Security Safeguards: Data should be protected in all processes (e.g., data collection, use, access, storage, share) to ensure that unauthorized access, processing, loss, disposal, destruction, use, modiﬁcation, damage, and disclosure are not allowed by ensuring reasonable technical and administrative security safeguards.

7.6 Limitation and Future Work

This article surveyed the studies using APIs that collect app usage patterns and other smartphone data in Android. As iOS is not open-source under the current iOS policy, customization is more difficult compared to Android, and therefore, it is not easy to collect user data and data-driven analytics in the iOS environment. However, the global mobile operating system share of iOS devices in 2020 was about 15% [6], which indicates that it is still widely used by users. Therefore, future research should investigate the existing studies related to collecting and analyzing smartphone user data for future research in the iOS environment. Furthermore, although we have separately identified the papers that entailed user-studies, there is a still limitation in that this review did not investigate the smartphone data collection conditions such as the number of participants, experiment duration, laboratory/field testing, questionnaires data (including experience sampling method data) for IP1 studies. In future studies, research based on the mobile context in the Android environment can be investigated by using keywords different to the ones used in this article. To understand the personal lifelogs, in addition to the app usage pattern, built-in sensors (e.g., Wi-Fi, GPS) and other APIs (e.g., Google Activity Recognition, SensorManager) that are frequently used in the studies surveyed in this article to collect user context data can be included as keywords. The surveyed studies utilizing the AS/US APIs were investigated to find studies that effectively utilized app usage patterns. However, although the app usage pattern was collected in the Android environment, studies that used their own developed API names instead of names of describing related to the AS or US API were excluded from the review such as References [163, 164, 165, 166, 39, 40, 43, 45, 47, 48]. Therefore, it is expected that further, broader insights can be obtained by subsequent researchers if studies that collected the app usage patterns, despite not describing AS/US APIs in their papers, are included in the review scope.

8 Conclusion

In this study, we focused on studies that employed Android APIs to collect app usage patterns, AS/US APIs, which are the most representative forms of app usage data that can identify personal lifelogs in Android environments. Through this, we identified the overall landscape of which types of data were collected by the AS/US APIs for used for diverse research purposes. In addition, we understand the Android built-in sensors and APIs used for device, system, and physical context sensing, and the types of mobile usage and sensor data that were collected through the Android APIs. First, we provided the categorization of research purposes through affinity diagramming by investigating existing studies that used Android APIs. Further, we presented the mobile usage and sensor data categorization for each type of data used in the papers with a hierarchy structure to make it easier for researchers to understand. Thus our study provides systematic guidelines for further research utilizing Android APIs centering on AS/US APIs in subsequent application fields. Furthermore, through a detailed study of the data types used for each research purpose, the types of mobile usage and sensor data that can be used for each research purpose were presented. Through this, we intend to contribute to the promotion of research reproducibility and activation of follow-up research by subsequent researchers.

References

[1] Stadd Allison. 79% of people 18–44 have their smartphones with them 22 hours a day, 2013. [2] Jory MacKay. Screen time stats 2019: Here’s how much you use your phone during the workday, 2019. [3] Lampros C Kourtis, Oliver B Regele, Justin M Wright, and Graham B Jones. Digital biomarkers for alzheimer’s disease: the mobile/wearable devices opportunity. NPJ digital medicine, 2(1):1–9, 2019. [4] Yunji Liang, Xiaolong Zheng, and Daniel D Zeng. A survey on big data-driven digital phenotyping of mental health. Information Fusion, 52:290–307, 2019. [5] Statista. Number of smartphone users worldwide from 2016 to 2021, 2020.

31 A PREPRINT -APRIL 26, 2021

[6] IDC. Smartphone market share, 2020. [7] Michael Winnick. Putting a finger on our phone obsession, 2016. [8] Android Developer. android.view.accessibility, 2020. [9] Android Developer. Accessibilityservice, 2020. [10] Android Developer. Android.app.usage, 2020. [11] Android Developer. Usagestatsmanager, 2020. [12] Anandi Dutta and Magdy Bayoumi. Introducing a novel smart design framework for a reconfigurable multi- processor systems-on-chip (mpsoc) architecture. In 2016 IEEE International Conference on Smart Computing (SMARTCOMP), pages 1–3. IEEE, 2016. [13] Aman Kr Singh, Ashish Kr Prajapati, Vikash Kumar, and Subhankar Mishra. Usage analysis of mobile devices. Procedia computer science, 122:657–662, 2017. [14] Jemin Lee, Jinse Kwon, and Hyungshin Kim. Reducing smartwatch users’ distraction with convolutional neural network. Mobile Information Systems, 2018, 2018. [15] Abhinav Mehrotra, Sandrine R Müller, Gabriella M Harari, Samuel D Gosling, Cecilia Mascolo, Mirco Musolesi, and Peter J Rentfrow. Understanding the role of places and activities on mobile phone interaction and usage patterns. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–22, 2017. [16] Yung-Ju Chang and John C Tang. Investigating mobile users’ ringer mode usage and attentiveness and responsiveness to communication. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 6–15, 2015. [17] Inyeop Kim, Rihun Kim, Heepyung Kim, Duyeon Kim, Kyungsik Han, Paul H Lee, Gloria Mark, and Uichin Lee. Understanding smartphone usage in college classrooms: A long-term measurement study. Computers & Education, 141:103611, 2019. [18] Seokjun Lee, Rhan Ha, and Hojung Cha. Click sequence prediction in android mobile applications. IEEE Transactions on Human-Machine Systems, 49(3):278–289, 2018. [19] Che-Hsuan Yu, Hung-Yuan Chen, Fang-Yie Leu, and Yao-Chung Fan. Understanding mobile user intent using attentive sequence-to-sequence rnns. In International Wireless Internet Conference, pages 51–61. Springer, 2019. [20] Karen Church, Denzil Ferreira, Nikola Banovic, and Kent Lyons. Understanding the challenges of mobile phone usage data. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 504–514, 2015. [21] Uichin Lee, Joonwon Lee, Minsam Ko, Changhun Lee, Yuhwan Kim, Subin Yang, Koji Yatani, Gahgene Gweon, Kyong-Mee Chung, and Junehwa Song. Hooked on smartphones: an exploratory study on smartphone overuse among college students. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 2327–2336, 2014. [22] Tobias Blanke, Giles Greenway, Jennifer Pybus, and Mark Coté. Mining mobile youth cultures. In 2014 IEEE International Conference on Big Data (Big Data), pages 14–17. IEEE, 2014. [23] Aku Visuri, Niels van Berkel, Tadashi Okoshi, Jorge Goncalves, and Vassilis Kostakos. Understanding smartphone notifications’ user interactions and content importance. International Journal of Human-Computer Studies, 128:72–85, 2019. [24] Martin Pielot, Rodrigo De Oliveira, Haewoon Kwak, and Nuria Oliver. Didn’t you see my message? predicting attentiveness to mobile instant messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3319–3328, 2014. [25] Swadhin Pradhan, Lili Qiu, Abhinav Parate, and Kyu-Han Kim. Understanding and managing notifications. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications, pages 1–9. IEEE, 2017. [26] Martin Pielot, Bruno Cardoso, Kleomenis Katevas, Joan Serrà, Aleksandar Matic, and Nuria Oliver. Beyond interruptibility: Predicting opportune moments to engage mobile phone users. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–25, 2017. [27] Seungchul Lee, Saumay Pushp, Chulhong Min, and Junehwa Song. Exploring relationship-aware dynamic message screening for mobile messengers. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pages 134–137, 2018.

32 A PREPRINT -APRIL 26, 2021

[28] Tadashi Okoshi, Hiroki Nozaki, Jin Nakazawa, Hideyuki Tokuda, Julian Ramos, and Anind K Dey. Towards attention-aware adaptive notification on smart phones. Pervasive and Mobile Computing, 26:17–34, 2016. [29] Chunjong Park, Junsung Lim, Juho Kim, Sung-Ju Lee, and Dongman Lee. Don’t bother me. i’m socializing! a breakpoint-based smartphone notification system. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pages 541–554, 2017. [30] Tadashi Okoshi, Julian Ramos, Hiroki Nozaki, Jin Nakazawa, Anind K Dey, and Hideyuki Tokuda. Attelia: Reducing user’s cognitive load due to interruptive notifications on smart phones. In 2015 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 96–104. IEEE, 2015. [31] Martin Pielot, Karen Church, and Rodrigo De Oliveira. An in-situ study of mobile phone notifications. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services, pages 233–242, 2014. [32] Yung-Ju Chang, Yi-Ju Chung, Yi-Hao Shih, Hsiu-Chi Chang, and Tzu-Hao Lin. What do smartphone users do when they sense phone notifications? In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pages 904–909, 2017. [33] Kyohei Komuro, Yuichiro Fujimoto, and Kinya Fujita. Relationship between worker interruptibility and work transitions detected by smartphone. In International Conference on Human-Computer Interaction, pages 687–699. Springer, 2017. [34] Christoph Anderson, Judith S Heinisch, Sandra Ohly, Klaus David, and Veljko Pejovic. The impact of private and work-related smartphone usage on interruptibility. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, pages 1058–1063, 2019. [35] Hao-Ping Lee, Kuan-Yin Chen, Chih-Heng Lin, Chia-Yu Chen, Yu-Lin Chung, Yung-Ju Chang, and Chien-Ru Sun. Does who matter? studying the impact of relationship characteristics on receptivity to mobile im messages. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2019. [36] Tilman Dingler, Dominik Weber, Martin Pielot, Jennifer Cooper, Chung-Cheng Chang, and Niels Henze. Language learning on-the-go: opportune moments and design of mobile microlearning sessions. In Proceedings of the 19th international conference on human-computer interaction with mobile devices and services, pages 1–12, 2017. [37] Jó Ágila Bitsch, Roann Ramos, Tim Ix, Paula Glenda Ferrer-Cheng, and Klaus Wehrle. Psychologist in a pocket: Towards depression screening on mobile phones. In pHealth, pages 153–159, 2015. [38] John Rooksby, Alistair Morrison, and Dave Murray-Rust. Student perspectives on digital phenotyping. 2019. [39] Saeed Abdullah, Mark Matthews, Elizabeth L Murnane, Geri Gay, and Tanzeem Choudhury. Towards circadian computing: " early to bed and early to rise" makes some of us unhealthy and sleep deprived. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing, pages 673–684, 2014. [40] Joost Asselbergs, Jeroen Ruwaard, Michal Ejdys, Niels Schrader, Marit Sijbrandij, and Heleen Riper. Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: an explorative study. Journal of medical Internet research, 18(3):e72, 2016. [41] Dror Ben-Zeev, Emily A Scherer, Rui Wang, Haiyi Xie, and Andrew T Campbell. Next-generation psychiatric assessment: Using smartphone sensors to monitor behavior and mental health. Psychiatric rehabilitation journal, 38(3):218, 2015. [42] Mehdi Boukhechba, Yu Huang, Philip Chow, Karl Fua, Bethany A Teachman, and Laura E Barnes. Monitoring social anxiety from mobility and communication patterns. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pages 749–753, 2017. [43] Larry Chan, Vedant Das Swain, Christina Kelley, Kaya de Barbaro, Gregory D Abowd, and Lauren Wilcox. Stu- dents’ experiences with ecological momentary assessment tools to report on emotional well-being. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(1):1–20, 2018. [44] Zhenyu Chen, Mu Lin, Fanglin Chen, Nicholas D Lane, Giuseppe Cardone, Rui Wang, Tianxing Li, Yiqiang Chen, Tanzeem Choudhury, and Andrew T Campbell. Unobtrusive sleep monitoring using smartphones. In 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops, pages 145–152. IEEE, 2013. [45] Paul Eskes, Marco Spruit, Sjaak Brinkkemper, Jacob Vorstman, and Martien J Kas. The sociability score: App-based social profiling from a healthcare perspective. Computers in Human Behavior, 59:39–48, 2016.

33 A PREPRINT -APRIL 26, 2021

[46] Vivek K Singh and Rishav R Agarwal. Cooperative phoneotypes: exploring phone-based behavioral markers of cooperation. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 646–657, 2016. [47] Thomas Stütz, Thomas Kowar, Michael Kager, Martin Tiefengrabner, Markus Stuppner, Jens Blechert, Frank H Wilhelm, and Simon Ginzinger. Smartphone based stress prediction. In International Conference on User Modeling, Adaptation, and Personalization, pages 240–251. Springer, 2015. [48] Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell. Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing, pages 3–14, 2014. [49] Billy Lau, Simon Chung, Chengyu Song, Yeongjin Jang, Wenke Lee, and Alexandra Boldyreva. Mimesis aegis: A mimicry privacy shield–a system’s approach to data privacy on public cloud. In 23rd {USENIX} Security Symposium ({USENIX} Security 14), pages 33–48, 2014. [50] Kassem Fawaz, Kyu-Han Kim, and Kang G Shin. Privacy vs. reward in indoor location-based services. Proceedings on Privacy Enhancing Technologies, 2016(4):102–122, 2016. [51] Ahmet Talha Ozcan, Can Gemicioglu, Kaan Onarlioglu, Michael Weissbacher, Collin Mulliner, William Robertson, and Engin Kirda. Babelcrypt: The universal encryption layer for mobile messaging applications. In International Conference on Financial Cryptography and Data Security, pages 355–369. Springer, 2015. [52] Shravan Aras, Chris Gniady, and Hari Venugopalan. Multilock: biometric-based graded authentication for mobile devices. In Proceedings of the 16th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, pages 100–109, 2019. [53] Lydia Kraus, Robert Schmidt, Marcel Walch, Florian Schaub, and Sebastian Möller. On the use of emojis in mobile authentication. In IFIP International Conference on ICT Systems Security and Privacy Protection, pages 265–280. Springer, 2017. [54] Gerardo Canfora, Paolo Di Notte, Francesco Mercaldo, and Corrado Aaron Visaggio. Silent and continuous authentication in mobile environment. In SECRYPT, pages 97–108, 2016. [55] Md Lutfor Rahman, Ajaya Neupane, and Chengyu Song. Iac: On the feasibility of utilizing neural signals for access control. In Proceedings of the 34th Annual Computer Security Applications Conference, pages 641–652, 2018. [56] Bram Bonné, Sai Teja Peddinti, Igor Bilogrevic, and Nina Taft. Exploring decision making with android’s runtime permission dialogs using in-context surveys. In Thirteenth Symposium on Usable Privacy and Security ({SOUPS} 2017), pages 195–210, 2017. [57] Mohammad Naseri, Nataniel P Borges, Andreas Zeller, and Romain Rouvoy. Accessileaks: Investigating privacy leaks exposed by the android accessibility service. Proceedings on Privacy Enhancing Technologies, 2019(2):291–305, 2019. [58] Yafei Li, Jiageng Chen, and Anthony TS Ho. Chatting application monitoring on android system and its detection based on the correlation test. In 2018 Asia-Paciﬁc Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1556–1563. IEEE, 2018. [59] Nishant Vishwamitra, Xiang Zhang, Jonathan Tong, Hongxin Hu, Feng Luo, Robin Kowalski, and Joseph Mazer. Mcdefender: Toward effective cyberbullying defense in mobile online social networks. In Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics, pages 37–42, 2017. [60] Rui Shao, Vaibhav Rastogi, Yan Chen, Xiang Pan, Guanyu Guo, Shihong Zou, and Ryan Riley. Understanding in-app ads and detecting hidden attacks through the mobile app-web interface. IEEE Transactions on Mobile Computing, 17(11):2675–2688, 2018. [61] Fanglin Chen, Kewei Xia, Karan Dhabalia, and Jason I Hong. Messageontap: A suggestive interface to facilitate messaging-related tasks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2019. [62] Woo-Jong Ryu, HyeonTaek Oh, and SangKeun Lee. siginterface: a personalized interface for intelligent services. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pages 185–188, 2017. [63] Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M Mitchell, and Brad A Myers. Appinite: A multi-modal interface for specifying data descriptions in programming by demonstration using natural language instructions. In 2018 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC), pages 105–114. IEEE, 2018.

34 A PREPRINT -APRIL 26, 2021

[64] Xiaoyi Zhang, Anne Spencer Ross, Anat Caspi, James Fogarty, and Jacob O Wobbrock. Interaction proxies for runtime repair and enhancement of mobile application accessibility. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 6024–6037, 2017. [65] Zachary Rauen, Fazel Anjomshoa, and Burak Kantarci. Empowering human-computer interaction in securing smartphone sensing. In 2018 IEEE 23rd International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), pages 1–6. IEEE, 2018. [66] Andreas Riegler and Clemens Holzmann. Measuring visual user interface complexity of mobile applications with metrics. Interacting with Computers, 30(3):207–223, 2018. [67] Andreas Riegler and Clemens Holzmann. Ui-cat: calculating user interface complexity metrics for mobile applications. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia, pages 390–394, 2015. [68] Syuan-Yi Lin and Chung-Ta King. User-centered context-aware cpu/gpu power management for interactive applications on smartphones. In Proceedings of the 16th ACM International Conference on Computing Frontiers, pages 247–250, 2019. [69] Ahmad Bisher Tarakji, Jian Xu, Juan A Colmenares, and Iqbal Mohomed. Voice enabling mobile applications with uivoice. In Proceedings of the 1st International Workshop on Edge Systems, Analytics and Networking, pages 49–54, 2018. [70] Hanan H Elazhary and Sahar F Sabbeh. The w 5 framework for computation offloading in the internet of things. IEEE Access, 6:23883–23895, 2018. [71] Ke Sun, Chun Yu, Weinan Shi, Lan Liu, and Yuanchun Shi. Lip-interact: Improving mobile device interaction with silent speech commands. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pages 581–593, 2018. [72] Ru Wang, Shizhan Chen, Zhiyong Feng, and Keman Huang. A client microservices automatic collaboration framework based on fine-grained app. In 2018 IEEE International Conference on Services Computing (SCC), pages 25–32. IEEE, 2018. [73] Filipe Arruda, Augusto Sampaio, and Flávia A Barros. Capture & replay with text-based reuse and framework agnosticism. In SEKE, pages 420–425, 2016. [74] Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. Practical gui testing of android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 269–280. IEEE, 2019. [75] Woramet Muangsiri and Shingo Takada. Random gui testing of android application using behavioral model. International Journal of Software Engineering and Knowledge Engineering, 27(09n10):1603–1612, 2017. [76] Jose Lorenzo San Miguel and Shingo Takada. Gui and usage model-based test case generation for android applications with change analysis. In Proceedings of the 1st International Workshop on Mobile Development, pages 43–44, 2016. [77] Guoquan Wu, Yuzhong Cao, Wei Chen, Jun Wei, Hua Zhong, and Tao Huang. Appcheck: a crowdsourced testing service for android applications. In 2017 IEEE International Conference on Web Services (ICWS), pages 253–260. IEEE, 2017. [78] Hao Lian, Zemin Qin, Hangcheng Song, and Tieke He. E-cat: Evaluating crowdsourced android testing. In International Conference of Pioneering Computer Scientists, Engineers and Educators, pages 493–504. Springer, 2018. [79] Pascal Bissig, Gino Brunner, Florian Gubler, Roger Wattenhofer, and Andreas Zingg. Towards measuring real-world performance of android devices. In 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), pages 1–8. IEEE, 2018. [80] Stas Negara, Naeem Esfahani, and Raymond Buse. Practical android test recording with espresso test recorder. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 193–202. IEEE, 2019. [81] Kuei-Chun Liu, Yu-Yu Lai, and Ching-Hong Wu. A mechanism of reliable and standalone script generator on android. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 372–374. IEEE, 2017. [82] Mattia Fazzini, Eduardo Noronha de A Freitas, Shauvik Roy Choudhary, and Alessandro Orso. From manual android tests to automated and platform independent test scripts. arXiv preprint arXiv:1608.03624, 2016.

35 A PREPRINT -APRIL 26, 2021

[83] Mattia Fazzini, Eduardo Noronha De A Freitas, Shauvik Roy Choudhary, and Alessandro Orso. Barista: A technique for recording, encoding, and running platform independent android tests. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 149–160. IEEE, 2017. [84] Toby Jia-Jun Li and Oriana Riva. Kite: Building conversational bots from mobile apps. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pages 96–109, 2018. [85] Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A Myers. Programming iot devices by demonstration using mobile apps. In International Symposium on End User Development, pages 3–17. Springer, 2017. [86] Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. Sugilite: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 6038–6049, 2017. [87] Clemens Holzmann, Dustin Steiner, Andreas Riegler, and Christian Grossauer. An android toolkit for supporting field studies on mobile devices. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia, pages 473–479, 2017. [88] Kyle Montague, André Rodrigues, Hugo Nicolau, and Tiago Guerreiro. Tinyblackbox: Supporting mobile in-the-wild studies. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility, pages 379–380, 2015. [89] Niels Van Berkel, Denzil Ferreira, and Vassilis Kostakos. The experience sampling method on mobile devices. ACM Computing Surveys (CSUR), 50(6):1–40, 2017. [90] Android Developer. Documentation for app developers, 2020. [91] Android Developer. Platform architecture, 2020. [92] Android Developer. Context, 2020. [93] Android Developer. Bluetooth overview, 2020. [94] Android Developer. android.location, 2020. [95] Android Developer. Create your own accessibility service, 2020. [96] Google. Talkback, 2020. [97] Android Developer. Accessibilityevent, 2020. [98] AndroidDeveloper. Activitymanager, 2020. [99] Android Developer. Usageevents.event, 2020. [100] Jemin Lee, Uichin Lee, and Hyungshin Kim. Pass: Reducing redundant interactions between a smartphone and a smartwatch for energy saving. IEEE Transactions on Mobile Computing, 2019. [101] Pascal Welke, Ionut Andone, Konrad Blaszkiewicz, and Alexander Markowetz. Differentiating smartphone users by app usage. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 519–523, 2016. [102] Ionut Andone, Konrad Błaszkiewicz, Mark Eibes, Boris Trendafilov, Christian Montag, and Alexander Markowetz. Menthal: a framework for mobile data collection and analysis. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pages 624–629, 2016. [103] Immanuel Schweizer and Benedikt Schmidt. Kraken. me: multi-device user tracking suite. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, pages 853–862, 2014. [104] Immanuel Schweizer, Roman Bärtl, Benedikt Schmidt, Fabian Kaup, and Max Mühlhäuser. Kraken. me mobile: the energy footprint of mobile tracking. In 6th International Conference on Mobile Computing, Applications and Services, pages 82–89. IEEE, 2014. [105] Android Developer. android.hardware, 2020. [106] Android Developer. Sensors overview, 2020. [107] Denzil Ferreira, Vassilis Kostakos, and Anind K Dey. Aware: mobile context instrumentation framework. Frontiers in ICT, 2:6, 2015. [108] Google Developer. Detectedactivity, 2020. [109] Android Developer. Detect when users start or end an activity, 2020.

36 A PREPRINT -APRIL 26, 2021

[110] Soban Ahmed Khan, Asma Ahmad Farhan, Labiba Gillani Fahad, and Syed Fahad Tahir. Personal productivity monitoring through smartphones. Journal of Ambient Intelligence and Smart Environments, (Preprint):1–15, 2020. [111] Android Developer. Locationlistener, 2020. [112] Android Developer. android.bluetooth, 2020. [113] Android Developer. android.bluetooth.le, 2020. [114] Android Developer. Trafficstats, 2020. [115] Android Developer. Networkstatsmanager, 2020. [116] Android Developer. Broadcast overview, 2020. [117] Android Developer. Intent, 2020. [118] Android Developer. Monitor the battery level and charging state, 2020. [119] David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, Prisma Group, et al. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS med, 6(7):e1000097, 2009. [120] Karen Holtzblatt, Jessamyn Burns Wendell, and Shelley Wood. Rapid contextual design: a how-to guide to key techniques for user-centered design. Elsevier, 2004. [121] Denzil Ferreira, Jorge Goncalves, Vassilis Kostakos, Louise Barkhuus, and Anind K Dey. Contextual experience sampling of mobile application micro-usage. In Proceedings of the 16th international conference on Human- computer interaction with mobile devices & services, pages 91–100, 2014. [122] Nalingna Yuan, Heidi M Weeks, Rosa Ball, Mark W Newman, Yung-Ju Chang, and Jenny S Radesky. How much do parents actually use their smartphones? pilot study comparing self-report to passive sensing. Pediatric research, 86(4):416–418, 2019. [123] Xiangang Qin, Chee-Wee Tan, Effie Lai-Chong Law, Mads Bødker, Torkil Clemmensen, Hequn Qu, and Diandi Chen. Deciphering the role of context in shaping mobile phone usage: Design recommendations for context- aware mobile services from a cross-cultural perspective. In Proceedings of the Sixth International Symposium of Chinese CHI, pages 39–48, 2018. [124] Xiangang Qin, Dechuan Wang, Torkil Clemmensen, Diandi Chen, and Yanuan Zhao. The association of actual behavior on wechat with psychological traits: evidence with mixed method. In Proceedings of the Seventh International Symposium of Chinese CHI, pages 49–56, 2019. [125] Jenny S Radesky, Heidi M Weeks, Rosa Ball, Alexandria Schaller, Samantha Yeo, Joke Durnez, Matthew Tamayo-Rios, Mollie Epstein, Heather Kirkorian, Sarah Coyne, et al. Young children’s use of smartphones and tablets. Pediatrics, 2020. [126] Kaige Yan, Jingweijia Tan, and Xin Fu. Bridging mobile device configuration to the user experience under budget constraint. Pervasive and Mobile Computing, 58:101023, 2019. [127] Ming-Yi Zheng, Hung-Yuan Chen, Huan Chen, and Yao-Chung Fan. On cleaning and organizing context logs for mobile user profiling. In 2017 Twelfth International Conference on Digital Information Management (ICDIM), pages 161–164. IEEE, 2017. [128] Tiantian Zhu, Zhengyang Qu, Haitao Xu, Jingsi Zhang, Zhengyue Shao, Yan Chen, Sandeep Prabhakar, and Jianfeng Yang. Riskcog: Unobtrusive real-time user authentication on mobile devices in the wild. IEEE Transactions on Mobile Computing, 19(2):466–483, 2019. [129] Tiantian Zhu, Zhengqiu Weng, Guolang Chen, and Lei Fu. A hybrid deep learning system for real-world mobile user authentication using motion sensors. Sensors, 20(14):3876, 2020. [130] José Torres, Sergio de los Santos, Efthimios Alepis, and Constantinos Patsakis. Behavioral biometric authentication in android unlock patterns through machine learning. In ICISSP, pages 146–154, 2019. [131] Panagiotis Andriotis and Atsuhiro Takasu. To allow, or deny? that is the question. In International Conference on Human-Computer Interaction, pages 287–304. Springer, 2020. [132] Earlence Fernandes, Oriana Riva, and Suman Nath. Appstract: on-the-fly app content semantics with better privacy. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, pages 361–374, 2016. [133] Vaibhav Rastogi, Rui Shao, Yan Chen, Xiang Pan, Shihong Zou, and Ryan Riley. Are these ads safe: Detecting hidden attacks through the mobile app-web interfaces. In NDSS, 2016.

37 A PREPRINT -APRIL 26, 2021

[134] Yonas Leguesse, Mark Vella, Christian Colombo, and Julio Hernandez-Castro. Reducing the forensic footprint with android accessibility attacks. In International Workshop on Security and Trust Management, pages 22–38. Springer, 2020. [135] Yingjie Wang, Xing Liu, Weixuan Mao, and Wei Wang. Dcdroid: automated detection of ssl/tls certificate verification vulnerabilities in android apps. In Proceedings of the ACM Turing Celebration Conference-China, pages 1–9, 2019. [136] Yingjie Wang, Guangquan Xu, Xing Liu, Weixuan Mao, Chengxiang Si, Witold Pedrycz, and Wei Wang. Identifying vulnerabilities of ssl/tls certificate verification in android apps with static and dynamic analysis. Journal of Systems and Software, page 110609, 2020. [137] Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. Interactive task and concept learning from natural language instructions and gui demonstrations. In The AAAI-20 Workshop on Intelligent Process Automation (IPA-20), 2020. [138] Zhengzhe Xiang, Shuiguang Deng, Javid Taheri, and Albert Zomaya. Dynamical service deployment and replacement in resource-constrained edges. Mobile Networks and Applications, 25(2):674–689, 2020. [139] Yu Zhong, Astrid Weber, Casey Burkhardt, Phil Weaver, and Jeffrey P Bigham. Enhancing android accessibility for users with hand tremor by reducing fine pointing and steady tapping. In Proceedings of the 12th Web for All Conference, pages 1–10, 2015. [140] André Rodrigues. Breaking barriers with assistive macros. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility, pages 351–352, 2015. [141] Sandesh Chinchole and Samir Patel. Artificial intelligence and sensors based assistive system for the visually impaired people. In 2017 International Conference on Intelligent Sustainable Systems (ICISS), pages 16–19. IEEE, 2017. [142] Bruna de Oliveira, Juliana Cristina Braga, and Rafael J Pezzuto Damaceno. Application for the configuration and adaptation of the android operating system for the visually impaired. In Proceedings of the Internet of Accessible Things, pages 1–4. 2018. [143] John Brooke. Sus: a “quick and dirty’usability. Usability evaluation in industry, 189, 1996. [144] André Rodrigues, Leonardo Camacho, Hugo Nicolau, Kyle Montague, and Tiago Guerreiro. Aidme: Interactive non-visual smartphone tutorials. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pages 205–212, 2018. [145] Daniel Trindade, André Rodrigues, Tiago Guerreiro, and Hugo Nicolau. Hybrid-brailler: combining physical and gestural interaction for mobile braille input and editing. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–12, 2018. [146] Xiaoyi Zhang, Anne Spencer Ross, and James Fogarty. Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pages 609–621, 2018. [147] Yu Zhong, TV Raman, Casey Burkhardt, Fadi Biadsy, and Jeffrey P Bigham. Justspeak: enabling universal voice control on android. In Proceedings of the 11th Web for All Conference, pages 1–4, 2014. [148] Mrim Alnfiai and Srinivas Sampalli. Brailletap: Developing a calculator based on braille using tap gestures. In International Conference on Universal Access in Human-Computer Interaction, pages 213–223. Springer, 2017. [149] Runze Chen, Zhanhong Tian, Hailun Liu, Fang Zhao, Shuai Zhang, and Haobo Liu. Construction of a voice driven life assistant system for visually impaired people. In 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pages 87–92. IEEE, 2018. [150] Ana GD Correa, Laisa CC De Biase, Erich P Lotto, and Roseli D Lopes. Development and usability evaluation of an configurable educational game for the visually impaired. In 2018 IEEE Games, Entertainment, Media Conference (GEM), pages 1–9. IEEE, 2018. [151] MJ Samonte, ED Laurente, KM Magno, and C Perez. Braille3d: using haptic and voice feedback for braille recognition and 3d printing for the blind. MS&E, 482(1):012027, 2019. [152] Eduardo Ghidini, Wagner DL Almeida, Isabel H Manssour, and Milene S Silveira. Developing apps for visually impaired people: Lessons learned from practice. In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 5691–5700. IEEE, 2016. [153] Hye-Yeon Hann, Yoon-Seok Jeong, and Won-Hong Jang. Best practices of the daisy standards in korea-lg sangnam library’s service for the print disabled. 2013.

38 A PREPRINT -APRIL 26, 2021

[154] Aura Ganz, James M Schafer, Yang Tao, Carole Wilson, and Meg Robertson. Percept-ii: Smartphone based indoor navigation system for the blind. In 2014 36th annual international conference of the IEEE engineering in medicine and biology society, pages 3662–3665. IEEE, 2014. [155] Sujeath Pareddy, Anhong Guo, and Jeffrey P Bigham. X-ray: Screenshot accessibility via embedded metadata. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pages 389–395, 2019. [156] Krzysztof Dobosz and Jakub Ptak. How to control a mobile game. In International Conference on Computers Helping People with Special Needs, pages 523–529. Springer, 2016. [157] André Rodrigues, Kyle Montague, Hugo Nicolau, and Tiago Guerreiro. Getting smartphones to talkback: Understanding the smartphone adoption process of blind users. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility, pages 23–32, 2015. [158] Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. Accessibility issues in android apps: state of affairs, sentiments, and ways forward. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pages 1323–1334. IEEE, 2020. [159] Android Developer. Android developer, documentation for app developer, 2020. [160] Corbin Davenport. Google will remove play store apps that use accessibility services for anything except helping disabled users, 2017. [161] Wenrui Diao, Yue Zhang, Li Zhang, Zhou Li, Fenghao Xu, Xiaorui Pan, Xiangyu Liu, Jian Weng, Kehuan Zhang, and XiaoFeng Wang. Kindness is a risky business: On the usage of the accessibility apis in android. In 22nd International Symposium on Research in Attacks, Intrusions and Defenses ({RAID} 2019), pages 261–275, 2019. [162] Google. Privacy, deception and device abuse, 2020. [163] Hossein Falaki, Ratul Mahajan, Srikanth Kandula, Dimitrios Lymberopoulos, Ramesh Govindan, and Deborah Estrin. Diversity in smartphone usage. In Proceedings of the 8th international conference on Mobile systems, applications, and services, pages 179–194, 2010. [164] Nikola Banovic, Christina Brant, Jennifer Mankoff, and Anind Dey. Proactivetasks: the short of mobile device use sessions. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services, pages 243–252, 2014. [165] Denzil Ferreira, Vassilis Kostakos, Alastair R Beresford, Janne Lindqvist, and Anind K Dey. Securacy: an empirical investigation of android applications’ network usage, privacy and security. In Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks, pages 1–11, 2015. [166] Choonsung Shin, Jin-Hyuk Hong, and Anind K Dey. Understanding and prediction of mobile application usage for smart phones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 173–182, 2012.