Smart Devices are Different: Assessing and Mitigating Mobile Sensing Heterogeneities for Activity Recognition Allan Stisenz, Henrik Blunckz, Sourav Bhattacharya∗, Thor Siiger Prentowz, Mikkel Baun Kjærgaardz, Anind Deyy, Tobias Sonnez, and Mads Møller Jensenz zDepartment of Computer Science, Aarhus University, Denmark ∗Bell Laboratories, Dublin, Ireland yCarnegie Mellon University, USA {allans, blunck, prentow, mikkelbk, tsonne, mmjensen}@cs.au.dk [email protected], [email protected] mobile devices in our everyday lives provides a unique opportunity to unobtrusively capture contextual information from the underlying ABSTRACT human behavior in real-time. This growth has also led to an easier development, deployment and a wide proliferation of publicly available The widespread presence of motion sensors on users’ personal mo- mobile sensing applications. Novel mobile sensing applications have bile devices has spawned a growing research interest in human activity also opened up new possibilities for mobile sensing research. recognition (HAR). However, when deployed at a large-scale, e.g., on Among the sensors available on mobile consumer device platforms, multiple devices, the performance of a HAR system is often signifi- the accelerometer is one of the earliest and most ubiquitous. The ac- cantly lower than in reported research results. This is due to variations celerometer has gained immense popularity in HAR research as it allows in training and test device hardware and their operating system char- recognizing a wide variety of human activities, while having a relatively acteristics among others. In this paper, we systematically investigate small energy footprint [8]. Accelerometer-based HAR has been de- sensor-, device- and workload-specific heterogeneities using 36 smart- ployed in a large number of domains including smart homes [5, 32, 38], phones and smartwatches, consisting of 13 different device models from health care [31, 37], daily activity tracking [29], fitness tracking [16], fall four manufacturers. Furthermore, we conduct experiments with nine detection of elderly people [18] and transportation mode detection [9, users and investigate popular feature representation and classification 24, 42]. Other motion-related sensors, such as compass and gyroscope, techniques in HAR research. Our results indicate that on-device sensor are becoming increasingly common-place and often used for assisting and sensor handling heterogeneities impair HAR performances signif- and complementing the accelerometer. Motion sensors can be further icantly. Moreover, the impairments vary significantly across devices paired with sensors, e.g., GPS, GSM, WiFi, and barometer, especially and depends on the type of recognition technique used. We system- for recognizing tasks beyond basic HAR. atically evaluate the effect of mobile sensing heterogeneities on HAR Although, a large body of motion sensor-based HAR research exists, and propose a novel clustering-based mitigation technique suitable for real-world performance variations across, e.g., device manufacturers, large-scale deployment of HAR, where heterogeneity of devices and models, OS types, and CPU load conditions, have been largely over- their usage scenarios are intrinsic. looked and not been evaluated rigorously yet. When a HAR system is deployed ‘in the wild’, i.e., across heterogeneous devices and usage Keywords situations, recognition performances are often significantly lower than Mobile Sensing, Activity Recognition what is suggested in previous research, as noted, e.g., by Amft [2] and Blunck et al. [13]. While device placement and orientation and differences in how users perform physical activities have been noted Categories and Subject Descriptors and mitigations for the adverse effects on HAR have been proposed, I.5.2 [Pattern Recognition]: Design Methodology; I.5.4 [Pattern heterogeneities across devices and their configurations have not been Recognition]: Signal processing studied rigorously. In this paper, we aim to bridge this gap in HAR research by systematically studying various heterogeneities in motion 1. INTRODUCTION sensor-based sensing, their impact on HAR, and propose mitigation solutions. Below we elaborate on three major types of heterogeneities, Off-the-shelf modern smartphones readily support an increasingly rich which yield impairments of HAR. set of embedded sensors such as accelerometer, gyroscope, compass, WiFi, NFC and GPS [30, 45]. This growing ubiquity of sensor rich Sensor Biases (SB): To keep the overall cost low, mobile devices are often equipped with low cost sensors, which are often poorly Permission to make digital or hard copies of all or part of this work for personal or calibrated, inaccurate, and of limited granularity and range, compared classroom use is granted without fee provided that copies are not made or distributed to dedicated sensors for HAR, e.g., a dedicated standalone Inertial for profit or commercial advantage and that copies bear this notice and the full citation Measurement Unit (IMU). Furthermore, sensor biases may shift over on the first page. Copyrights for components of this work owned by others than time through everyday device usage, e.g., accidental device dropping ACM must be honored. Abstracting with credit is permitted. To copy otherwise, might increase sensor biases. or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Sampling Rate Heterogeneity (SRH): As of 2014, on the Android SenSys’15, November 1–4, 2015, Seoul, South Korea.. 18;000 c 2015 ACM. ISBN 978-1-4503-3631-4/15/11 ...$15.00. platform alone there are more than distinct smartphone mod- DOI: http://dx.doi.org/10.1145/2809695.2809718. els [36]. Often popular smartphones vary in terms of the default and supported sampling frequencies for accelerometer and other sensors. To highlight this in Table 1 we summarize the supported maximum accelerometer sampling frequency across 36 devices, spanning over domain of aerospace engineering, and modeled bias as a combination 13 device models, used in our experiments. of time and temperature dependent variables [22]. By conducting a set of experiments in a controlled laboratory setting the author reported that Sampling Rate Instability (SRI): A number of factors, including 99% of the drifts could be eliminated. Batista et al. also proposed a cali- delays in OS level timestamp attachment to sensor measurements and bration technique, based on time-varying Kalman filtering and gravity es- instantaneous I/O load, affect both the actual as well as the reported timation, for estimating biases observed on micro-electrical-mechanical- sampling rate of sensors on a device. For example, heavy multitasking systems accelerometers (tri-axial) [7]. Through a set of simulation exper- and I/O load on mobile devices, as exhibited in typical usage scenarios, iments using a motion rate table the authors show good performance of often lead to unstable sampling rates as the mobile OS frequently their approach. Sensor biases are prevalent not only for accelerometers fails to attach accurate timestamp information as the measurements but also for other sensing modalities: For example, Barthold et al. studied arrive. Unpredictability (and inevitability) of such loads makes the the influence of heavy magnetic environments and motion on gyroscope- multitasking impairments challenging in HAR. Further irregularities based orientation estimations [6]. The authors also point out that offline in sampling rate may be introduced by modern sensing strategies. For sensor calibration techniques are hindered by that the sensor hetero- example APIs supporting continuous sensing on the mobile platforms geneities, e.g., biases and offsets, are often dependent on time and tem- often rely heavily on dynamic duty-cycling to lower the overall power perature, thereby limiting the applicability of such calibration techniques. consumption [8]. Within the robotics community, calibration of IMUs via sensor fusion has been investigated, utilizing cameras and multiple sensors modalities, In this paper, we empirically quantify the above heterogeneities, exper- e.g., gyroscopes, magnetometers and accelerometers [26]. Banos et al. imentally evaluate their effects on the HAR performance, and discuss report on translating—across several sensor modalities—HAR from mitigation of these impairments. In the following, we summarize the one sensor platform domain to another one, e.g., from a Kinect to an main contributions of our work: IMU [4]. Such methods could be used to align and potentially calibrate sensors to mitigate device heterogeneities. • We present several sources of intrinsic heterogeneities in mobile sens- ing, focusing on accelerometer sensors. We conduct an extensive anal- In regards to the heterogeneities as discussed herein, a number of ob- ysis of such heterogeneities using 31 smartphones, 4 smartwatches servations have been made in regards to the challenges these yield for and 1 tablet, representing 13 different models from 4 manufacturers, real-world deployments: Amft [2] notes that HAR performances are running variants of Android and iOS, respectively (see Section 3). often overly optimistic and that significant performance impairments may occur when utilized ‘in the wild’. Blunck et al. [13] give a taxon- • We systematically study the influence of these heterogeneities on omy of device-, user- and project-specific heterogeneities and sketch