ActDetector: Detecting Daily Activities Using Smartwatches Xiao Sun, Li Qiu, Yibo Wu and Guohong Cao Department of Computer Science and Engineering The Pennsylvania State University, University Park, PA xxs118, [email protected], [email protected], gcao @cse.psu.edu f g Abstract—Detecting daily activities is helpful for health care As a popular research area with potential applications in and clinical medicine. In this paper, we present ActDetector, a health care, activity detection has been studied in previous smartwatch based application which detects 8 common daily works [6]–[12]. Radio based systems have been proposed in activities, including sitting, walking, running, going upstairs, going downstairs, eating, driving and sitting in a vehicle. By [6]–[10] to detect human activities by using radar or Channel leveraging the built-in sensors on smartwatch, a multi-level State Information (CSI). However, these systems rely on the classification system is proposed which considers both detection wireless devices installed indoor, which makes them unsuitable accuracy and energy efficiency. ActDetector is designed to work for detecting activities outdoor. In [11], nodes with motion unobtrusively, no matter on which wrist the smartwatch is worn. sensors are placed at different parts of the body and data from We have implemented ActDetector on Sony Smartwatch 3 and evaluated its performance in real experiments involving 12 users. all these sensor nodes were collected for activity detection. Experimental results show that ActDetector is energy efficient Although the system can work both indoor and outdoor, it is and can detect the daily activities with high accuracy. not practical for daily monitoring since users have to wear multiple sensor nodes on their wrists or ankles. Smartphone I. INTRODUCTION based approaches are proposed in [12], [13], where inertial Activities such as sitting, walking or eating, reflect routine sensors equipped on the smartphone are exploited for activity movements in our daily lives. Monitoring these daily activi- detection. However, in these approaches, to achieve more ties provides helpful information for health care and clinical accurate detection, the smartphone has to be placed at a fixed medicine. For example, in [1], by analyzing the walking data position with respect to the body, which is inconvenient for of patients with stroke, researchers evaluated the efficacy of users. Some simple activities such as walking or running can a task-oriented intervention in stroke rehabilitation, and in be detected by commercial wrist-worn devices like Fitbit or [2], eating activity was monitored to help people with obesity Jawbone. However, to use these devices, users have to set assess dietary intake and establish healthy eating habits. Even the preferred wrist (i.e., the wrist on which the device is for people in good health, maintaining activity dairies for worn) manually and these devices cannot detect complicated over an extended period of time plays an important role activities such as eating or driving. in improving their fitness. For example, by detecting and Different from the aforementioned works, ActDetector is recording activities like sitting and walking, daily physical implemented on smartwatch to detect 8 common daily ac- activities can be checked to help decide if more physical tivities. Several practical issues are considered in developing exercises are needed or not. ActDetector to meet the specific requirements of smartwatch, The rapid development of wearable technologies such as such as sensing capabilities and resource constraints of smart- smartphones and smartwatches, has opened up opportunities watch, and different wearing habits of smartwatch users. The for smart health [3]–[5], and it is possible to detect and accelerometer and gyroscope data is collected and a multi-level record a user’s health related daily activities unobtrusively classification system is designed for activity detection. First, a by using wearable devices. Since smartwatch is integrated few time-domain features are extracted from the accelerometer with motion sensors (e.g., accelerometer, gyroscope, etc.) and data to detect the predominant activity sitting. Then, the re- worn by a user on his/her wrist most of the time, we can maining activities are classified into two categories, depending collect motion related sensor data from the smartwatch to on whether they are activities with repeated patterns or not. detect daily activities through analysis of wrist movement Lastly, more time-domain and frequency-domain features are patterns. In this paper, we present a smartwatch based system extracted and different classifiers are trained to identify each called ActDetector, which leverages two widely equipped daily activity. ActDetector is designed to work unobtrusively motion sensors, accelerometer and gyroscope, to detect 8 daily and robustly, and it functions no matter on which wrist the activities, including sitting, walking, running, going upstairs, smartwatch is worn. We have implemented ActDetector on going downstairs, eating, driving, and sitting in a vehicle. Act- Sony Smartwatch 3 and evaluated its performance in real Detector is designed to record these activities in an unobtrusive experiments involving 12 users. Experimental results show that way without interrupting users’ normal smartwatch wearing ActDetector is energy efficient and can detect daily activities and usage patterns. with high accuracy. Preferred Wrist Detection r a t Moving average filter Wrist down detection Left/right wrist e ) a t 15 l walking i 2 d f r s e s t wrist down a e running p 10 - m wrist up o w r o No Yes upstairs l e l Frames and Sitting Repeated pattern r e Feature interaction with smartwatch e c windows detection detection 5 d c downstairs r extraction A o - t d s s n r eating i s 0 a o Magnitude (m/s w F e s o N Daily Activity Y e d driving 0 5 10 15 20 25 n m Detection i Classifier a Time (s) w r sitting F vehicle Fig. 2: The magnitude of the total acceleration Sensor Data Collection Gyroscope data when starting an app on smartwatch. Fig. 1: The architecture of ActDetector. The rest of this paper is organized as follows. Section II are different under most circumstances. Therefore, ActDetec- discusses the design considerations. Section III presents the tor should be able to detect the daily activities no matter on system architecture and illustrates the design details of each which wrist the smartwatch is worn. component. The implementation of ActDetector on Android Wear based smartwatch is presented in Section IV. We eval- III. SYSTEM DESIGN uate the performance of our system in Section V. Section VI In this section, we present the design of ActDetector. As reviews the related work and Section VII concludes the paper. shown in Figure 1, there are three main components in Act- Detector. The sensor data is sampled from the accelerometer II. DESIGN CONSIDERATIONS and gyroscope for processing. By using the accelerometer data, ActDetector is designed to unobtrusively detect and record a light weight scheme is designed to detect on which wrist the a user’s daily activities using off-the-shelf smartwatches. To user wears the smartwatch. Based on the detected result, the achieve this goal, the following design issues should be corresponding classifiers are chosen and a group of features considered. are extracted from sensor data to identify each daily activity. First, the system should use sensors which are commonly The design details of each component are described as follows. built in different types of smartwatches. A lot of smartwatch manufacturers (e.g., Sony, LG, Apple, Samsung, Huawei) have A. Sensor Data Collection released their smartwatches recently. Depending on the brand In ActDetector, accelerometer data and gyroscope data are and the device version, these smartwatches are equipped with collected and analyzed to detect daily activities. Once Act- different kinds of sensors. For example, the LG G Watch R is Detector is started, the accelerometer data is sampled with a equipped with barometer, which can be used to measure the sampling rate of 60 Hz. Since gyroscope consumes much more floors a user climbs, and the Samsung Gear S2 3G is equipped power than accelerometer, the angular velocity is sampled with with GPS, which can be used to track a user’s movement. a sampling rate of 100 Hz only when an activity with non- However, these two sensors are not supported by many other repeated pattern is detected (details in Section III-C). The smartwatches. To make ActDetector device independent, it accelerometer on the smartwatch measures all the accelera- should only use sensors which are commonly equipped on tions that affect the device, which also include the gravity. different types of smartwatches. However, in ActDetector, to detect a user’s daily activities, Second, the system must be lightweight. ActDetector is we are only interested in the acceleration which is associated implemented on smartwatch to continuously detect and record with the movement of the smartwatch. Therefore, a low-pass the common activities occurred in user’s daily life. However, filter is applied to subtract the gravity components along X, compared to other mobile devices like smartphone, smartwatch Y and Z axis from the raw accelerometer data. For simplicity, usually has much less computing resources and power. To in the remainder of this paper, we use acceleration to denote monitor user’s daily activities for hours or even days, a large the filtered acceleration without the effect of gravity. amount of sensor data needs to be collected and processed on smartwatch. Therefore, ActDetector must be lightweight. B. Preferred Wrist Detection Third, to make ActDetector work unobtrusively and robust- When a user performs a certain daily activity (e.g., walking, ly, users’ wearing habits should be considered when designing running, etc.), the movement of his/her wrist follows some the system. Although most users wear smartwatches on their patterns. By collecting accelerometer data from the wrist- wrists, different users have different preferences.