Let the Objects Tell What You are Doing Gabriele Civitarese Claudio Bettini Abstract University of Milan University of Milan Recognition of activities of daily living (ADLs) performed in Via Comelico 39, Milano Via Comelico 39, Milano smart homes proved to be very effective when the interac- [email protected] [email protected] tion of the inhabitant with household items is considered. Analyzing how objects are manipulated can be particu- larly useful, in combination with other sensor data, to de- tect anomalies in performing ADLs, and hence to support early diagnosis of cognitive impairments for elderly people. Recent improvements in sensing technologies can over- Stefano Belfiore come several limitations of the existing techniques to detect University of Milan object manipulations, often based on RFID, wearable sen- Via Comelico 39, Milano sors and/or computer vision methods. In this work we pro- stefano.belfi[email protected] pose an unobtrusive solution which shifts all the monitoring burden at the objects side. In particular, we investigate the effectiveness of using tiny BLE beacons equipped with ac- celerometer and temperature sensors attached to everyday objects. We adopt statistical methods to analyze in real- time the accelerometer data coming from the objects, with the purpose of detecting specific manipulations performed by seniors in their homes. We describe our technique and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed we present the preliminary results obtained by evaluating for profit or commercial advantage and that copies bear this notice and the full citation the method on a real dataset. The results indicate the po- on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, tential utility of the method in enriching ADLs and abnormal to post on servers or to redistribute to lists, requires prior specific permission and/or a behaviors recognition systems, by providing detailed infor- fee. Request permissions from [email protected]. Ubicomp/ISWC’16 Adjunct , September 12-16, 2016, Heidelberg, Germany mation about object manipulations. © 2016 ACM. ISBN 978-1-4503-4462-3/16/09...$15.00 DOI: http://dx.doi.org/10.1145/2968219.2968285 ACM Classification Keywords ing omissions, substitutions and improper manipulations. I.5.1 [Pattern Recognition: Models: Statistical] For example, these include reaching and opening a wrong medicine box, using the wrong tool to perform an action Author Keywords or unnecessarily repeating a given manipulation. The sys- Activity recognition; smart homes; sensing tem described in this paper is not intended by itself to sup- port early diagnosis based on improper object manipula- Introduction tions. However, reliable object manipulation monitoring is The recent improvements in sensor technologies are hav- an essential subsystem of a more complex monitoring en- ing a deep impact on a long lasting challenge in ubiquitous vironment. In particular, what we describe is intended to computing and ambient intelligence, namely the recognition substitute the RFID-based subsystem used in [12] to mon- of human activities [4] and in particular activities of daily itor the use of items in preparing and consuming meals as living (ADL). well as taking medicines. As shown in Figure1, in order to recognise anomalies in performing these high level activi- The advantages of having sensors on everyday artifacts ties other sensor subsystems are used, including sensors for ADL recognition have been identified long ago, explor- revealing presence, pressure, temperature, power con- ing solutions mainly based on accelerometers and RFID [2, sumption and more. 11], however the technology has not been sufficiently reli- able and cost-effective for a wide scale deployment. A com- In our experience on deployments in the real homes of the mon argument against using sensor-augmented objects as elderly for continuous monitoring, solutions based on wear- opposed to wearables for ADL recognition, in addition to ables are critical: there is no guarantee that wristbands or technological issues, has been the difficulty in identifying pendants are constantly worn, not to mention smartphone the subject that is performing the activity in case of multiple or RFID readers that have been proposed for the advan- inhabitants of the same space [1]. On this respect, there tage of identifying the specific manipulated object. There has been some progress on this issue both on the tech- are also indications of a general adversity or disaffection of nological side (miniaturization of identifying beacons) and users to wearables targeted to healthcare related applica- on wearable-free solutions based on data analysis [7]. An tions [5]. Similarly, cameras and microphones are some- other approach to recognize specific object manipulations times tolerated in retirement residences, but much less in without neither sensors on objects nor wearables takes ad- private homes. vantage of audio and/or video recording [13], but this solu- tion is often perceived as too obtrusive. Our major contribution are experimental results on the ef- fectiveness of unobtrusive object manipulation recognition, Our investigation is driven by a specific application domain: using current commercial low cost and low energy con- the recognition of fine grained anomalies in performing in- sumption multi-sensor devices that can be attached to ev- strumented activities of daily living by elders at risk of cog- eryday objects. A closely related work is [10], which uses nitive impairment [12]. Clinicians need to identify manip- acceleration data acquired from sensors on items to eval- ulations of specific objects in a home environment includ- uate surgeons’ skill in manipulating precision tools. With respect to that work we monitor manipulations relevant to interesting for monitoring ADL execution, hence we divide our application domain, which are more coarse grained and the manipulation types in two categories: relevant ad irrele- of a different nature. We collected a dataset of more than vant. We consider a manipulation relevant if the task that is two thousands labeled manipulations, and we report en- achieved by performing the manipulation is crucial to mon- couraging preliminary results on their recognition through itor a particular ADL; irrelevant otherwise. Of course, clas- machine learning techniques applied on accelerometer data sification of manipulation types in relevant and irrelevant collected from the objects. We believe that our study con- has to be decided accurately by domain experts. Manipula- tributes to the design of a sensing subsystem that could tions considered as relevant are further classified in specific be effectively integrated in the smarthome environments sub-classes. used in several previous works on monitoring complex ac- tivities at home [4,9,6], independently from the algorith- Example 2 Suppose that we’re interested in monitoring the mic method being used, since object manipulations may be activity of taking medicines. In this scenario, we consider considered as simple events. a manipulation relevant if a medicine package is extracted from its repository, or if it is opened; while it is considered Modeling manipulations irrelevant if a medicine package is just displaced inside the We define as object manipulation the interaction of an indi- repository while searching. vidual with an object of interest with the objective of achiev- ing some task within the execution of a particular ADL. The technique More formally, we define a manipulation instance as m = In this section, we illustrate our technique to analyze the data coming from accelerometer positioned on objects, in ho;M;ts;tei, where o is the object manipulated, M is the order to recognize specific manipulations. manipulation type, ts and te are respectively the start and end time of the manipulation. Given i an instance of an A Recognition Framework ADL A, we say that a manipulation m 2 i if m is performed A The system is considered as part of a smart home envi- during i . A ronment instrumented with several environmental sensors. Example 1 Considering as object of interest a glass, some The general architecture is shown in Figure1. Each object possible types of manipulation of that object could be: using of interest has attached a wireless device which incorpo- the glass to drink while eating a meal, moving the glass on rates a 3-axis accelerometer sensor. Each device commu- the table while preparing the table, emptying the glass in nicates periodically the raw sensor data to the Smart-object the sink, inserting the glass in the dishwasher, and so on. data processing module, along with the device’s unique A manipulation instance could be: m = hglass, drinking, identifier. This module is in charge of: a) segmenting the accelerometer data in order to identify the manipulation 12:45:32, 12:45:45i where m 2 ieating (a manipulation which consists in using the glass to drink during the consumption occurrences, b) extracting several features and c) apply- of a meal). ing machine learning techniques in order to recognize the specific manipulation performed. Since each type of ob- We point out that not every type of object manipulation is ject has an associated set of specific manipulations (e.g., a by: the object o manipulated, the start time ts (i.e. the time instant where the object started moving), the end time te (i.e. the time instant where the object stopped moving) and the accelerometer data on the three axis. The output of segmentation module is a set of n manipulation occur- rences O = focc1;occ2;:::;occng. From each manipulation occurrence, we build a feature vec- tor which comprises more than 40 different features regard- ing statistics on accelerometer data and the duration of the manipulation. Manipulation recognition The next step is to infer, for each feature vector, the specific Figure 1: General architecture manipulation performed with the related object.