Mobeacon: an Ibeacon-Assisted Smartphone- Based Real Time Activity Recognition Framework

Mobeacon: An iBeacon-Assisted Smartphone- Based Real Time Activity Recognition Framework

Mohammad Arif Ul Alam, Nilavra Pathak, Nirmalya Roy Department of Information Systems University of Maryland Baltimore County [email protected], [email protected], [email protected]

ABSTRACT General Terms Human activity recognition using multi-modal sensing tech- Algorithm, Experimentation, Human Factors, Performance nologies to automatically collect and classify daily activities has become an active field of research. Given the prolifer- ation of smart and wearable devices and their greater ac- Keywords ceptance in human lives, the need for developing real time activity recognition, adaptive lightweight classification, se- lightweight activity recognition algorithms become a viable mantic knowledge, multi-modal sensing system and urgent avenue. Although variants of online and offline lightweight activity recognition algorithms have been developed, realizing them on real time to recognize peo- 1. INTRODUCTION ple’s activities is still a challenging research problem due to Automatic Activity Recognition (AR) has becoming a fun- the computational complexity of building, training, learning damental research concern of the ubiquitous computing com- and storing activity models in resource constrained smart munity [2] and plays an important role in the context-aware and wearable devices. To navigate the above challenges, applications and interaction design for cognitive computa- we build Mobeacon: a mobile phone and iBeacon sensor- tion. With the rapid advancement of wireless networking based smart home activity recognition system. We investi- and sensing technologies in recent years, recognizing Activi- gated the viability of extending Bagging Ensemble Learning ties of Daily Living (ADLs) based on wearable sensors com- (BEL) and Packaged Naive Bayes (PNB) classification al- bined with ambient sensors have drawn much research inter- gorithms for high-level activity recognition on smartphone. est. In this paradigm, wearable smart devices with sensing We incorporated the semantic knowledge of the testing en- and wireless communication capabilities are synchronized vironment and used that with the built-in adaptive learning with ambient sensors to capture different motion patterns models on smartphone to ease the ground truth data annota- of a user. Continuous sensor readings are collected and pro- tion. We demonstrated that Mobeacon outperforms existing cessed for extracting useful features, training an appropriate lightweight activity recognition techniques in terms of accu- activity model, and recognizing a variety of activities. racy (max. 94%) in a low resource setting and proves itself When aiming at building ubiquitous applications that are substantially efficient to reside on smartphones for recogniz- ready to work in uncontrolled environments, it is often nec- ing ADLs in real time. essary to retrieve activity data from one or more wearable sensors i.e., multi-modal sensing. For daily living, even the Categories and Subject Descriptors most compact solution tends to be computationally expen- sive, intrusive and difficult to maintain. The ubiquitous us- C.3.2 [Special-Purpose and Application-based Sys- age of mobile devices makes these sensing platforms ideal tems]: Real-time and embedded systems —Ubiquitous com- in terms of user adoption to gather low-level activity infor- puting; I.5.5 [PATTERN RECOGNITION]: Implemen- mation and further to get adherence to activity logging ap- tation—Interactive systems plications. For example, collecting locomotive data through ambient motion sensor requires an intermediate server phase to process the location signature. The extracted locomotive information needs to be sent to mobile devices in real time for online classification of activities. The entire process is computationally inefficient and prone to transmit unreliable and noisy data stream. Clearly, the use of the smartphones as activity data loggers, transmitters, receivers and proces- sors has operational and functional limitations (such as low battery power). Although high-level activity recognition and classification through hierarchical learning algorithms (e,g., Dynamic Bayesian Network [23], Hidden Markov Model [4] etc.) offer better accuracy based on multi-modal environmental sensing, offline training and apriori ground truth data annotation, they can be hardly implemented on smart devices due to their heavyweight resource hungry computa- lected from four individuals for a period of 2 weeks. The tional need. On the other hand, activity classification based experimental study shows that our system provides an ac- on ambient sensor and wearable smart devices (such as wrist- curacy of up to 94% for high-level activity recognition. Our band, smart-phone etc.) is practically challenging due to the high-level activity prediction methodology reduces classifi- unavailability of adequate resources (CPU, memory, battery cation cost significantly compared to traditional high-level power etc.) to support real time recognition. Enumerating activity recognition frameworks. the tradeoffs between activity recognition accuracy and cost associated with the classifiers, and executing the pipeline of the activity recognition tasks either offline (server) or online 2. RELATED WORKS (mobile) necessitate further investigation to benchmark the The idea of combining ambient sensors along with smart performance of state-of-the-art hardware and viability of im- device sensors to recognize daily activities has been inves- plementing lightweight in-situ machine learning algorithms tigated in the past [6, 7, 10]. We briefly summarize several there. works which have investigated activity recognition models Existing activity recognition frameworks suffer from many and algorithms. other practical problems. Many of them [9] are based on su- Activity classification with multiple on-body sensor nodes pervised learning where the training data requires a ground (i.e., wearable sensor devices or smart phones) along with truth with accurate labeling of all activities. To ensure high ambient sensor technologies have been proposed recently. accuracy, the classifiers need to be trained with long traces SAMMPLE [3] classified high-level activities exploiting dis- of data that may range from months to years. However, criminatory power of activity structures along the dimension gathering and accurately labeling ground truth data for such of statistical features by using a sequence of individual lo- a long period is difficult on smart devices and hinders the comotive low-level activities. Roy et. al. [4] investigated real time decision making on activity recognition. There are a Coupled Hidden Markov Model (CHMM) model to infer some existing unsupervised activity recognition algorithms spatial contexts and then considered spatial contexts as an that do not require ground truth ( [11]) but they either re- additional parameter to classify high-level contexts activi- quire mining activity models from web definitions or they ties. Other works [17] proposed multiple on-body sensor depend on detailed domain knowledge about activities and motes to detect user activities, body posture, or medical the environment. conditions with an offline analysis which hinders user mo- bility and real time analysis due to the periodic communi- 1.1 Research Questions cation with a fixed base station. A model has been pro- Our research attempts to answer the following research posed in [12] to train with a short amount of initial data, questions: but model updating was performed using a back-end server. • Can semantic knowledge about the environment and Pering et. al. in [13] proposed the use of a sensor mote with indoor localization help reduce the costs of ground truth la- an SD card attachment to interface with a mobile phone beling and bootstrap the classifier learning? If so, what kind in order to meet the computational requirements of body- of non-intrusive sensor devices and software applications are area sensor networks application. Lu et. al. [14] proposed required to design lightweight activity recognition method- a component-based approach to mobile phone-based classi- ologies with reduced reliance on intermediate server and ease fication for different sensors and different applications, but the process of semantic knowledge base creation? each sensor component required a separate classifier imple- • What kind of algorithmic models and their embodiments mentation. Activity recognition and energy expenditure was with the underlying learning parameters are required to sup- calculated by Albinali et. al. [8] using an accelerometer spe- port smart-phone based real time activity recognition that cific sensing model for on-body wireless nodes. Peebles et. can take benefit from prior defined semantic information? al. [15] used AdaBoost for activity recognition with mobile • How we can scale the activity recognition approaches phones, but focused mainly on ground truth labeling incon- across multi-modal sensing and indoor localization concern- sistencies without delving a practical system for long term ing accuracy and battery power consumption? use. Wang et. al. [16] focused on duty cycling mobile phone sensors to save energy based on a rigid rule-based recogni- 1.2 Key Contributions: tion model that must be defined before runtime but failed The main contributions of our work are summarized as to provide any user-friendly interface to define the rules. below. Our proposed framework is closest to PBN [17]. The cen- • We exploit indoor location contexts based on the iBea- tral difference between PBN and Mobeacon is that, prior one con provided signal and correlated that with the user pro- does not exploit semantic location information which helps vided semantic definition of smart home and accelerometer- train the classifiers in absence of ground truth data. Though, detected low-level activities as inferred by the smart-phone. PBN partially relies on ground truth labeling and offers sig- • We investigated Bagging Ensemble Learning (BEL) and nificant reduction of sensing cost, it incorporates multiple Packaged Naive Bayes (PNB) classification algorithm for wearable sensor devices and intermediate server dependen- employing them on smart-phone for real time activity recog- cies which hinders it’s use in practice. Combining the indoor nition. location knowledge with a semantic definition, we propose • We propose an intelligent inference caching technique to a lightweight high-level activity recognition framework with opportunistically sense the semantic correlations of context reduced reliance on ground truth labeling and any intermedi- tuples and add a lightweight adaptive semantic definition of ate server. We propose lightweight classification algorithms smart home. based on ensemble learning and packaged naive Bayes and • We designed a Mobeacon smart home and evaluate our investigate the model building, training, storing and learn- proposed algorithmic models using real time data traces col- ing on the fly to recognize activity using smart-phone. Table 1: Mobeacon Classification Groups possible for mobile deployed real time activity recognition Context Semantic & Location/Postural/High-level Contexts system where time is critical. A minimal need for ground Inference Category truth reduces the burden on the user to label training data Indoor Environmental Exercise Bike, Couch, Dining Table, and accelerate just-in-time processing and decision making Contexts Bed, Closet, Reading Table, Bookshelf, for activity recognition. Bathroom, Kitchen, Porch Minimal Need of Intermediate Server: Relying on Low-level contexts Standing, Walking, Cycling, Lying, intermediate server and constant connectivity violate un- Sitting derpinning requirements of real time activity recognition. High-level contexts Exercising, Prepare Food, Dining, Involvement of intermediate server implies problems such Watching TV, Prepare Clothes, as, data inconsistencies due to lossy channel, network over- Studying, Sleeping, Bathrooming, flow and communication failure, security and privacy breach Cooking, Past Times, Random of data etc. Thus, conducting efficient intermediate server based classification, it is necessary to employ several energy 3. APPLICATION REQUIREMENTS intensive fault tolerant encryption techniques. To design real time lightweight energy efficient activity recognition frame- Our Mobeacon system design is motivated by the require- work, the system should refrain from relying on intermediate ments of a real time application for activity recognition. server. Data from multi-modal sensor devices are captured and an- Adapting Fault Tolerant Classifiers: The system must alyzed online on mobile devices. Thus the system must be be fault tolerant and adaptive in nature thus can accu- able to accurately and efficiently classify typical daily activ- rately maintain high accuracy in case of noises in sensing, ities, postural and environmental contexts (Table 1). De- device position, data transmission etc., environmental am- spite these categories being common for many individuals bient dynamics (changes in the semantic location of ob- and previous work [17] has identified some of them, reliance jects/furniture etc.) or spatiotemporal changes of user con- on ground truth and intermediate servers are disregarded. text behaviors. From the Table 1, we break down our target classifications Implementing Lightweight Classifiers: smart-phone and into three groups: indoor environmental contexts, low-level ambient sensing-based technologies are subject to severe pro- contexts and high-level contexts. With the environmental cessing, storage and energy constraints. Redundant classi- and low-level contexts, we can infer high-level activity pro- fications result huge computational overhead and unneces- viding insight into the physical states of users for personal sary drainage of battery power. smart-phone-based activity health and physical wellness monitoring applications. Our recognition system must be capable of selecting relevant sen- goal is to detect high-level activities in which a user engages sor for classifying activities while powering down the other in different indoor area with particular low-level activity sensors. Furthermore, the system must avoid extensive pa- sets, for example, ‘watching TV’ in ‘couch’ area where ‘sit- rameter tuning as well as relying on complex sensing models ting’ represents a low-level context state or ‘preparing food’ for activity recognition in real time. in the ‘dining table’ area where low-level context is composed of set {‘standing’,‘walking’}. We define this indoor areas of interest in a smart home environment a-priori with the un- 4. OVERVIEW OF OUR SYSTEM derlying low-level contexts to infer high-level activities. This Our Mobeacon system assumes a multi-inhabitant smart piece of information is referred as Semantic Knowledge Base home instrumented with ceiling mounted iBeacons in each (SKB) about the environment and user contexts in our work room transmitting RSSI signals which are being captured in and eventually aided in on-the-fly to accelerate the real time the occupant’s smart-phone. Our system also assumes the high-level activity recognition on resource constrained de- presence of a smart-phone in the right-pocket of each in- vices. The requirements to provide such a practical activity habitant to collect the fine-grained postural-level activities. recognition system are as follows. The basic steps are summarized as follows: i) ‘Sensing pla- Application Interface: Real time activity recognition re- nar’ gathers sensor data from smart-phone and RSSI signals lies on different sensor modalities and knowledge base in- from iBeacons; ii) ‘Feature processing’ module is involved put. Appropriate and user-friendly application design re- with feature extraction of sensor attributes, signal process- quirement differs based on application goal and system ar- ing of RSSI signals and feature processing for small training chitecture. Both supervised and semi-supervised activity sets collected for low-level activity recognition; iii) ‘Seman- classifiers need an user-friendly software application design tic knowledge processing’ engine is involved with building which must provide an intuitive interface for adding, remov- semantic knowledge base (SKB), expression tree (ET) and ing, and configuring different sensors, ambient objects and training classifiers based on the small training sets; iv) ‘Clas- ground truth related knowledge base geared to detect the sification’ module is involved with sub-region classification user’s intended activities. Defining the correlations to build and utilizing the context information and low-level activity such a semantic knowledge base should also be a simple, training sets, for classifying high-level activities employing adaptive and non-invasive effort, facilitated by an easy to two lightweight classifiers, packaged naive Bayes (PNB) and use mobile phone interface. bagging ensemble learning (BEL). v) ‘Intelligent inference Reduced Reliance on Ground Truth: Ground truth cache’ opportunistically updates semantic knowledge based labeling is one of the major issue for supervised and semi- on computationally lightweight rule mining technique. We supervised learning algorithms. Efficient ground truth data evaluate the practicability of our proposed system consist- annotation is challenging and generates some arduous prob- ing of the above processing engines by quantifying the re- lems such as time synchronization, labeling ambiguity, in- duction in computational complexity and improvement in terleaved activity dilemma, constant observer etc. There- high-level activity recognition accuracy we perceive as a fore, reliance on ground truth should be reduced as much as result of combining iBeacon based indoor localization and Table 2: Semantic Knowledge Base Correlations Wearable High-level Sub- Low Wearabl Build SKB Occupied Sub-Region Feature e sensor Selection Processing activities regions activities Build Expr. Exercising SR1 Cycling, Sitting Packaged Naive Bayes Sensing Tree Algorithm Prepare Food SR4 Standing, Walking Ambient Ambient Classification Dining SR4 Sitting sensor Feature Intelligent Processing Inference Watching TV SR2, SR3 Sitting Bagging Ensemble Cache Feature processing Learning Algorithm Prepare Clothes SR8 Standing, Walking Application Labeled low- Studying SR7 Sitting Mobeacon level activity Semantic Know- ledge processing Sleeping SR5 Lying Bathrooming SR9 Standing, Walking, Sitting, Lying Figure 1: Overview of our System Cooking SR10 Standing, Walking Past Times SR11 Standing, Walking, Sitting smartphones provided low-level activities with lightweight Random SR1-SR11 Standing, Walking, Lying, Sitting classifiers based naive Bayes and ensemble learning. The schematic representation of all the components, and under- 5.3 Ambient Sensors Positioning lying logical steps are shown in Fig 1. For this task, user needs to localize the ambient sensor. This localization process addresses two important scenar- 5. DESCRIPTION OF OUR FRAMEWORK ios. 1) Single ambient sensor rooms; 2) Three ambient sensor rooms. For single ambient sensor rooms (for example, Our activity recognition framework consists of the follow- kitchen, bathroom etc.), user needs to select ‘single ambient ing components. 1) Semantic knowledge base; 2) Applica- sensor room’ from the drop-down list of our develop app, tion; 3) Ambient Sensors Positioning; 4) Semantic Correla- bring the smart-phone as close as possible to the ceiling tions Definition; 5) Building Expression Tree; 6) Occupied mounted iBeacon and then press the button ‘sensor localize’ Sub-Region Detection; 7) Bagging Ensemble Learning clas- to scan. This action records the particular sensor UUID, sifier; 8) Packaged Naive Bayes classifier, and 9) Intelligent major-minor values and the distance measure (in meters) Inference Cache. Next we discuss each of these components and stores the information as origin (x=0, y=0) coordinate in detail. of the room. For three sensors room, user needs to do the above procedure once for both the sensors 1 and 2 (illus- 5.1 Semantic knowledge base trated in Fig 2(c)). Our application automatically calculates Our proposed framework depends on a simplified domain X and Y values based on the average of l, m and n using knowledge consists of the spatio-contextual correlations. Thus following equations: r we define two modules: a semantic knowledge base and a se- l 2 X = l; Y = n2 + (1) mantic correlation rule sets. Prior to setting up Mobeacon 2 in smart home environment, user needs to define a set of correlations to build semantic knowledge base. User also 5.4 Semantic Correlations Definition needs to define the sub-regions prior to Mobeacon smart In order to define the semantic correlations though our home installation as follows: 1) First, presume the high- user-friendly application interface, user needs to bring the level activity sets to be recognized. In Mobeacon smart smart-phone at the center of each of the pre-defined sub- home, we selected 11 high-level activities as shown in Ta- regions, select/add new sub-region, select correlated low- ble 1 which are relevant to functional health of older adults, and high-level activities, type approximate length of the re- 2) Identify and define the critical sub-regions which are in- gion considering it as a square shaped one and click ‘Set’ volved with pre-selected high-level activities, for example, button to confirm definition (Fig 2(a)). The above proce- sub-region ‘SR1’ is involved with high-level activity ‘Exercis- dure calculates the four corners’ coordinates using the iBea- ing’ as shown later in Fig 9, We selected 11 sub-regions that cons provided distance values (Fig 2(a)), and saves all values are involved with single or multiple high-level activities; 3) in the smart-phone memory. Locate the ambient sensors and demark the area of each sub- region; 4) Set an approximate minimum time threshold for 5.5 Building Expression Tree any high-level activity recognition (we consider 2 minutes). An expression tree for a tuple a = v is a tree represen- Our adaptive heuristic changes the minimum threshold over tation of the Boolean expression that implies the value v time based on the activity monitoring; 5) Finally, define the for the attribute a. It is generated by combining various correlations among high-level activities, low-level activities rules that imply a = v directly, or indirectly via transitive and sub-regions. Table 2 shows our Mobeacon smart home relationships with other rules. The expression tree for a semantic knowledge base correlations. given high-level activity ‘cooking’ is constructed as follows. We start with ‘cooking’ high-level activity and continue ex- 5.2 Application panding the rules stated in the semantic knowledge base We design an Android (version 5.0 Lollipop) based user- (‘cooking’⇒‘walking’ AND ‘kitchen’, ‘cooking’⇒‘standing’ friendly mobile application that simplifies the procedures of AND ‘kitchen’) until we reach the leaf nodes which can’t be defining, controlling, and monitoring the semantic knowl- expanded any further(illustrated in Fig 3(a)). edge and context recognition as shown in Fig 2(a). We combine Estimote iBeacon SDKs with Android smart-phone 5.6 Occupied Sub-Region Detection sensor manager and provide intuitive application interfaces This application takes the semantic knowledge base (SKB) to confirm different sensors inclusiveness and their usability. information where sub-region set is defined as SR = {sr1, sr2 Procedure BESTree_Training (input: Training set S; Ensemble X 1 2 classifier E; Integer T (number of bootstrap samples l output: Trained Ensemble Classifier E) 1. for i = 1 to T 2. Sb = bootstrap sample from S // sample with replacement Y n m 3. Soob =out of bag sample 4. train base classifiers in E on Sb 5. Ei =BESTree_Selection(M,S) 6. Return E

3 Figure 5: BESTree Training Algorithm (a) (b) (c)

Figure 2: (a) Semantic knowledge base creation app (b) Classiﬁcation app (c) iBeacon position ‘Random’ type of high-level activity. The detailed algorithm coordination is stated in Fig 4

standing kitchen walking couch standing kitchen walking kitchen sitting kitchen 5.7 Lightweight Bagging Ensemble Learning We propose an extension of bagging ensemble learning AND AND AND AND AND algorithm adding an expression tree based inference engine OR OR to help guide the activity recognition process in real time. cooking Our proposed algorithm is described in two folds, training cooking and classification. (a) (b) We use BESTree selection algorithm [20] to train our lightweight activity recognition model and Decision Tree (DT) Figure 3: ‘Cooking’ high-level activity tuple ex- as a base classifier. We consider smart-phone provided ac- pression tree (a) at the beginning of inference (b) celerometer values (x, y and z) and Subregion Detection al- after 1 hour of running rule mining gorithm provided spatial context (SR), in other words the locomotive micro activity contexts and location context.. We have also incorporated the minimum duration values of . . . , sr11}. Consider sr = {hx1, y1i, hx2, y2i, hx3, y3i, hx4 activities as obtained from the semantic knowledge base. , y4i} where x and y values define the four corner coordi- Let x be an instance and mi; i = 1...k , be a set of base nates of sub-region, sr. Our application continuously senses classifiers associated with probability distributions m (x, c ) the iBeacon features (id and distance measure) and calcu- i j for each class label cj , j = 1..n. The output of the final lates the coordinates of the smart-phone device inside the classifier ensemble y(x) for instance x can be expressed as: smart home using trilateration [19]. It checks whether the Trilateration provided values (x and y coordinates) are in- k X side any of the pre-defined critical sub-region space not using y(x) = arg max ωimi(x, cj ), (3) cj simple scalar dot products of vector method. i=1

M of coordinates (x, y) is inside the where ωi is the weight of base classifier mi. We employ, rectangle ABCD iff (0 < AM · AB < AB · AB) ensemble learning strategies as underlying methods for cal- culating optimal weights for each base classifier given the ∧(0 < AM · AD < AD · AD) (2) hierarchical approach consisting of micro activity recogni- If it does not satisfy any of the pre-defined sub-region occu- tion and combining this with SKB and location context for pancy, it returns ‘Other’ sub-region which is defined only for higher-level activity recognition. Bagging: Bagging is based on the instability of base classifiers, which is in our approach exploited to improve the predictive performance of any unstable base classifiers. The Procedure Occupied_Sub-Region_Detection basic idea is that, given a training set T of size n and a clas- (input: Sub-region definition SR, iBeacon IB; sifier A, bagging generates m new training sets, Ti , each output: Occupied Sub-region ID S) 0 1. if IB.size = 1//one iBeacon is discovered of size n ≤ n. Bagging then applies classifier A with each 2. for each sr in SR training set Ti to build m models and use simple voting 3. if sr.size = 1 and sr.UUID = IB.UUID mechanism to finalize the output. 4. Return sr 5. else if IB.size = 3// 3 iBeacons are discovered Lightweight Design: We extend BESTree classification al- 6. {x, y} ⇒ Trilateration (SR,IB) gorithm incorporating sub-region detection, expression tree 7. For each sr in SR based high-level activity composition and activity duration. 8. if {x, y} inside sr// Equation 2 9. Return sr We bring in lightweight expression tree based inference en- 10. else gine after recognizing the micro-activities based on each bag 11. Return ‘Other0 of BESTree base classifier. Expression tree based inference engine extracts the expression trees saved in the inference Figure 4: Subregion Detection Algorithm cache, augments the sub-region information obtained from the Sub-Region Detection algorithm, applies inference on m Procedure BESTree_Classification Procedure Training_PNB (input: Test set T ; Trained Ensemble Selection classifier E; (input: nCL , D;//D probability distribution iBeacon IB; smart-phone accelerometer values (x,y,z); Semantic output: PNB Model M) Knowledge Base SKB; output: High-level activity H) 1. for each cLi ∈ CL// low-level activity state 1. F = Feature_Extraction(x,y,z) 2. Compute P (cLi) from D// observation probability 2. C = BESTree_Ensemble_Classifier(F ,E) 3. for each pair of attributes Ai and Aj 3. LA(t) = Max_Vote(Hset)// low-level 4. for each ai,aj , and cLi to Ai,Aj , and CL // activity LA 5. Compute P (ai|aj , cLi) from D 4. HAt = Expression_Tree_Classification(L,ET ) // conditional probability between two attributes // high-level activity HA 6. for each pair of attributes Ai and Aj 5. if HA(t) 6= HA(t − 1)&HA(t) = HA(t − 2) 7. Compute Ip(Ai; Aj |CL) and Wij from D 6. &HA(t − 1) < MinDuration(HA(t − 1)) // Conditional Mutual Information between two attributes 7. HA(t − 1) ⇐ HA(t − 2) 8. for each Ai 8. Duration(HA(t))=Duration(HA(t − 1))+ for each Ai 6= Aj in D 9. Duration(HA(t − 2)) If Ip(Ai; Aj |C) ≥threshold HNBi → Aj 10. Return HA // conditionally dependent attributes 9. else NB→ Ai// conditionally // independence attributes Figure 6: Bagging Ensemble Learning Classifica- 10. M ⇒ HNB(Ip,A,P ,C,W ) 11. Return M tion model predicted low-level activities along with sub-region Figure 7: Packaged Naive Bayes Training Algo- information and finally votes on each expression tree out- rithm put to predict high-level activities. In addition, we design a duration model based heuristic algorithm to confirm more fine-grained prediction of high-level activity. Whenever, we its corresponding HNB bag with a higher accuracy. Though find an end of a high-level activity, we check the last ended the classification process in this case compared to ensemble high-level activity duration. If it does not meet the mini- learning takes a little longer, it it performs better because mum duration, it is merged with its previous one. The entire the dimension of datasets remain low. When the thresh- algorithm is stated in 6. old is small enough (such as a negative value), this model evolves into HNB. So if the threshold is set an optimal value, 5.8 Lightweight Packaged Naive Bayes Algo- the model would be a perfect combination of HNB and NB, rithm which would have a high accuracy and short classification We investigate the viability of Packaged Naive Bayes (PNB) time. The idea is if just the most correlative attributes are for implementing real time activity recognition on smart- concerned, it is not only reducing the classification time but phones. We consider PNB over hidden naive Bayes or naive also improving the accuracy. The classifier corresponding to Bayes [1] due its advantage in terms of reduced test time in PNB on an instance E is defined as follows: number of HNB bags the recognition phase and higher accuracy over multi-modal Y sensor datasets [5]. c(E) = arg max P (c) P (ai|aφ, c) c∈C Assume that A1,A2,...,An are n attributes correspond- i=1 ing to attribute nodes in a Bayesian Network. An example number of attributes in NB Y instance E is represented by a vector < a1, a2, . . . , an >, P (aj |c) (6) where ai is the value of Ai. Let C represent the class vari- j=1 able corresponding to the class node in a BN. We use c to The conditional probability of the attribute and its hidden represent the value that C takes and c(E) to denote the class parent as well as W are defined below. of E. Assume that all attributes are independent given the ij class, the resulting classifier Naive Bayes is represented by, IP (Ai; Aj |C) Wij = (7) P#attributes HNBi Y j=1,j6=i IP (Ai; Aj |C) c(E) = arg max P (c) (ai|c) (4) c∈C X P (ai, aj |c) IP (Ai; Aj |C) = P (ai, aj , c) log (8) If we denote hidden parent by aφ, HNB can be defined as P (ai|c)P (aj |c) ai,aj ,c follows. #attributes HNBi c(E) = arg max P (c)P (a1, a2, a3, ..an|aφ, c) (5) X c∈C P (ai|aφ, c) = Wij × P (ai|aj , c) (9) In Packaged Naive Bayes (PNB) each attribute either has j=1,j6=i a bag of HNB as its hidden parent node or a bag of NB. We extend the PNB by adding an expression tree based clas- During the classification phase, the attributes which have sifier at the end of PNB based low-level activity classification bag HNB use HNB algorithm, and other attributes in the and applying our heuristic model to confirm a fine-grained bag NB correspond to NB algorithm. At training phase, high-level activity classification. Fig 7 and Fig 8 show our each attribute selects the attributes whose dependence with modified PNB algorithm training and inference algorithms it are greater than the threshold for being considered in cor- respectively. responding bag HNBi (HNBi is hidden parent). However, if none of the attributes are selected, the attribute is being 5.9 Intelligent Inference Cache considered into the bag NB (similar to bagging). When the The relationships among various context attributes (i.e., dataset dimension is low, the threshold is set to a smaller low-level, location and high-level context attributes) can- value, and that there are more attributes being chosen with not be static within an application. It is also tedious and minSup and minConf over merged segments, generates rules Procedure Inference_PNB (if meets the threshold) and helps update the existing ex- (input: an instance E; a PNB model M output: classified instance HA) pression trees. Note that Inference Cache can sometimes 1. for each value cLi of CL// each low-level return results that are i. incorrect, if user’s behavior devi- 2. for each ai ∈ E// each attribute ates from context rules, or ii. stale, if results are inferred 3. If HNBi 6= ∅, Calculate P (ai|aφ, cL) from stale values due to idle nature of iBeacon and signal //aφ hidden parent attribute 4. else NaiveBayes(NB,E,CL) strength distraction by human obstruction. The severity 5. Compute K → cL(E) of incorrectness is reduced by the rule miner conservatively 6. HAt = Expression_Tree_Classification(K,ET ) choosing rules that potentially hold in the activity merging //high-level activity HA at time t 7. if HA(t) 6= HA(t − 1)&HA(t) = HA(t − 2) event. Similarly, to reduce the effect of staling, we choose a &HA(t − 1) < MinDuration(HA(t − 1)) small cache expiry time of 2 minutes. HA(t − 1) ⇐ HA(t − 2) Rebuild Expression Tree: Intelligent Inference Cache is 8. Duration(HA(t)) = Duration(HA(t − 1))+ 9. Duration(HA(t − 2)) activated when a high-level activity merging happens and 10.Return HA generates a set of rules using Apriori algorithm based rule mining. These set of rules are used to generate new expression tree for merged high-level activities as follows. Suppose Figure 8: Packaged Naive Bayes Inference Algo- a new rule for ‘cooking’ high-level context is generated by the rithm inference cache: ‘cooking’⇒{‘sitting’ AND ‘kitchen’}. The rebuilding of expression tree for ‘cooking’ context starts with expanding ‘cooking’ node with all associated base rules in- error-prone for a single person to define all possible context cluding the new rule (i.e., ‘cooking’⇒{‘standing’ AND ‘kitchen’ relationships within a smart-home environment. Behaviors }, ‘cooking’⇒{‘walking’ AND ‘kitchen’} and ‘cooking’⇒{ of the person are also volatile and they change over time ‘sitting’ AND ‘kitchen’}). Finally a new expression tree through a specific pattern, regularity or correlation remains is formed and cached in our intelligent inference cache for consistent. We introduce an intelligent inference cache mod- ‘cooking’ activity classification as shown earlier in Fig 3(b). ule to solve these problems employing an adaptive update of the expression tree using rule mining techniques. Our intelligent inference cache learns additional rules that are not 6. EXPERIMENTAL EVALUATION so-obvious to semantic knowledge definers and dynamically In this section, we introduce our, Mobeacon smart home adapts over time. Meanwhile, applying rule mining in ev- system setup, configuration and data collection. We per- ery time segment may drastically consume battery power of form preliminary experiments to show the performance of smart-phone device. Therefore, we propose to aid in rule our proposed lightweight algorithms for real time activity mining technique on temporal segments when there is con- recognition using bare minimal sensing, processing, comput- fusion within the classifier to recognize two activities. Thus ing, and storing resources. we propose to merge one activity with prior activity and consider that new activity as an occurrence of behavioral 6.1 Hardware Interfacing pattern change. We update this pattern change in the ex- We developed an integrated hardware software toolkit fo- pression tree configuration to make our inference process cusing on having an intuitive application interface and re- more robust. These techniques are described below. duced reliance on ground truth and intermediate server. We Rule Mining: Through the rule mining, we aim to generate first describe two Bluetooth Low Energy (BLE) technology rules that have the following general form: hC1,C2, ..., Cn based devices as part of our Mobeacon smart home system ⇒ Ri which implies that R holds whenever all the tuples setup. hC1,C2, ..., Cni are true. For example, the rule hwalking=True; Estimote iBeacon: Estimote Beacons [22] are small wire- kitchen=True ⇒cooking together=Truei implies that if a user less sensors that can be attached to any location or object. is in the location ‘kitchen’ with low-level context state ‘walk- They broadcast tiny radio signals (RSSI) which smartphones ing’ then he is in high-level context state ‘cooking’. We can receive and interpret, thus unlocking micro-location and investigated Apriori algorithm to operate on naive activity contextual awareness. With the Estimote Android SDK, state space model to pinpoint the underpinning spatial con- applications on Android based smart-phone are able to ap- texts [18]. A threshold thresh is defined, that is used by proximate their proximity to nearby locations and objects, Apriori algorithm to identify the sets of low-level context recognize their types, ownerships and measure temperature states which are subsets of at least T high-level activity. and motion. We integrate Estimote Android SDK with our We formulated the Association rules with a support and a Mobeacon Android application that gives distance measure confidence. For example, if a state space model has 1000 (in meter) between smart-phone and the Estimote Beacon context states, out of which 200 include both context states along with the UUID, major and minor values. A single area A and B and 80 of these include context state C, the associ- covered by multiple Estimote Beacons generate same UUID ation rule A, B → C has a support of 8% (= 80/1000) and but different major and minor values. Thus we can sepa- a confidence of 40% (= 80/200). The algorithm takes two rate individual sub-room level occupancy using the unique input parameters: minSup and minConf to generate all the ID (UUID). rules with support ≥ minSup and confidence ≥ minConf. We assume minSup = 90% and minConf = 10% which help 6.2 Testbed setup and Data Collection strike good balance between tolerating occasional inconsis- We develop a real testbed for Mobeacon smart home sys- tencies and highlighting the viable rules for our hierarchical tem in a family apartment with a bedroom, living room, context state space model. Our heuristic model based high- kitchen, long corridor, a porch and a bathroom (Fig 9(a)). level activity merging event activates rule mining, calculates We installed 8 iBeacons and one SensorTag Beacon on the 2 4 푅

1 SR4 SR3 SR1 5 SR2 3 SR11 SR5 SR9

SR10 SR6

SR7

6 SR8 7 8

SR7 SR5

SR8

SR4

SR9

SR6 SR3

SR1

SR2

SR12 SR11 iBeacon 9 (a) (b) SensorTag (a) (b) Figure 10: (a) iBeacon distance accuracy (b) Tri- lateration based localization accuracy for each Figure 9: Mobeacon Smart Home Set Up sub-region (sorted by area) ceiling as displayed in Fig 9(b). The rooms which are in- tradeoffs in our experiment. In each approach, we choose volved with multiple high-level activities and complex sub- the following equation to measure accuracy: region correlations (such as, bedroom is involved with study- #TP + #TN ing, sleeping, pressing clothes etc.) are equipped with three accuracy = (10) iBeacons. The rooms with less activities are equipped with #TP + #FP + #FN + #TN single iBeacon (such as porch area). We also set up three IP where TP, TN, FP and FN represent True Positives, True cameras in three appropriate places thus we can exclusively Negatives, False Positives and False Negatives respectively. collect the ground truth of all the performed activities. To measure accuracy of activity classification, we used datasets We recruited one married couple to live in this apartment chunk by chunk to maintain its continuity. for two weeks. They were trained of using our Mobeacon An- iBeacon Sensing Accuracy: To test the accuracy of dis- droid application and showed them how to build their own tance measures as obtained from the iBeacon, we calculated semantic knowledge base of daily livings (took 20 minutes on errors between the actual and iBeacon SDK provided dis- average to complete semantic knowledge base inputs). They tance measures in a no obstacle environment. We tested were asked to perform our pre-defined 5 low-level activities the distances ranging from 0 to 10 meters for 40 times each keeping the smart-phone in their right pocket for two min- and plotted the error measures as shown in Fig 10(a). We utes each. During this phase we gathered a small amount observed that the error measures are not exactly linear but of sensor readings and created training sets for our study. proportional to the physical distances between the physical Finally, they were left alone to perform their natural daily phone and the iBeacon. The error within a range of 1 meter activities. Besides the two inhabitants we have recruited an- was almost ≈ 0 but it became ≈ 2 meter at the range of 10 other two participants in the test apartment to perform all of meters. the scripted ADLs several times without any prior instruc- Indoor Localization Accuracy: We implemented tri- tions. They have been asked to keep the Mobeacon Android lateration based sub-region detection algorithm using Java application installed smart-phone on their right pocket. Af- and evaluated its accuracy through our Mobeacon dataset. ter gathering entire testbed data, we recruited two graduate Fig 10(b) shows the regions with one iBeacon (‘SR9’, ‘SR10’ students to label sub-regions, low-level activities and high- and ‘SR11’) and depicts ≈ 100% accuracy for any number level activities based on the recorded videos and validate of inhabitants. Other sub-regions that were involved with each others’ annotations. This generated 12 hours of fine- multiple iBeacons, accuracies were proportional to their area grained labeled ADLs data with 35 chunks of continuous measure. Furthermore, we noted that the sub-region detec- data sets. These chunks are defined individually for (cou- tion accuracy has been decreased as the number of inhabi- ple), (couple + 1) or (couple + 2) inhabitants smart home tants are increased (87%, 71% and 65% average accuracies activity logs. The entire dataset finally reflects the, natural are noted in case of 2, 3 and 4 inhabitants respectively). ADLs of 4 individuals with semantic knowledge base, smart- Low-level Activity Recognition Accuracy: To clas- phone accelerometer values (x, y and z axis), iBeacon values sify the low-level activities, the 3-axis accelerometer data (UUID, major, minor, distance), sub-region IDs and time. streams collected from the smart-phone were broken up into successive frames. We extracted a number of statistical fea- 6.3 Cost Accuracy Tradeoffs tures (i.e., mean, variance, standard deviation, maximum In this section we present the results comparing the cost- and minimum, magnitudes, energy etc.) for accelerometer accuracy tradeoffs of our models with other traditional mod- to create a 20-dimensional feature vector set. We chose 20 els. We implemented our proposed BEL and PNB algo- samples per second and 1.5 seconds windowing for enumer- rithms and investigated their performance over existing al- ating the feature sets for low-level activity recognition. The gorithms. We chose threshold = 0.138 for PBN and a set low-level ground-truth annotated training set was then fed of parameters P = hnum bag = 4, num execution slots = into the classifiers. We identified 6 best optimized feature 4, num features = 2, num trees per bag = 7i for BEL which sets (avg. X, max. Y, avg. magnitude, std. magnitude, were found optimal through experiment for getting reduced XY correlations and energy) through Correlation Feature overhead and improved accuracy. Selection (CFS) algorithm [24]. To employ BEL and PNB for low-level activity recognition, we excluded the expression 6.3.1 Accuracy Tradeoffs tree based inference. We also fed the entire training dataset We have taken different approaches in different accuracy into Weka toolkit [21] and trained 4 classifiers: Naive Bayes, osmto.W ics h eal ottaeff below. tradeoffs cost power details 3) the and discuss usage, We memory 2) consumption. usage; CPU 1) perspectives: more ones. be other to the tend than model etc. BEL lightweight food’ in ‘prepare accurate ac- ‘cooking’, similar like show tivities HMM and 93 94 HMM BEL BEL lightweight accu- Although, (lightweight that recognition curacy, 2%. activity see by high-level we racy improve 3, helps Table PNB model From and BEL lightweight [21] with implementation. Weka accuracy in the APIs LogitBoost compared java and using (HMM), (RF) Model RandomForest Markov and (LB) Hidden algo- as classification such activity rithms popular other Accuracy: several plemented Recognition Activity High-level achieving NB, and and DT activ- algorithms, low-level popular classification most ity two outperform that PNB note We and the BEL activities. illustrates low-level each 11(a) for Fig measures PNB. accuracy and BEL (DT), Tree Decision Logit- PNB, and traditional BEL PNB lightweight lightweight accuracy HMM, BEL, RandomForest, recognition Boost, activity traditional High-level using 3: Table Ac- Usage (BLE), Memory Energy (c) WiFi Low (%) and Bluetooth utilization (CB) for CPU Bluetooth measure (b) Classical (miliWatt) and (ACC), Consumption accuracy Power celerometer recognition (d) activity and Low-level (MegaBytes) (a) 11: Figure .. otTradeoffs Cost 6.3.2 edfiecs rdosfrcasfiaini he different three in classification for tradeoffs cost define We Overall Random Times Past Cooking Bathrooming Sleeping Studying Clothes Pressing TV Watching Dining Food Prepare Exercising Ac- tivities High Accuracy % 100 ≈ 60 80 8 cuayrespectively. accuracy 88% . %,casfiainacrc ncs fcmlxac- complex of case in accuracy classification 7%),

Standing DT 92 75.2 75.4 99.6 97.5 100 95.8 93.6 100 94.1 76.9 100 BEL T-

Walking NB Cycling BET 92.2 84.3 97.9 94.2 93.7 96.9 99.3 96.3 72.3 100 90.6 100 PNB T-

(a) Lying PNB . % ihwih N 93 PNB lightweight 5%,

91.1 75.1 72.2 98.3 96.5 100 94.3 93.6 100 94.1 76.9 100 LB Sitting

91 75.2 75.5 99 97 100 95 93 97.5 93.2 75.5 100 RF CPU (%) 20 30 10 BLE 0 93.7 81.2 72.6 94.2 95.5 100 91.5 95.2 96.2 92.5 75.9 100 HMM Acc

(b) 4.1 94.5 87 85 99.6 99 100 99 96 100 99 87 100 BEL L- CB eim- We 3.5 ≈ . 87% 95.3 96.9 99.3 97 75 100 91 100 PNB L- 94.3 85.2 98.2 94.8 9%, 5.2 WiFi 15.6

rebcmssal thlsipoeteacrc iha with accuracy 12(d). Fig the in improve shown helps expression as cost When it lower stable accuracy. in- lower becomes trees with tree phase, expression cost initial the higher the updates curring At and frequently time. location) cache over and inference tree high- expression (low-, the contexts updates the senses odically Performance Cache Inference Intelligent 6.4 that, power. see less 24% we consuming where 12 clas- occupying lightweight sifiers other Fig than in better performs results BEL lightweight PowerTutor. these using min- illustrated application 10 We each total for and for memory separately consumption average algorithm power CPU, average each measured ran and We utes im- smart- to too. on helps sup- phone classifiers that RandomForest it platform and that, LogitBoost Android plement is in MobileWeka toolkit using WEKA of ports lightweight advantage and The BEL our lightweight PNB. of algorithms, performance proposed the two evaluate to (MobileWeka) platform 11(c)). Fig other Cost: and that Classification usage 11(b) memory (Fig and technologies CPU sensing approaches lesser iBeacon based much occupy 11)(d). sensing also (Fig accelerometer transmission smartphone’s data and for less CB times 1.3 than Blue- consumes power Classical iBeacon transmission hand, than data other power based the transmission the technology On of data protocol. times based communication 42 tooth WiFi as much that as note uses We Android PowerTutor powerapplication. using and for application memory each separately CPU, for average applications consumption respec- measured 4 and their the WiFi, minutes ran over 10 measure We rate to performance. second technologies tive per BLE bytes and We 100 tasks. Bluetooth at individual record- their data executing sensors, transmitted and relevant applica- values their Each sensing activating ing technologies. with sensing started Blue- WiFi application tion Classical and Each iBeacon, (CB), accelerometer, tooth smartphone. with 4 involved appli- Nexus was Android Google four in implemented cations other we iBeacon. with of technologies, consumption protocol power sensing transmission sensing signal iBeacon a compare To as used been Cost: has Sensing iBeacon Memory (MB) eicroae nitlietifrnecceta peri- that cache inference intelligent an incorporated We 10 0 5 BLE

Acc 6.8 (c)

≈ 5.1 CB %ls P n 0Bls eoyand memory less 30MB and CPU less 4% 8.9 WiFi eipeetdWK nAndroid in WEKA implemented We 9.5 leot o nry(BLE) Energy Low Bluetooth Power (mW) 100 50 0 CB BLE 1.9

(d) 2.1

WiFi Acc 2.5 89.2 (a) (b) (c) (d) Figure 12: (a) CPU utilization (%) (b) Memory Usage (MegaBytes) and (c) Power Consumption (mili- Watt) (d) Accuracy measure for Bagging Ensemble Learning, RandomForest, LogitBoost and Packaged Naive Bayes classiﬁers

7. CONCLUSION [6] K. Lin, A. Kansal, Lymberopoulos, D., Zhao, F. Energy- accuracy trade-off for continuous mobile device location, Mo- Our iBeacon sensor assisted smartphone-based activity biSys (2010). recognition system holds promises to recognize activity in [7] E. Hoque, J. Stankovic, AALO: Activity recognition in smart real time with higher accuracy and lesser CPU, memory, homes using Active Learning in the presence of Overlapped cache and power usage. Our contribution is mainly divided activities, PervasiveHealth (2012). [8] F. Albinali, S. Intille, W. Haskell, M. Rosenberger. Using Wear- into two parts: first, we set up a Mobeacon smart home able Activity Type Detection to Improve Physical Activity En- with cheap available energy efficient hardware and smart- ergy Expenditure Estimation, UbiComp (2010). phone software system. Second, we extend two types of [9] M. Buettner, R. Prasad, M. Philipose, D. Wetherall, Recogniz- lightweight robust classification algorithms separately: 1) ing daily activities with rfid-based sensors, UbiComp (2009). [10] M. Alam, N. Roy, GeSmart: A gestural activity recognition bagging ensemble learning classifier; and 2) packaged Naive model for predicting behavioral health, Smartcomp (2014). Bayes classification algorithms. We combine iBeacon with [11] T. Dimitrov, J. Pauli, E. Naroska, Unsupervised recognition of smart-phone; and exploit an indoor localization based en- ADLs, SETN (2010). ergy efficient activity recognition framework which help re- [12] E. Miluzzo, C. Cornelius, A. Ramaswamy, T. Choudhury, Z. Liu, A. Campbell, Darwin Phones: The Evolution of Sensing duce the reliance on ground truth data collection and in- and Inference on Mobile Phones, MobiSys (2010). termediate server processing. We also design an effective [13] T. Pering, P. Zhang, R. Chaudhri, Y. Anokwa, R. Want, The smart-phone application interface for defining and creating PSI Board: Realizing a Phone-Centric Body Sensor Network, an initial semantic knowledge base about the smart home BSN (2007). [14] H. Lu, J. Yang, Z. Liu, N. Lane, T. Choudhury, A. Campbell, environment. We effectively use our semantic knowledge The Jigsaw Continuous Sensing Engine for Mobile Phone Appli- base, expression tree based activity construction, and in- cations, SenSys (2010). ference cache to accelerate the activity recognition process [15] D. Peebles, T. Choudhury, H. Lu, N. Lane, A. Campbell, of our lightweight bagging ensemble learning (BEL) based Community-Guided Learning: Exploiting Mobile Sensor User to Model Human Behavior, AAAI (2010). approach. Our proposed BEL attests the promise of our [16] Y. Wang, J. Lin, M. Annavaram, Q. Jacobson, J. Hong, B. methodologies and outperforms several existing lightweight Krishnamachari, N. Sadeh, A Framework of Energy Efficient classifiers in terms of CPU, memory, and cache occupancy Mobile Sensing for Automatic User State Recognition, MobiSys and power consumption performance. We also show that our (2009). [17] M. Keally, G. Zhou, G. Xing, J. Wu, Andrew J. Pyles, PBN: efficient duration model based high-level activity merging towards practical activity recognition using smart-phone-based provides higher accuracy in high-level activity recognition body sensor networks, SenSys (2011). even with less accurate low-level activity detection. [18] Suman Nath, ACE: Exploiting Correlation for Energy-Efficient and Continuous Context Sensing, MobiSys (2012). [19] R. Misra, S. Shukla, V. Chandel, Lightweight Localization Us- Acknowledgement ing Trilateration for Sensor Networks, IJWIN (2014). [20] Q. Sun, B. Pfahringer, Bagging Ensemble Selection for Regres- This work is supported partially by the NSF Award #1344990, sion, ACALCI (2012). UMB-UMBC Research and Innovation Partnership Grant, [21] I. Witten, E. Frank, Data Mining: Practicial Machine Learn- 2 ing Tools and Techniques with Java Implementations, Morgan and Constellation E : Energy to Educate Grant. Kaufmann (1999). [22] Estimote iBeacon, http://estimote.com/ [23] A. Parate, M. Chiu, D. Ganesan, B. M. Marlin, Leveraging 8. REFERENCES graphical models to improve accuracy and reduce privacy risks [1] L. Jiang, H. Zhang, Z. Cai, A Novel Bayes Model: Hidden of mobile sensing, MobiSys (2013). Naive Bayes, IEEE Trans Knowl Data Eng (2009). [24] Mark A. Hall, Correlation-based Feature Selection for Discrete [2] L. Atallah, G. Yang, The use of pervasive sensing for behaviour and Numeric Class Machine Learning, ICML (2000). profiling - a survey, PMC (2009). [3] Z. Yan, V. Subbaraju, D. Chakraborty, A. Misra, K. Aberer, Energy-Efficient Continuous Activity Recognition on Mobile Phones, An Activity-Adaptive Approach, ISWC (2012). [4] N. Roy, A. Misra, D. Cook, Infrastructure-assisted smart-phone- based ADL recognition in multi-inhabitant smart environments, PerCom (2013). [5] Y. Ji , S. Yu , Yafeng ZhangA, Novel Naive Bayes model: Pack- aged Hidden Naive Bayes, ITAIC (2011).