Human Activity Recognition Using Inertial, Physiological And

Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 1 Azra Bihorac, Senior Member, IEEE, In terms of algorithmic implementation, HAR research data presents manytational issues regarding requirements privacy [10].rich and While contextual compu- information, video privacy issuesled cameras limitations produce have many researchersembedded sensors, to including workpreserving depth alternative. with images as other a ambient privacy- and has seen an explosionsulting in in Deep an increase Learning inDL (DL) recognition methods, methods accuracy re- produce [5], high7]. [ ity While accuracy datasets, results in onLearning large many (CML) activ- models HAR mightsmall applications be size Classic of better the suited Machine data, dataset, due lower and to dimensionality availability the ofthe of the problem expert input [11]. knowledge The increasingsociated in interest with formulating in growing HAR use can ofin be sensors all as- and aspects wearable ofand devices daily well-being applications. life, This especially increasing interest withis in evident respect HAR from to the health numberﬁve of years, from papers 2015 published to ina 2019. the total As past of1.(a) Figure shows, 149based among selected on published DL models, papersDuring and on the 96 HAR, same were 53and based time were 20 articles on period proposing CML not were ML-based models. threshold methodologies published (e.g., models).1.(b) Figure 46 showsrecognition surveys the accuracy, average among activity thethe 53 96 DL-based CML-basedand papers papers, 92.2% and CML-based) that present almost asquality. the In same visible recognition addition, (93%2 Figure published DL-based shows HAR the distribution papers ofof over the (a) the CML past andof CML-based (b) ﬁve HAR DL years models models.the in was, It number except terms shows 2019, of that greaterwill DL-based the review than both HAR number DL-based models. andWe CML-based In will methodologies. this limitlimit paper, our we the review scope. to Interested non-image-based readers are sensors, encouraged to to read Senior Member, IEEE, ! Graziano Pravadelli, and, Parisa Rashidi, Comprehensive Survey Member, IEEE, —Human Activity Recognition (HAR), Deep Learning (DL), Machine Learning (ML), Available Datasets, Sensors, —In the last decade, Human Activity Recognition (HAR) has become a vibrant research area, especially due to the spread of Florenc Demrozi, Human Activity Recognition using Inertial, Physiological and Environmental Sensors: A NTRODUCTION Abstract electronic devices such as smartphones,learning smartwatches and and other video machine cameras learning presentwell-being algorithms in applications. has our For allowed daily example, researchers HAR lives. to In isdaily use addition, considered life HAR by the as monitoring in advance one various of their oflearning domains deep cognitive the in and including most developing physical sports, promising HAR function health assistive applications through and technology based daily tools on activities. to inertial This support survey sensors elderly’s focuses inIndex on conjunction Terms critical with role physiological of and machine environmentalAccelerometer. sensors. The goal of HAR is to recognize human activities in

1 I F.Demrozi and G.Pravadelli areUniversity of with Verona, the Italy e-mail: DepartmentP. [email protected] Rashidi of is Computer withFlorida, Science, the Gainesville, Department FL, USA of e-mail:A. Biomedical parisa.rashidi@ufl.edu Engineering, Bihorac UniversityTransplantation, is of College of with Medicine,USA the University e-mail: of abihorac@ufl.edu Division Florida, of Gainesville, FL, Nephrology, Hypertension, & Renal HUMAN ACTIVITY RECOGNITIONa (HAR) popular has topicin become in many the areas, lastsports, including decade and health due monitoring care,Besides, to systems interactive nowadays, its for gaming, the importance general agingof population the purposes world’s is [1]. primary becomingpopulation concerns. one aged It over was 65to estimated would that 2 increase the billion fromsignificant social 461 by and million 2050. healthphysical, care This functional, consequences. and To substantial monitor cognitivetheir health increase home, of HAR will older is adults emerging have in as a powerful toolcontrolled [2] and uncontrolledplications, settings. HAR Despite algorithmscluding myriad still face ap- 1) many complexityintra-subject challenges, and and in- inter-subject varietytivity, variability 3) of the for trade-off daily the betweencomputational same efficiency activities, performance in and ac- embedded 2) and privacy, 4) portableand devices, 5) difficultyand of testing data HAR algorithms annotationmain is sources, [3]. typically 1) obtained Data ambient from sensors, forAmbient and two 2) training embedded sensors sensors. cantemperature sensors or be video cameras environmentalpoints positioned in in sensors specific the suchintegrated environment into as [4], personal [ 5]. devicessmartwatches, Embedded such or sensors as are are smartphonescific integrated and medical into equipment clothes [6]–[9].used Cameras or have other in been spe- widely the HAR applications, however collecting video arXiv:2004.08821v2 [eess.SP] 19 Nov 2020 Nov 19 [eess.SP] arXiv:2004.08821v2 Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 2 Nweke et al. [18] (2019) provide a detailed analysis such 46 survey papers,exclusively we excluded video-based 23 papers HARprovides which unique papers. were contribution Our toproviding, the a survey review broad of paper vision literature ofin by the the past evolution 5 of years. HAR Unlikefocus existing on research surveys, the we algorithmic do details, notthe rather solely we will data also describe sourcesthis context. (aka We are sensors particularlysensors interested because and in they accelerometer devices) haveapplications shown and are because excellent their used results use insensors in in conjunction HAR with such other sensors as is physiological rising rapidly.sensors sensors The proliferation is or of strongly accelerometer relatedsure environmental the to movement their of abilityaccelerometer the sensors to human is directly body. affordable, mea- integrated In and into addition, most the wearable using sensors devices. Recently, can Wang.colleagues J be and [15] (2019)three survey aspects: sensor existing modality, literature DLscenarios, models, presenting based and detailed on application informationworks. of Wang. Y the and reviewed colleaguesof-the-art sensor [2] modalities (2019) in present HARtechniques the mainly state- associated focusing on with the sensors, each data step preprocessing, of featureactivities, learning, HAR including classification, both in conventional and termssides, DL methods. of they Be- present theing ambient sensor-based camera-based, HAR, and includ- ambient sensors. systems Sousa et combining al.state-of-the-art [16] wearable (2019) outline provide and of a the complete, context current of HAR inertial solutionset sensors in the in al. smartphones, [17]HAR and, (2019) Elbasiony systems introduce on aGyroscopes, portable detailed and inertial survey sensors Magnetometer), onare (Accelerometer, whose used multiple for temporal modeling and signals recognition of different activities. of data/sensors fusion andtechniques multiple for classification HAR systems able with devices. emphasis on Faustfocused on mobile et physiological sensors and used al. wear- intions healthcare [19] such applica- as (2018), Electromyography(ECG), (EMG), studied Electrooculogram Electrocardiogram (EOG), and 53 Electroencephalogram (EEG). papers Ramasamy [20] (2018) presentedand an data overview mining of ML techniques(AR), used empathizing for Activity withchallenges. Recognition Finally, the Morales fundamentaloverview et of the problems al. state-of-the-art concerning: and data relevant [21] signals, capturing (2017) and providecations preprocessing, an and calibrating orientation, on-bodyactivity selecting lo- the models right andusability set classifiers, of of a and HAR features, system. waysof Moreover, it repetitive to covers activities, the postures, evaluate detection falls,1 Table the andsummarizes inactivity. 23 surveys onchronological HAR order methods from sorted by 2019that to all 2015. these It surveys,sideration should (video-based), including be had those not noted notreview reported taken process their into systematic (e.g., con- Systematic using reviews Preferred and Reportingble1, Meta-Analyses Column Items five, (PRISMA)). we for In reportof the Ta- start/end the publication reviewed year papersnumber and of Column reviewed sixfocus articles. their on data approximate Most management methods of andmodels. activity these recognition To HAR the reviews best of our knowledge, no existing survey URVEYS S XISTING 3 Figure presents the standard workflow in designing 2 E Since HAR ismany surveys emerging have as beenAmong an the published initial in important 293 the46 published research past papers were few topic, that survey years. weing identified, papers survey published paperssources can since and be 2015. the categorizedwidely The activity used based recognition exist- data on sources algorithm.environmental are the devices, The a) data and most inertial, b)terms physiological video and of recording the devices.on In HAR CML models algorithm, and most more recently algorithms DL are algorithms. Among based The rest ofprovides the a brief paper overviewfrom is of 2015 organized the to existing as 2019,criteria, surveys Section3 follows: on Section4 describes HAR Section2 thewillCML, article DL, provide and selection existing background sensors/wearablewill devices. introduce material Section5 the definition on ofcategorization of human the activity, published followed works by indevice terms of (Section6). sensor Section7 and willfor present HAR available research datasets activity.papers Section8 basedwill review ontion9 published thewill discuss model the limitationsHAR and and research, challenges evaluation followed of by existing metrics.direction a Sec- discussion in on future Section concluding research remarks. 10. Finally, Section 11 reports some references on vision-based HAR [10], [12]–[14]. HAR-based methodologies. Whenapplication, developing the HAR-based first stepand is device to that determine is used the toThe type collect second of data step (device is sensor identification). to determinetion the process, details of including the the data annotationany collec- process necessary and possibly preprocessingstep includes (data identifying the collection).model appropriate and The machine training learning third thechine model, learning typically model aand on supervised training). annotated However, ma- as data shownthe in backwards (model arrow),3 Figure selection the(indicated selected by the model can preprocessing also influence datais step. evaluated In in thesuch as terms final accuracy, precision, of step, recall,evaluation). and the the other In metrics this model activity (model work,metric we recognition between use metrics the accuracy various asthe articles a only comparison due common to metric. the Notobtained fact all in that articles it terms present is the ofArea results Under precision, the recall, Curve sensitivity,acteristics (AUC) F1-Score, (ROC) or curve, Receiver Operating despitemetrics, Char- especially being with more unbalanced representative flow data. as Using a this reference, work- state-of-the-art this in paper HAR provides by an examiningcess. each overview Finally, phase of we of the are the particularly pro- sensors interested because in they accelerometer haveapplications shown and because excellent their results use insensors in conjunction HAR is with rising other rapidly.sensors The proliferation is of strongly accelerometer rectly related the movement to of their theaccelerometer human sensors ability body. is In to affordable, addition,integrated measure and using into most the di- wearable sensors electronic objects can people be own. Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 3 16 2019 92.2 15 2018 Classic Machine Learning Machine Classic 4 5 Model 2017 evaluation (b) (b) 13 2016 93.0 3 Deep Learning Deep Model 4 Training 2015 Selection& 0

5 0

90 80 70 60 50 40 30 20 10

40 35 30 25 20 15 10

100 papers learning deep of # accuracy recognition average lished papers wherekeywords mentioned found above. Our by keywords producedof a using total 249110 the records, among combinationthe which of quality we selected ofwere 293 the based selected publication on from venue.Electrical the and The Electronics following chosen EngineersComputing publishers: (IEEE), articles Machinery Association Institute for (ACM), of average Elsevier, number and of Sensors. citationsthe was The papers 46, for and each the yearour distribution is of shown retrieval in2. Table process4 Figure tematicshows based reviews on [37]. PRISMA First,and we template not excluded for accessible all sys- papers surveysexcluded). Next, ( papers we e.g., excluded requiring allvision-based books paid papers (4 access) (31 excluded) (91 and excluded). all Finally, we excluded all 2 Noise removal Noise extraction Feature Datacollection • • 8 2019 96 1 Device 36 2018 sensor”. All these pub- Identification 1 > Classic Machine Learning Machine Classic 8 2017 name (a) (a) < RITERIA Human Human Activity 22 2016 C 53 Deep Learning Deep 22 2015 ELECTION 0

5 0 80 60 40 20

35 30 25 20 15 10 40 100 120 papers papers # of papers of # # of classic machine learning learning machine classic of # 1. e.g., accelerometer, gyroscope, magnetometer, barometer, light, 3 S article is (1) presenting aexisting comprehensive surveys, meta-review (2) of providing a the comprehensivedifferent overview sensors, of (3) reporting andmetrics, comparing and performance (4) reporting onlarity. dataset availability and popu- Fig. 3. Standard workflow for implementing HAR based application. We used Googlebetween Scholar January to 2015 search tocluded September the for 2019. term studies Allin “human published searches combination activity in- with recognition,” “deep“wearable or sensors,” learning”, and “HAR” “machine “ learning,” Fig. 2. Distribution of published papers per year in HAR research based on (a) CML and (b) DL. Global Positioning System (GPS) Fig. 1. (a)published Distribution papers, for of DL published vs. CML papers implementations. in HAR research area, for DL vs. CML implementations. (b) The average recognition accuracy of Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 4 Papers - Reviewed 4 excl. 14 excl. # criteria (n=149) (n=149) (n=167) (n=167) (n=163) accelerometer based based methods Year Removing Removing video Removing Removing articles that the do not use inclusion/exclusion Start/End Records after applying after applying Records # 89 2008/2014 2011/2014 74 128 4 2013/20196 77 6 2001/20194 2005/20196 48 2005/20187 309 2008/20176 275 2007/2018/87 6 53 2010/2018 2012/20177 43 119 4 2010/20177 2010/20173 78 2005/20174 64 2007/20177 39 2010/20177 88 2005/20165 13 2010/2016 85 2005/2016 145 225 10 2006/2019 149 31 excl. Keywords 91 excl. 4 excl. surveys surveys (n=293) (n=293) (n=202) (n=198) on payment on payment and Removing Removing books Removing Removing articles on Google on Google scholar Records Records identified Fig. 4. PRISMA-based flowchart of the retrieval process. Used Keywords TABLE 1 293 Existing HAR Surveys Total online AR, real time,bile phone smartphones, sensing, mobile HAR phone, review,MEMS survey, accelerometer mo- sensor technologies,tions, research activity human in Italy, centered healthcare,physical rehabilitation, applica- activities, sportsensing science, safety, environmental Computing ML, transfer learning, wearable sensors Senior citizens, Activity recognition,Intelligent Internet sensors, of Aging, Things, Task analysis DL, health care,bioinformatics, genomics, biomedical electronic health informatics, records Artificial Neural translational Networks, DL, Time-SeriesHAR, Activity recognition, 3D action data,Inertial Depth sensor, sensor, Sensor fusion, Multimodal dataset Accelerometer, Gyroscope, ActivitySmartphone Recognition (AR), AR, Sensors, 3Aceelerometer, Smartphone, Survey, Processing ActivityState-of-the-art, of Energy Daily efficientworks, Human Living, wearable context recognition sensorartificial net- intelligent, 2007/2017 humanfeature extraction, body classification postureHuman recognition, activity monitoring,face, Human Wearable sensors, Computerterface, Smart Inter- Biomedical, sensors, Shared 60 control Multimodal architecture Bioinformatics, in- DL,imaging, health public health, informatics, wearable devices Wearable, ML, sensors, medical survey,classification, monitoring activity detection,AR, Sensors, activity ADL 3 2001/2015 138 HAR, ML, InertialGyroscope measurement unit, Accelerometer, Activity detection, Data fusion ,Multiple DL, classifier Health systems, monitoring, Multimodal sensors DL, Mobile and wearablesentation sensors, HAR, FeatureDL, repre- Physiological signals, Electrocardiogram, Electroen- cephalogram, Electromyogram, Electrooculogram HAR, Wearables sensors, DL, Features, HealthcareHAR, activity recognition, smartphones, mobileinertial phones, sensors, accelerometer, gyroscope,cation algorithms, ML, DL classifi- 5 2005/2019 258 Main Focus HAR Care ing care based DL Inertials based HAR Smartphones based HAR based HAR overview HAR care HAR based HAR SignalsHAR based data system based HAR on physiological signals Care in Smartphones based HAR TABLE 2 , Wearable sensors”. > 52 60 45 90 46 2015 2016 2017 2018 2019 Name Year < Publication Year [2] 2019 HAR in Health- [35][36] 2015 2015 Wearable based HAR in Health- [31][32] 2017[33] Wearable 2016[34] based 2016 DL for health- 2016 Wearable based Smartphone [27][21] 2017[28] Video 2017[29] 2017[30] and HAR 2017 Smartphones with 2017 Smartphones HAR general [20][23] 2018[24] 2018[25] ML based HAR 2017[26] active HAR learning, for activity recognition, Age- 2017 data mining, DL DL, for health- 2017 Time Series DL based HAR DL, Activity Recognition, Video, Motion 4 2010/2017 24 [17][18] 2019[22] Temporal 2019[19] 2018 HAR on multi 2018 Smartphones DL for health [15] 2019[16] DL based HAR 2019 DL, Activity Recognition, Pattern Recogniton, Pervasive Inertial Sensors Reference Total # of Papers following keywords: ”Human Activity Recognition (HAR), Sensor Distribution of the selected published articles for year by including the the papers that do notall use the papers accelerometers performing (4 activity excluded),daily recognition and life different human from activities, such asdriving, swimming, riding publications horses, prior tomachine learning 2015, techniques and such papers asexcluded). simple using As thresholding non- a (4 result,show. 149 were eligible, as1 Figures 4 and Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 5 Black-box models, interpretation isherent, not easy and in- Require large datasets for training, High computational cost. • • • Fig. 5. Exampleclassification of [44]. a Convolutional Neural Network (CNN) for image clusters where clustersmeasure are of combined/split dissimilarityis based between a on sets. probabilistic the of A model observations mixture for within model techniques representing an are subpopulations overall particularlydatasets population suitable lacking when labels [4].ity/dissimilarity between or These working classes is a when with primary[43]. outcome the [41]– measure of similar- 4.2 Deep LearningOn Overview the otherbecome side, popular in inperformance many recent [4]. domains, Since years, DL duerepresentation, DL is such to based techniques algorithms their on canoptimal have the superior automatically features, idea generate starting of from theany the data human raw input intervention, data, makingunknown without it patterns possible that to otherwise identifyunknown would the [44]. remain However, as hidden already or also mentioned, present DL some limitation models [45]: Because of such limitations,ods in are some areas preferred, stillquite especially CML small, when or meth- when the fastthe training training most is dataset common a requirement. is DLral Some algorithms of Network are: (CNN), ConvolutionalLong Recurrent Neu- Short-Term Neural Memory Networkscurrent Networks (RNNs), Unit (LSTMs), (GRU), Stacked Gatedvolutional Autoencoders, Re- Temporal Network Con- (TCN)(VAE) and [46]. VAriational Nowadays, Autoencoders cially CNNs in are the a imageimpose processing prevalent local research tool, connectivity community. CNN’s on espe- important the features raw by data viewinglocal extracting the pixel more image as patches.series a Furthermore, collection can a of also one-dimensionalsegments.5 Figure be time shows viewed anwith example as two of a convolutional CNN layers, architecture collectionlayer. each Instead, of followed RNNs by are localrepresented a a signal sequentially pooling proper alternative as whento time-series data deal data with is such and long-rangeone-dimensional designed temporal dependencies. sequences While cansulting be extracted fed features toized are a relations shallow. CNN, between Only thethe a closely re- feature local- few representations. neighborsStandard LSTM’s are RNNs are factored are an into comprised RNN of variant. interconnected hidden Supervised learning, Unsupervised learning. ACKGROUND Among unsupervised and particularly clustering algo- • • ıve Bayes (NB), k-Means Clustering, Support Vector Ma- ¨ 4 B chine (SVM), Lineardom Regression, Forests Logistic (RF), Decision Regression, Trees (DT) Ran- bours and (k-NN). k-Nearest Neigh- DT’s classifybased data on instances the by features/data sortingfeature values. them to Each be node classified,that represents and a each the branch node representsclassifiers can a based value assume. on NBindependence applying classifiers assumptions between Bayes’ are the theorem probabilistic based features. on SVMs with the are strong notion ofthat a margin-either separates side two ofthereby a data creating hyperplane the classes. mosttween Maximizing the significant separating the hyperplane possible and distance margin, theside, instances be- on has either beenexpected proven generalization error. to Finally, K-NN reducerithm is a that an CML stores upper algo- all availablebased bound cases on on and classifies a the newEuclidean, similarity cases Manhattan, measure Minkowski) [39]. (e.g., Furthermore,HAR since distance imposes functions specific constraints, as memory such as constraint, reduced and latency, classifiers, except computational for SVM, constraints, areenvironments appropriate these for given low-resource their lowrequirements. computational and memory rithms, the most well-known algorithmsarchical are clustering, and k-Means, Mixture Hier- models. K-Meansaims clustering to partition groups ofa samples into similarity k measure clusterssure based (intra-group) (inter-groups). on Each and sample belongs dissimilaritythe to mea- the nearest cluster cluster with a centers cluster or prototype. clustercluster Hierarchical analysis centroid, Clustering method serving Analysis that as is seeks to a build a hierarchy of The goal ofical supervised model learning based isoutput to on data create the and aunseen relationship mathemat- to data between points. use inputto In the and identify unsupervised model patterns learning,of in for the the input output predicting goal [4]. data Typically, is future onewill without or be any more also preprocessing required, knowledge including steps featuretion/segmentation, extraction, vectoriza- normalization orprojection standardization, [40]. and Some of the most commonNa supervised CML algorithms are: 4.1 Machine Learning Overview Machine Learning (ML) is a(AI), branch for of developing Artificial Intelligence algorithmspatterns that given can a identify traininginto and dataset two infer major [39]. classes: Such algorithms fall The mainhuman objective activity of basedenvironmental HAR on sensors algorithms data [15], gatheredactivities is [38]. by The is to wearable recognition mainlyRecently, recognize and the of use based these of a on wideinterest variety in CML of sensor sensors, fusion has and techniques.basic generated This ML DL section introduces and algorithms. sors DL market concepts, evolution, and wearable/environmental sensor fusion sen- techniques. Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 6 capturing 65.7% of2018. the [54]. market, an Smartcomplex almost devices innovations in 12% lend sensing more and themselveswhen actuation. than For to acceleration example, and increasingly inertialalgorithms sensors can are be available,ing implemented. HAR additional Furthermore, electronic byEnergy modules, (BLE) includ- such as andantennas Bluetooth Wireless or Local Low GPS, wearable Area devicesalerting can Network be and used (WLAN) determining for location real-time and to identify report risky activity situations ]. [55 smartwatches, In other addition types tosystems of with smartphones communication data and capabilities collectionInternet are of adding and Things to (IoT). sensing the 4.4 Sensor Fusion Techniques Each type ofFor sensor example, an provides accelerometer can benefits measurecannot acceleration, and but accurately disadvantages. evaluateSimilarly, velocity the or gyroscope positional canthe changes. detect magnetometer angular can velocities,However, and measure most sensors the can magneticmental easily noise, field be hardware deceived value. noise, byin or environ- imprecision external inputs, andaddress resulting uncertainty. these limitations Sensor by fusionsensors. combining The input techniques use from of various mogeneous) multiple combined with sources data (heterogeneous fusion techniques orseveral provides ho- advantages, including 1) noiseuncertainty, reduction, 2) 3) reduced increased4) robustness robustness of toknowledge the interference, of fusion the and perceived phase, the 5) signals number [56], integration [57]. of of Generally,more sensors as challenging. prior increases, The the mostods fusion common are step sensor becomes typicallyFilters, fusion based meth- and on Particleit Bayesian Filtering is possible estimation, techniques tohardware Kalman [58]. implement level Nowadays, these inside techniquesthe the directly application sensing at input the modules, andment, standardizing simplifying maintenance, application and develop- extensibility.of In the sensor future, fusion the techniquesplications use [27]. will Sensor span fusion aitations techniques wide by address range combining these lim- of theuse ap- input of from multiple variouscombined sensors. sources The with (heterogeneous or dataadvantages, homogeneous) including fusion 1) techniquestainty, noise 3) higher provides reduction, robustness, 2) 4) several integration robustness lower of to prior uncer- interference, knowledge 5) of[57]. the perceived Generally, signals the [56], morechallenging the is number the offusion sensors, fusion methods the step. are more TheFilter typically most and based common Particle onnowadays, sensor Filtering Bayesian, these techniques Kalman techniques [58].hardware are level Furthermore, directly inside imprintedthe the at sensing application the modules, inputopment, standardizing and maintenance, and simplifyinguse extensibility. application of In sensor devel- the fusionapplications, future, techniques given the the will specific spanand functionality the a of need wide to each obtain range sensor accurate of and robust estimations [27]. Y t t+1 t+2 Y Y Y sensors are employed to . The final hidden state after t t+1 t+2 2 H 2 H H − t H CO , 1 − t t , but also on the previous hidden state X t+1 t+2 X t X X at time based not only on the activation of the current t t X X H , updated by 1 These sensors are becoming more and more prevalent in − t H input units, each unitcell in that a contains Gated anof RNN internal gates that recurrence is controls the loop replaced flow andan of by information. a RNNs6 Figure a system shows that special operatesstate by sequentially updating a hidden 4.3 Sensors Sensors and wearablelife. devices The surround mostrecognition common us are types in accelerometers, ofsize our mainly sensors due daily and used toof low in their accelerometer activity small cost. sensorsaccelerometers are7 Figure used used inillustrates inincluding conjunction gyroscopes, with the HAR. magnetometers, others compasses, In sensors pressure sensors, prevalence many body cases, oximetry temperature sensors,kinds and sensors, of sensors electrocardiographs. electromyography, For have Many example, been the used other Globalor in Positioning different WiFi System applications. (GPS) aremicrophones sensors used and to Bluetoothinteractions determine are [48], the used user’s to and analyze location human [47], Fig. 6. Extended representationfor of an a example Recurrentunits, with Neural and an a Network single (RNN) input output44]. [ sequence of length three, three hidden processing an entire sequence containsits information previous from elements. LSTM all and GRURNN models are variants, successful alsoare known comprised as of Gatedin interconnected a RNNs. hidden Gated Basicinternal units. RNN RNNs recurrence loop is Each and substituted a unit the system by of information gates flow. a that Gated cell manages RNNsin have that shown modeling includes advantages sequential an series [44]. dependencies in long-term time- estimate the air qualityconstantly [49]. decreasing, such The size thatinto of they clothes these are [50], sensors beingobjects are integrated [52]. smart In glasses moredaily [51] advanced environment applications, and objectsIdentification are other (RFID) in tags. the wearable enriched The tagsthe make with user’s it possible in-house Radio tolaundry, activities infer washing Frequency (e.g., dishes) [53]. preparing coffee, doing our daily life [29]. Shipmentssmartwatches, of basic wearable devices, watches, including and wristmillion bands, reached units 34.2 duringyear over the year second [54].Fitbit, Companies part or as Samsung Xiaomi, of are Apple, pushing 2019, Huawei, forward with up new products 28.8% Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 7 2 91.5 2 89 4 84.5 1 98 4 93.5 1 95 2 93.5 7 93.4 (a) (b) 3 79.7 5 96.8 7 85.1 27 92.8 83 93 149 92.5 ATA 0 0

80 60 40 20 90 80 70 60 50 40 30 20 10

160 140 120 100 100

average accuracy average # of papers of # In addition to the direct measurements that such sensors 7 D Fig. 7. (a) Distributiongorized of by published the papers sensoraccuracy data in obtained source, HAR from and, research the (b) area papers average cate- using activity such recognition sensors. [87], light [60],radio signals90], [ (WiFi Passive andequipment Bluetooth Infrared ( [56], Sensor ECG [62],build (PIR) [87]), [56], in sensors medical [88]), [77], (GPS[93], [7], EMG [ 43], heart [72]) [87], [90]–[93], rate orstretch compass [91], [63], other [89], [97], type94]–[96], [ audioHAR. [62], of barometer [90], [91], [67], [98]) [73], are common [80], in provide, the indirectof usage smart of metrics thethe is measurements system promising [192] in or (e.g.,(RSSI) form the energy [56]) Received harvesting in Signalto Strength of order direct Indicator to measurements recognizevariations. from human Furthermore, the activity the bodyand related or importance smartwatches environmental of indue smartphones to HAR their isthese explosion devices increasingly currently among contain many clear, consumerssensors. of the mainly and Finally, aforementioned as givenreviewed shown that published in papers,8.(a), Figure are based the among mostly proposed on all standalonenumber HAR the of devices. smartphone- However, methods and the smartwatch-based total higher methods are than those based onshows standalone that devices. in8.(b) Figure termsbased of recognition on accuracy smartphone methodologies with and smartwatch those devices obtainedsmartphones are from in and standalone line smartwatchesdevices, devices. [193], provide Moreover, computational unlikepossible to capabilities standalone directly that executedevice, and make HAR in models it manydevices cases, on used they the in have wearable the a medical very field). high cost (e.g., The second step of thedata HAR type. workflow Such regards data the can collected mainly be categorized as follows. HAR N I EVICES D CTIVITY A OURCE S UMAN ATA 3 Table and4 Table respectively show the sensor/device Among all ADL’s, the most popular activities in HAR 5 H 6 D The first step of the HARthe workflow includes data identification source of sensor/device to7.(a),Figure be small, used, low-cost and, and asas non-invasive shown accelerometers, sensors in gyroscopes, such andmost magnetometers commonly are used and the depicted appropriate in sensors7.(a), Figure in 149 HAR. papersused used As gyroscopes accelerometers, in 83 addition toa accelerometers, magnetometer and in 27 addition to used all the the accelerometer. Therefore, selected papersleast use one at accelerometer least inFurthermore, one combination7.(b) Figure accelerometer with shows or the other averagenition at sensors. activity accuracy obtained recog- form combination of such device. type and provide referencessors/device. to Besides,3 the Table and papers4 Table Three usingshow to in such Five, Columns sen- theaverage average number number of of tested recognized datasetsof activities, and testing the subject. average These number sensors tables like illustrate accelerometer, theHowever, gyroscope, importance other and of type magnetometer. (temperature of [7], sensors as [60], environmental [76], sensors [79], [87]–[89], humidity [ 79], The definition of ActivitiesADL’s of are Daily the Lifeeating, (ADL’s) activities bathing, is dressing, that broad. working,leisure we homemaking, and all enjoying perform ofment. daily, these Our activities such review involving as overview of physical of HAR move- the scientific most studied literature ADL’s. presents an research are walking,upstairs running, and standing, walkingactivities sitting, downstairs. However, have walking other beencluding type complex explored of activities, in suchcooking [59], the as house the cleaning past [4], differentsmoking [ 59]–[61], few phases [66], driving62]–[65], [ of years, swimming [67],Several in- or studies biking [6], focuslocations, [43], on such [64], as [68]. activities sitting[71], on performed the on walking/standing ground, lying specific ing/running in on on bed a the [69]– treadmill,cising walking on elevator in a a stepper [71]–[74], parking [71],[75]. lot, or exercising walk- exer- Other on a detailed crosscific trainer movements movement [71], of recognition theobject, arms, involves such releasing spe- as it, carrying/reachingthat an frontal people elevation, can[78]. perform and A in other major relation activities population area to of and other HAR the objectswith research [76]– involves increasing physical the and of agingHAR cognitive the of models function number are impairments.avoid being of risky Many used situations, people to such[85] help as or falls users Freezing in recognize ofFurthermore, elderly and Gait activity people (FoG) [79]– tracking inpopular devices Parkinson’s for are disease monitoring becoming [38]. approximate ADLs. physiological very and Those physical parametersheart devices such rate, as are blood pressure, able steps,consumed. level to changes, Advanced and devices calories the can neurological recognize stages of sleeping sleep(stages and (i.e., cycling 1-4) through nREM andinformation can be REM) used as [86]; input to furthermore, HAR algorithms. all the stored Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 8 } 32 22 26.9 45.38 43.74 84.25 29.45 65.59 Average Average } # Subjects # Subjects 1 1.32 1.33 1.44 2.09 1.28 1.48 1.18 Average Average # Datasets # Datasets 7 17.4 12.84 14.22 17.45 21.09 15.63 10.55 Average Average # Activities # Activities 7.3 Environmental sensors data The environmental datarepresenting covers the all stateperature, of the humidity, the pressure, collection or environment, of brightness.suring including However, the data mea- tem- status ofmental the environment measures. goes It beyondsures can environ- related also to people includeFor and more objects example, complex inside recognizing the mea- theenvironment environment. number and of their peopleon inside position a the or certain object thein inside many actions the application performed environment scenarioshealthcare, could related and to be service human delivery. useful assistance, 5 Table shows the categorization of theon revised articles based the typethe of data data, type wheredata and Column types. the Columns reference Oneaverage to Three and number the to Two of articles Five show of using recognized respectively tested such activities, show datasets, average the andHowever, as number average we number discussed earlier, of theon largest testing daily amount life of subjects. is data collected viaphones, electronic devices, smartwatches, such as activity smart- trackers,and smart video cameras. thermostats Asdevices shown like in8, Figure smartphone the andthe use smartwatch use of is smart of outnumbering standalonestandalone devices. column It identifies shouldsmartphones all be and those noted devices that smartwatchesand the other dedicated as than instruments, such forFlorida/USA), or as Bioharness3 example, (RAE Actigraph Systems (Actigraph, clinical California/USA). by Honeywell, Furthermore, during the data collection TABLE 4 TABLE 3 Device based paper categorization Sensor based paper categorization , humidity, or PIR. [7], [8], [56],], [60 [69], [71], [73], [74],76], [ [77], [82], [89], [90],111], [ [7], [43], [56], [60], [63], [72]–[74], [76], [77], [79], [80], [87]–[98], [107], [4],9], [7]–[ [32], [42],43], [ ]–[58], [56 [60], [69], [71]–[74], [76]–[80], [82], [7], [43], [59], [62],74], [ [75], [87], [90], [95], [117],135], [ [ 147],149], [ [4],9], [6]–[ [32],], [42 [43],61], [56]–[ [63],68]–[85], [ 176], [87]–[ ] [176]–[190 2 [4], [6], [32], [56]–[58], [60], [ 63], [68]–[72], [76]–[78], [83]–[85], [88], [ 89], [7]–[9], [32], [42], [43],59], [ [ 61],73], [ [74], [79]–[82], [87], [90]–[93], [99], [76], [79],87]–[89], [ [91],97], [ [98],107], [ [152],154] [ [87]–[91],93], [ [95], [99]–[101], [107],[127]–[129], [110],]–[139], [137 111], [ [142]–[145], [113]–[120],], [147 ]–[125], [122 [148],[157], [150]–[152],159], [ [154], [160], [155], [ 162],164]–[166], [ [190], [173]–[175],] [191 179], [ [ 182], [184]–[186], [116],], [119 [124],], [125 [134], [136],137], [ [143],148], [ [125], [150],], [152 152], [ [154], [166] ], [164 [184] Article Reference temperature, humidity, light, PIR, WiFi, Bluetooth, heart rate, barometer, strech, audio, medical devices [94], [96]–[98], [100],[118]–[121],124]–[126], [ [ 102]–[104], [131]–[133],106]–[108], [ [136]–[140],144]–[146], [ [153], [ 111], [148],], [157 [151], [158], [112], [160],162]–[164], [ [114]–[116], [166],167], [ [170],171], [ [183],189] [ Article Reference [101], [105], [109], [110], [113],[143], [114], [149], [ 122], [150], [123], [152],]–[130], [127 [154]–[156], [ 134],[176], [159], [141]– ]–[182], [176 [],161 [ 165], [184]–[188],168], [ 190], [ [ 169], [191] [172]– [152],], [154 [185] { CO temperature, humidity, light, presence, WiFi, Bluetooth, ECG, EMG, GPS, compass, heart rate, barometer, strech, audio { Other = Other Other Inertial sensors datascopes, such magnetometer, or as compass, accelerometers,Physiological sensors gyro- data such asRate, ECG, or EMG, blood Heart pressure, Environmental sensors datapressure, such as temperature, Gyroscope Other= Standalone Smartwatch Smartphone Source Sensor Accelerometer Source Device Magnetometer • • • 7.2 Physiological sensorsPhysiological Data sensors perceive physiologicalin signals, contradiction which with other sources(facial, of emotional gestures, knowledge andtages. speech), Those providing signalsare are essential mostly quite advan- involuntary insensitivecontinuously and, measure to as the deception. such, affectiveused physiological They events. signals [195] are can brain Thebeat, electrical be activity, most heart- muscle used electricalconductance to acquired activity, by blood the followingsition pressure, external system: data and acqui- Electroencephalogramgram skin (EEG), (ECG), Electrocardio- and Electromyography (EMG). 7.1 Inertial sensorsAccelerometer, data gyroscope, and magnetometera sensors with maximum ofavailable nine at degrees a very of lowlar freedom cost. velocity Besides, are are acceleration commercially the andhuman most angu- common activity. data This usedin to is characterize the reinforced previous by section,gyroscopes are what given the that we most widely accelerometersinertial described used and sensors devices the in are HAR.applications. widely Such [194] used in clinical and healthcare Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 9 11 45.54 19.59 Average # Subjects 1.32 1.57 1.47 Average # Datasets 12.88 12.71 20.76 Average # Activities 8 Table presents a categorization of the reviewed papers two [202] to thirty-threestudied [205]6). activities (Table The are mostas primary common activities walking, of running,walking daily downstairs, sitting, and life, sleeping. standing, such walking upstairs, 7.4 Preprocessing and FeatureThe Extraction mentioned data sourcestion generate identifying time-series the informa- statuscharacterized by of noise, which the makes device. ittheir difficult raw However, to state. be data used The is in presencecessing of the noise raw is data handledpare to by the eliminate prepro- data this for interferenceThe being and feed preprocessing pre- to is the oneHAR recognition and of models presents [35]. the different mostsuch noise as important management digital techniques, phases and statistical in feature filters, data extraction. normalization, The and featuressically extraction step tow explores domains: ba- time,Time-domain frequency features and are spectral thecheaper domain. most than used because thethe they transform frequency are from domainstandard time classification features models to are because not frequencythis suitable of domain for phase raw [35]. data, iswhich Since time-series anticipated sensor data is by segmentedfeatures. before a extracting Besides, segmentation manylapping step methodologies part during maintain between an twoprovides the consecutive over- model segments. with knowledge This7 ofTable part thepresents previous context. an overviewtime on and the frequency most domain features. commonly used based on the utilization offrequency noise domain removal, features time domain, extractionOne and techniques. to Columns Four show:DL), the if machine any learning noise category removalor (CML technique frequency-domain is or features used, were ifcontains extracted. time-domain Column the Five referencesthe to number the ofColumns papers, papers Seven using and andused features such Column Eight and the configuration. show Six, averageConcerning Finally, the activity the recognition CML-based average accuracy. models,reviewed number as articles shown, of most8, (Tab. ofand row the frequency 7) domain features, maketially and pre-processed use with the noise raw of removal8,Tab. techniques. data both Instead, row was time ini- 3 showsdomain features articles without that applyingnique. any use However, noise time other removal and tech- methodologies9) frequency 8, (Tab. do rows not 8and make and in use some cases, ofas the in any presence [4], features of [43], noise [60], extractionmethodologies [94], are is based technique, [98], not on [].170 the considered, In miningand of [ 4], their temporal symbolic [60] patterns representation, and or [170], asmake in the [43] use were, authors of clustering technique, discriminating between TABLE 5 Data source used in HAR paper. 18 89.4 Smartwatch Smartwatch 69 91.8 (a) (b) Smartphone Smartphone [4],6]–[9], [ ], [32 [42],43], [ [7],], [56]–[61 56], [ [63], [72],68]–[85], [ 77], [ ] [87]–[191 ], [89 [7], [94]–[96] 60], [ [73],76], [ [79], [80], [87]–[91], [98], [] 192 Article Reference 77 93.2 Standalone Sensors Standalone Standalone Sensors Standalone 0

Inertial

10 90 80 70 60 50 40 30 20

# of papers of # 100 accuracy recognition Data Type average Physiological Environmental Column One refers to the name and the article proposing step, sometimes activitiesmanner (aka are scrippted). performed Thatpatterns is in are because very human a hard movement subject controlled to and recognize intra-subject variability. due Sucha to variability considerable the entails difficulty large inmanages inter- developing to a generalize methodology among that data all collected subjects. from a Also, very thehelp large lack number researchers of of find subjects a does solution not toWith this regard problem. to suchknown issue, and6 Table openshows source datasets some for of HAR the studies. best the dataset. Columnthe Two dataset, presents Column the ThreeColumn shows activity the Four labeled number shows in sensing of activities. devices. the Column numbernumber Five of and and subjects type from Columnthe whom number of Six the of citations data show the that wasber the the collected used dataset 2019. and received Such by datasets Septem- gyroscope, are largely and based magnetometer onsensors sensor accelerometer, are embedded data. into smartphones Mostand and the smartwatches, of number such of activities in these datasets ranges from Fig. 8. (a) Distributiongorized of by published the papers deviceaccuracy in data obtained HAR source, by research and the area (b) identified cate- devices. Average activity recognition Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 10 1020 72 194 30 635 910 397 10 319 120 9563 204 8 1939 394 17 80 264 1939 367 gyroscope in uncontrolled environments ter, gyroscope) 2 onone the on wrists, theankles waist, and 2 on the eter, gyroscope surement unitsgyroscope, (accelerometer, magnetometer) leg, trunk) magnetometer, ECG) right wrist (accelerometer, gyroscope, magnetometer) and leftcelerometer, gyroscope, ankle magne- (ac- tometer) 8 smartphonewatches and 4 smart- controlled environments) arm, right leg, lefton leg each 9 sensors unitters, (x,y,z x,y,z gyroscopes, accelerome- x,y,z magnetometers) netometer, 4D9 quaternions positions: on leftright calf, calf, left right thigh, thigh,lower arm, back, left left upper arm,lower right arm, right upper arm ter(controlled environments) sors 21 ambient sensors Furthermore, concerning the DL-based methodologies, 13 5 motion sensors (accelerome- 1018 20 accelerometers2 3 colibri wireless inertial mea- 12 1 3 accelerometers (ankle, upper chest (accelerometer, gyroscope, 6 38 19 accelerometer, gyroscope from 5 units on torso, right arm, left 33 accelerometer, gyroscope, mag- 17 Smartphone accelerometer 30 75 18 23 body sensors12 12 object sen- MotionNode accelerometer 14 180 since DL networkswithout perform human automatic intervention,learning feature algorithms, unlike the extraction traditional majorityof of machine- any them Noise Removal do and notin Feature make Extraction8, Tab. step use rows as shown 10 and 14. The achieved average accuracy, TABLE 6 Publicly Available Datasets for HAR research right-circle, turn left,jog, jump, turn push right, wheelchair upstairs, downstairs, standing up fromstanding up laying, from lying sitting,stairs, running, down upstairs, sitting walking, from down, jumping down- standing, down cending and descending stairs,still standing and in moving anparking elevator around lot, in walkingkm/h an on (in elevator, a flat walking treadmillon and with in a 15 a treadmill a with deg speedstepper, a exercising of inclined speed on 4 positions), of aercise running 8 cross bike km/h, trainer, in exercising cycling horizontal onjumping, on and a and an vertical playing ex- positions, basketball rowing, walking, jogging,front and running, back,open/closed jump , jump sideways, jumpstretched), rope, jump trunk trunk up, leg/arms twist twist (elbows (armsforward, bent), jump waist out- waist bends rotation,with waist opposite bends hand), (reach reacheral foot bend heels (10 backwards, to lat- bend the with left + armright), 10 repetitive up to forward the stretching, (10and right), upper lower lateral to trunk body theof opposite left arms, twist, frontal lateral +claps, elevation elevation 10 frontal of to crossingamplitude arms, of the rotation, frontal shoulders arms, hand tation, low-amplitude shoulders ro- arms high- to inner the rotation, breast,knees heels knees (alternating) bending (alternating) tobending the (crouching), forward, backside, rotation kneeselliptical on (alternating) bike, the cycling knees, rowing, door gaps, open door, closecheck door, open/close trunk gap, two open/close doors, trunk, check steering wheel nordic walking, watching TV, computering, work, ascending car stairs, driv- descendinging, stairs, ironing, vacuum folding clean- soccer, rope laundry, jumping house cleaning, playing gait block) climbing stairs, waistof bends forward, arms, frontal kneesrunning, elevation jump bending front and (crouching), back cycling, jogging, sandwich, eat sandwich, cleanup, break,fridge, open and close: dishwasher, drawers,lights, drink door standing, drink sitting 1, door 2, on/off ning forward, jumping, sitting,vator up, standing, elevator sleeping, down ele- [207]WARD stand, sit, lie down, walk forward, walk left-circle, walk UniMiB SHAR[82] ActiveMiles[206] Activities of daily life 7 Smartphone accelerometer and DSADS[204] Sitting, standing, lying on back and on right side, as- REALDISP[205] HHAR[6]WISDM v2[196] biking, sitting, walking, standing, jogging, upstairs, walking, downstairs, sitting, stair standing up and 6 stair Smartphone accelerometer (un- Skoda[200][201]PAPAM2 write notes, open engine hood, close engine hood, check lying, sitting, standing, walking,Daphnet[202] running, cycling, mHealth[203] freeze (gait block), no freeze (any activity different from standing still, Sitting and relaxing, lying down, walking, UCI-HAR[198]USC-HAD[199] walking, upstairs, downstairs, sitting, standing, laying walking: forward, left, right, upstairs, 6 downstairs, run- Samsung Galaxy S II accelerom- DatasetWISDM v1[196]][Opportunity walking,197] jogging, upstairs, downstairs, sitting, Start, standing groom, relax, prepare Activities coffee, drink coffee, 6 prepare Smartphone accelerome- # Activities Data Sources # Subjects Citations different human activities.terms of About accuracy, the the methodologies thatremoval results make methods use obtained of and noise in frequency feature domain extraction show inby promising the the results time as numberconfiguration. and also of shown methodologies that make use of this Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 11 7 17 91.1 Other Other Other 17 17 91.5 LSTM LSTM LSTM (c) (a) (b) 8 95 14 RNN RNN RNN 11 30 93.7 CNN CNN CNN 0

8 6 4 2 0 5 0

90 80 70 60 50 40 30 20 10

18 16 14 12 10 35 30 25 20 15 10

# of activities of # average # of papers of # 100 accuracy averagerecognition Fig. 9. a) DistributionAverage of activity Deep recognition Learning accuracy Modelsused of mostly in Deep used Learning in HAR, HAR,Models Models and mostly b) mostly used c) in Average HAR. number of activities of Deep Learning 8.2 Machine Learning (ML) basedAmong methodologies the 14995 reviewed presented an papers, HAR Figure as methodology10 basedshows shown (a) on in the classicalobtained distribution ML. 1, Figure average of these accuracy models,recognized and (b) activities the (c) of daily the life.of average Among the number classical different types of MLwas models, the Support the Vector Machine most[60], (SVM) model69], [ commonly [78], [4], used [6],79], [ model [ 58], [82],85], [ [92],95], [ [109],118], [ [127], VALUATION E ODELAND M LASSIFICATION Concerning the Noise Removal step, 48 CML-based ar- 8 C 8.1 Deep Learning (DL)The based DL methodologies models, asof shown in the1 Figure comprised 149distribution 54 papers papers of we DLaverage models reviewed. accuracy, and (c) among9 Figure the thedaily averageshows number life 54 of activities (a) recognized for articles, each the is model. (b) The the most the Convolutional popular model referenced Neural in Network 30 (CNN), papers[99], which [100], [7], [ 104], was [32], [106], [ 73],[141], [108], [143], [ 112], [77],146], [ [119], [147], [81], [120], [149],[191], [88],150], [ [ 125], [153], [90], [126], [214], [154],160], [ [ 215]. [180], accuracy The of CNN 93.7% modelsnumber in obtained of activity 11 an recognition activities average model over of was daily an life. the average Thewhich Long second was Short-Term most Memory used used[112], (LSTM) [125], in130], [ model, 17 [137], [139],[216], papers152], [ [218]. [153], [7], [172], It176], [ [83], obtained [213], an an [89], average average number [102], accuracy ofNeural of [107], 17 Network 91.5% activities (RNN) over of were[129], daily used [213], life. in [216], Recurrent [8],obtaining [56], an217], [ [89], average over [ 112], the accuracy papers an of (indicated average by 95%. Othermodels Finally, number in such the9) Figure as of rest where Autoencoders basedNetworks of 14 [71], on (INN), or [123], the Inception other frameworks7 Neural [105] papers for with a an total average of number accuracy of of 17 91.1% activities and of an daily average life. The third andidentification fourth and step evaluation of of theis the classification model used HAR that for workflow activity2,Figure includes CML recognition. models As still enjoy shownto great in popularity compared 1 Figure thoseand basedadvanced models on such as the themany DL relatively articles models. We made point moreand out use recent that not of just and differentand one more classification as model mentioned models for inparison achieving Section1 metric betterwe between performance, use theaccuracy accuracy various is the articles. as only This a common beacouse metric com- among them. among all thesebased 34 articles do articles, make wasfeatures, use frequency of 93%. time-domain domain Besides,8,8, (Tab. (Tab. rowand row other 12) 11) frequency DL- and domain bothDL-based8, (Tab. time rows models 13 eliminateto and the process 14) data latency with features. models due the require above to a techniques.ML the more However, models such considerable and need longer amount training of times. data than ticles and 12 DL-basedremoval articles techniques. make Among useused all of different ones such noise techniques are:[127], the z-normalization and linear most [75], interpolation102], [ normalization [120], [ 111] steps, are min-max the preceded most [70], bythe used a application of filtering outlier stepworth detection based [82], [70], [101], on [117], [117], [163], [123],[189], Butter- median [ 127], [74], [128], [],101 [152], [ 117], [127], [155],[183], [132], [174], [147], high-pass [],155 [92], [ 182], [96],statistical [117], [58] [ 128], filters. [169], [173], [208], or Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 12 Average Accuracy Recognition Average Number of Features 1820 19 5627 92% 90% 8934 93% 0 93% # of Papers ıve Bayes (NB) [4], [42], [94],122], [ ¨ ISCUSSION Frequency Domain Features 1) fast fourier transform(DFT), (FFT) 3) coefficients, discrete wavelet 2) transform discrete5) (DWT), fourier 4) ratio transform first dominant between frequency, power, 6) the ratio between power thethe at power at total the frequencies power, higher 7) dominant than twoin 3.5 signal frequency FFT Hz fragmentation and and spectrum, features, 10) 8)of the DC energy the total component spectrum, wavelet coefficients, 11)and 13) entropy energy squared spectrum, of sum 12)crossing the of sum wavelet the rate, coefficients, wavelet 16)entropy 14) coefficients values, auto-correlation, 19) spectral 15) mean frequency, entropy, mean- 20) energy 17) band spectral energy, 18) wavelet 9 D of 93.5% over anused average models of are 8 the[136], activities Na of [142], daily [159], life.namic [169], Other Bayesian [171], Network (DBN)184], [ Markov [101], [185], Models [103], (HMM) [210], [166], [68], Hidden Extreme the [ 69], Dy- Learning [151], [ 179], MachineComponent [182], Analysis [ 208], (ELM) (PCA), [153], Linear(LDA), Discriminant Quadratic [154], Discriminant Analysis Principal Analysis[134] (QDA) and [84], [109], many[110], others [116],117], [ [9], [121], [124],43], [ [156],128], [ [158], [133], [59], [135], [161], [70],140], [ 163], [ [144], that74]–[76], [ [170], some [174], [82], of [188]. the It articlesdifferent models. have is tested noteworthy their approaches using In this paper, we providedresearch. an HAR overview of is the anition, current HAR pervasive critical computing, research and areaments. human In in assistive the environ- activity last decades, recog- and with the with rise of growing newis needs technologies becoming such even more as essential. aging In recent population, years, HAR DL-based Papers Reference TABLE 7 TABLE 8 [6], [59], [68],72], [ [75], [82], [92],101], [ [110], [117], [61], [95], [109], [124],], [131 [134], [138], [140], [148], [9], [69], [80], [84], [85], [91], [113], [114], [116], [122], [42], [121], [156], [164][4], [43], [60], [94], [],98 [ 170][90], [106], [218][8], [83][56], [71], [125], [154][88], [102], [103], [108], [],111 [123], [ 126], [147], [150], [152], [180] [172] 6 4 8 0 0 0 90% 3 4 89% 4 90% 341 2 20 285 92% 9 92% 90% 93% [76], [78], [79], [93], [ 166][58], [151][63], [187][57], [70], [74], [87], [ 96], [162], [168], [169], [174], [183] 10 5 13 0 92% 94% 2 2 0 68 94% 88% [7], [32], [],73 [ 77], [81], [89], [ 97], [99], [100], [104], [105], [107], [112], [115],[141], [119], [143], [ 120], [146], [129],[213]–[217] [149], [130], [ 153], [137], [160], [139], [176], [181], [191], [157], [159], [165], [167], [],184 [ 185], [188], [209], [210] [135], [136], [142],[212] [144], [ 145], [177], [186], [208], [211], [118], [127], [128],[171], [132], [173], [ 133], [175], [155], [178], [179], [158], [ 182], [161], [189], [163], [190] Most used Time and Frequency domain features Preprocessing and Feature extraction on the reviewed papers. Domain Features Frequency ---- Time Domain Features Noise Removal DL DL DL DL DL DL ML CML CML CML CML CML CML CML CML CML Model 1) maximum, 2)mean square, minimum, 6) 3)weighted range, mean, variance, 7) median, 4) 11)density 8) function, standard interquartile skewness, 13) deviation, range, percentiles 9)above (10, or 12) 5) kurtosis, below 25, 10) empirical root percentile 75,above (10, time- and cumulative or 25, 90), below 75, 14) and percentileabove sum 90), (10, of 15) or 25, square values 75, below sumdeviation, and of percentile 18) 90), values 16) (10, meansignal number 25, power of vector deviation, crossings 75, magnitude,of 19) and 21) sum signal 90), of covariance, magnitudestandard 17) range 22) area, deviation of mean simple 20) of a amplitude average moving a signal, of signal, average 23) sum of 25) sum variances maximum of of slope a range signal, of of 26) simple autoregression. a moving signal, 24) sum of Time Domain Features [131], [132], [ 136], [138], [142],[173], [145], [182],155], [ [184], [168],186], [ [169], [187],171], [ was [189], [190], used [209]–[212] which in92.3% 35 over an papers, average of achieving 12model activities. an is The the second average most classical accuracy used k-Nearest[6], Neighbor of [42], (kNN) [ 60], [61], model [69], [4], [122], [ 78], [127], [79],136], [ [92], [142], [ 95], [145],which [96],162], [ was [ 113], [164], used [118], [169], in173], [ 23 papers, [186], of achieving 93.7% an average over accuracy anthird average and of fourth 12(DT) most activities model [6], of used [78], daily [85], model[159], life. [94], [165], are173], [ [95], The [177], [113], the [178], [136],[212],184], [ Decision [142], [185], which [145], [193], Tree was208], [ [210], usedaccuracy in of 19 94.2% papers,life, over obtaining and an an the average average [80], Random of [82], Forest 8 [92], (RF) activities [93],used [6], [95], of in [ 57], [96], daily 15 [175], [69], papers, [185],over [72], obtaining [ 212], an [78]– which an average was average ofused accuracy 10 of model activities 93.3% of is daily[98], the life. [114], Neural The [136], fifth [142], Networkswhich most [145], (NN) was [ 148], used in [157], 14 [173], [4], papers, [183]–[185], obtaining [78], an average [92], accuracy Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 13 1 14 6 1 112 Unavailable datasets Unavailable 5 6 HAR algorithm performance HAR algorithm 4 5 3 30 21 # of datasets used to test to test used datasets # of Available datasets Available 2 28 0

5 0 80 60 40 20

30 25 20 15 10 120 100 # of papers of # # of papers of # Furthermore, the lack of public heterogeneous datasets Fig. 12. Number ofarticles datasets methodologies (x-axis) (y-axis) used tested toarticles on test were such an tested number article on of and only datasets number one (166 dataset). Fig. 11. Availability ofologies. datasets used to evaluate the proposed method- research community for the identiﬁcation ofand the best benchmarking models thestarting from results. the Asof initial surveys 293 shown and papers indatasets, on and payment Figure only after articles, the 3011, which among removal datasets are a shown are in total publicly the of6. Table available, 142 some of reduces the possibility ofgeneralization creating HAR capabilities. models This within better is the becausecontrolled the investigated data environment. papers used Thisthe are inter-subject problem and collected intra-subject is variabilityscripted primarily exacerbated absent datasets, in in as by such mosttested a proposed on HAR a modelsa are limited single only number controlled environment. ofHAR Among activities models, the and 87 149with models captured analyzed the were in remaining 62 testedshown tested on in on Figure a more12, than singlewere we one dataset, found tested dataset. that As on 28three HAR two datasets, methodologies datasets, lessdatasets, 21 and than HAR only 10 one methodologiestotal methodology HAR on of [219] 14 methodologies was datasets. tested on This on situation a 4-6 shows the challenge of 18 27 90.1 10 12 92.7 7 3 94 8 6 95.8 8 19 94.2 (c) (a) (b) 8 14 93.5 10 15 93.3 12 35 92.3 9 23 93.7 0

8 6 4 2 0 5 0

90 80 70 60 50 40 30 20 10

18 16 14 12 10 20 35 30 25 20 15 10 40

# of activities of # average # of papers of # 100 accuracy recognition average HAR methods have producedrecognition excellent performance. However, results CML-based in approaches are terms still of widely used, andwithout they the generate computational outstanding costs. results the However, in reproducibility recent of years, MLimportant. models Based has on become ourHAR increasingly research, methodologies, for the 78% results ofdue the are to proposed not proprietary fully datasets. reproducible This results in barriers for the Fig. 10. a) Distributionactivity of recognition CML Models accuracy mostly ofc) used CML Average in number Models of HAR, mostly activities b) of used Average CML in Models HAR, mostly and used in HAR. Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 14 CML Server Normal CML Local <10 DL Server Large CML Local HAR ONCLUSION Server CML/DL 11 C algorithms is thecan lack generalize of to standardized heterogeneous methodologiesby set a that of activities diverse performed learning set could of reuse users. thelem As knowledge a to acquired potential solve inacquired solution, one a based transfer prob- similar on problem.a a For specific specific example, inertial bodydifferent knowledge sensor location sensor positioned can location on or potentiallysensor. with The be a extent reused different to which type within transfer of various a learning scenarios, inertial can is be notmanner helpful investigated and in needs a to comprehensive provides be further a studied. promising Sensor path.sensors fusion In could also particular, address merging issued different relatedracy to of reliability and a accu- singleformation. When sensor data and from one couldsystem modality could also is switch not enrich to reliable, a collectedrobust different the sensor in- data modality to collection. ensure grained Another activity recognition research based on direction examininginteractions. daily This is object will fine- allow ussequence to recognize of sub-actions actions and information and to will downstream provide applications.also much Sensor fusion richer be can context helpfulor when proximity sensors ather are large attached advance number to the ofof daily progress recommendations. inertial objects. First, in To sensors developing this fur- should benchmark area, datasets be we amodels priority provide should for a be the set on compared HAR benchmark community. with New data availablecreation HAR to HAR of models show datasetsand with improvement. diverse an set Furthermore, of adequate activitiesgrained number is activity strongly of recommended. recognition subjects Fine- alsoscale, could standardized benefit benchmarks. fromHAR Researchers large- algorithms working shouldand on system also issues, pay besidesing solely attention HAR developing algorithms. to andprimary On-device hardware improv- goal, computation as well should astery be analysis consumption, to a of explore memory, CPU, theutilization trade-off and between bat- and resource recognitionorientation accuracy. dependence Finally, should position beerwise, extensively and studied; the oth- design ofniques position/orientation-dependent could tech- resultstream in applications. inconsistent and non-robust down- HAR systems havethe past become decade, a achieving growing impressive progress. research In area partic- in Normal Local CML/DL >10 DL Server Large Local CML/DL IRECTION D Final Choice Final # of Activities of # Amount of Data of Amount ESEARCH Computation model Computation R Human Activity Recognition Activity Human UTURE the number of activities tothe be amount recognized, of available (labeled)local data, or remote computation. • • • Another significant issue concerns the interpretability Regarding the HAR models,9 Figure and Figure 10 10 F Based on reviewedtions papers, are a noted few below. One possible of research the direc- main limitations of HAR We observed that themodel selection is of primarily thements and precise based the DL on amount orIn of terms the CML of available the computational training sensors, thedispensable, (labeled) require- most sensor data. widely is used the used, accelerometer, ifin which not conjunction can in- with be other used the sensors magnetometer. such as the gyroscope or identifying a methodology superior to the others. of the results, mainlymethodologies and related tested to on papersachieve the almost presenting same the similar dataset, same results claimingtion in accuracy. to terms Such an of issue activity ising recogni- related commercial to tools, tests lack performed ofwho us- open source do code, and not authors the publicly heterogeneity of provide themethodology their data that source and can code. theby recognize definition people Besides, the of with different a activitiescollides physical HAR carried and with motor out the characteristics datawe have sources seen, used a fordata variety data of collection. sensors collection. However, and As theusually devices proposed very are methodologies used rigid for regarding are becomes the difficult data to source. have Specifically, a it ular methodology individual tested by on making asubsequently use partic- of changing a the particularhave sensor(s) sensor and model. different Various technicaltheir sensors specific characteristics, state, which e.g.,that the also a measurement specific error entail sensor or presents. the noise show that CMLcomplex models DL-based are models.require still This a used smaller is amount more becausecomputational of requirements. widely training CML In data, addition, than models DL asherently models well difficult are as to in- interpret. lower Nonetheless,a DL unique models ability have to recognizemaintaining more high complex accuracy. activities, In while addition,a they data do not preprocessing require workflow stage. for developing Figure HAR13 applications basedshows on: a suggested Fig. 13. Model selection diagram. DL=Deep Learning, CML= Classic Machine Learning Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 , , , . 15 Wiley . IEEE, , vol. 2018, , 2015, pp. 6th International , vol. 19, no. 14, , vol. 51, no. 2, pp. Expert Systems with , vol. 119, pp. 3–11, , vol. 46, pp. 147–170, Proceedings of the IEEE Sensors , vol. 521, no. 7553, pp. Computer methods and pro- IEEE communications surveys , 2015, pp. 4305–4314. nature Expert Systems with Applications Proceedings of the IEEE conference on Information Fusion European conference on computer vision Biocybernetics and Biomedical Engineering , vol. 90, pp. 138–152, 2015. , vol. 181, pp. 108–115, 2016. , “Deep learning for computer vision: A , “Human activity recognition using inertial , vol. 161, pp. 1–13, 2018. Pattern Recognition Letters , vol. 4, pp. 1810–1820, 2016. , “A survey of sensors in healthcare workflow , “Learning human identity from motion pat- , “Data fusion and multiple classifier systems , vol. 49, pp. 46–56, 2019. , “Deep learning algorithms for human activity , “Long-term recurrent convolutional networks et al. , “Convolutional neural networks for human et al. . Springer, 2019, pp. 734–745. , “Deep learning,” , “Smart devices are different: Assessing and mit- . ACM, 2015, pp. 127–140. et al. , “Deep learning analysis of mobile physiological, , “A survey on wearable sensor modality centred , “Deep learning for sensor-based activity recog- Computational intelligence and neuroscience , “Deep learning for healthcare applications based , “Action recognition with trajectory-pooled deep- et al. et al. ACM Computing Surveys (CSUR) , “Sensor-based human activity recognition sys- et al. et al. , 2019. , “From action to activity: sensor-based activity recog- et al. , “Spatio-temporal lstm with trust gates for 3d human et al. , vol. 15, no. 3, pp. 1192–1209, 2012. et al. et al. et al. et al. et al. et al. Neurocomputing et al. IEEE Access et al. et al. Proceedings of the 13th ACM Conference on Embedded Networked 2019. on physiological signals:grams A in biomedicine review,” chine learning forInterdisciplinary human activity Reviews: recognition—a Datavol. survey,” 8, Mining no. 4, and p. e1254, Knowledge 2018. Discovery smartphones, a survey,” vol. 37, no. 3, pp. 388–400, 2017. recognition using mobile andthe wearable art sensor and networks: research Statevol. challenges,” of 105, pp. 233–261, 2018. igatingmobile sensing heterogeneitiesin for activitySensor recognition,” Systems sensors in a smartphone: An overview,” for human activity detectionopen and research health directions,” monitoring: Review and 2014, pp. 197–205. environmental and locationInformation sensor Fusion data for emotion detection,” terns,” 2625–2634. 436–444, 2015. convolutional descriptors,” in computer vision and pattern recognition action recognition,”Springer, in 2016, pp. 816–833. brief review,” 2018. nition: A survey,” 2019. p. 3213, 2019. nition based on temporalInternational signals Conference of on portable Advancedand inertial Machine Applications Learning sensors,” Technologies in human activity recognition in health care,” activity recognition usingConference on mobile Mobile sensors,” Computing, Applications in and Services tem withKnowledge-Based a Systems multilayered model usingfor time visual recognition series andconference description,” shapelets,” in on computer vision and pattern recognition monitoring,” 1–37, 2018. Applications recognition using wearable sensors,” & tutorials nition,” EFERENCES 12 ACKNOWLEDGMENT R [19] O. Faust [20] S. Ramasamy Ramamurthy and N. Roy, “Recent trends[21] in ma- J. Morales and D.[22] Akopian, “PhysicalH. activity F. Nweke recognition by [6] A. Stisen [7] E. Kanjo [8] N. Neverova [11] Y. LeCun [12] L. Wang [14] A. Voulodimos [15] J. Wang [16] W. Sousa Lima [17] R. Elbasiony and W. Gomaa, “A survey on human activity recog- [18] H. F. Nweke [9] L. Liu [10] J. Donahue [13] J. Liu [2] Y. Wang [3] O. D. Lara and M. A. Labrador, “A survey[5] on human activity M. Zeng A.B. and P.R. wereNational supported Institute by R01 ofreceived GM110240 General grant from the Medical NIH/NIBIBREER 1750192. Sciences. 1R21EB027344 The content and P.R. isauthors solely has and NSF the does responsibility not of CA- necessarily the of represent the official the views NationalFoundation. Institutes of Health or National Science [1] R. S. Antunes [4] Y. Liu

Deep Learning September 2019 September Survey Existing (Section II) (Section ) Video Video Based Human Human Human Human Activity Activity Activities (Section V) (Section Recognition (not of interest of(not Data Time Feature Domain Domain Extraction Frequency (Section VII) (Section (Section VIII) (Section (Section VII.D) (Section Sensor Based Performance Evaluation Inertial Inertial Sensors Data Sensors Sensors Data Sensors Sensors Data Sensors Physiological Physiological (Section VII.C) (Section (Section VII.B) (Section (Section VII.A) (Section Environmental Environmental Source Source Devices

Standalone (Section VI) (Section Smartwatch Smartphone Devices Type Devices Classic Machine Learning Machine Classic January 2015 January surveys, we analyzed themost reviewed widely literature studied based humanused activities on electronic (Section the sensors V), as the thethe most data most source (Section known VII),(Section devices and that VI) integrate withoutmethodologies. with taking these In into sensors detail, accountphysiological, sensor-based inertial, the data and video-based primary perceived environmental interest. sensors by Device were types of ied, were categorizing also them extensivelyand stud- c) in: smartwatch a) devices.shown For standalone, in each terms b) of category, the resultsties, smartphone, average were the number average number of of recognized datasetsologies, activi- used to and test the method- thecussed average methodologies based accuracy.and This on magnetometer. accelerometer, survey gyroscope, We alsoapproaches also dis- and discussed theirnoise the results removal, and preprocessing based normalization techniques. ondiscussed Moreover, we datasets feature primarily extraction, inpublicly the available literature, datasets. Finally, emphasizing wetion presented of a the descrip- recognitionpurpose, models we most have used presented inML the HAR. models most For widely and this used theirof DL results, quality and both (accuracy) fromactivities). and the We quantity point concluded (number of thatclassic of view HAR ML recognized researchers models, still mainlyamount prefer because of they data requiremodels. a and However, smaller the lesspacity DL computational in models recognizing power many have complexshould shown than activities. focus Future higher DL work on ca- more the advanced development generalization of capabilitiesof more and methodologies complex recognition with activities.a To summarize, Graphical Figure Abstract15 (GA) ofshows the workflow of this survey. Fig. 14. OverviewHAR of research results the from proposed 2015 to survey 2019. structure on sensor-based ular, sensor-based HAR haveto many vision-based advantages HAR compared concerns methodologies, and which are posements. privacy constrained Activity recognition byDL algorithms computational are based require- becoming onHAR central ML methodologies in between and January HAR.2019. 2015 Figure and Starting14 September fromsummarizes a meta-review of the existing HAR Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 . . 16 . IEEE, 2019 Design, arXiv preprint , vol. 13, no. 6, , vol. 22, no. 5, pp. , vol. 150, pp. 304– Proceedings of the 7th Proceedings of the 10th Proceedings of the 12th Pervasive and Mobile Com- . Machine Learning Mastery, . ACM, 2010, pp. 43–56. , vol. 16, pp. 251–267, 2015. Neurocomputing . IEEE, 2018, pp. 102–107. IEEE Sensors Journal Proceedings of the 8th ACM Conference on , vol. 5, no. 2. Multidisciplinary Digital 2018 IEEE International Conference on Informa- , “An energy harvesting wearable ring plat- , “Adaptive mobile activity recognition system , “Sensloc: sensing everyday places and paths , “Activity detection and analysis using smart- , 2018. , “An indoor localization system to detect areas , “Deep ehr: a survey of recent advances in deep , “A hierarchical approach to real-time activity et al. et al. , “Multi-scale conditional random fields for first- Master Machine Learning Algorithms: discover how they , “Soundsense: scalable sound sensing for people- et al. et al. et al. Informatics , “Comon: Cooperative ambience monitoring platform et al. et al. et al. et al. . ACM, 2014, pp. 162–175. et al. , vol. 8, no. 1, pp. 115–130, 2012. and smartwatch accelerometernition,” data in for physicalPublishing Institute, activity 2018, recog- p. 29. phone sensors,” in tion Reuse and Integration (IRI) centric applications oninternational mobile conference phones,” on in Mobile systems, applications, and services with evolving data317, streams,” 2015. learning techniques forIEEE electronic journal health of record biomedical1589–1604, and (ehr) 2017. health analysis,” informatics arXiv:1801.00631 using less energy,”Embedded in Networked Sensor Systems ACM, 2009, pp. 165–178. with continuity and benefitinternational awareness,” conference in on MobileACM, systems, 2012, applications, pp. and 43–56. services person activityPervasive recognition and Mobile on Computing elders andform disabled for patients,” annual gestureinput international on conferenceservices surfaces,” on Mobile in systems, applications, and pp. 2284–2291, 2013. recognition in body sensorputing networks,” causing the freezingAutomation & of Test in gait Europe Conference2019, & in pp. Exhibition (DATE) parkinsonians,” 952–955. in work and implement them from2016. scratch heat for wearable sensors,” continuing their ascendance whilehttps://www.idc.com/getdoc.jsp?containerId=prUS44000018, wristbands face flataccessed: growth,” 2019-09-23. [41] C. Dobbins and R. Rawassizadeh, “Towards clustering of mobile [42] A. Vaughn [43] Z. S. Abdallah [44] B. Shickel [45] G. Marcus, “Deep learning: A critical appraisal,” [48] H. Lu [49] Y. Lee [50] K. Zhan [51] J. Gummeson [53] L. Wang [46] J. Brownlee, [47] D. H. Kim [52] V. Leonov, “Thermoelectric energy harvesting of human body [54] , “New wearables forecast[55] from idcF. Demrozi shows smartwatches , , , IEEE arXiv Sensors , vol. 17, Biosensors . springer, IEEE journal . IEEE, 2017, pp. Annals of internal , vol. 19, no. 6, pp. 2017 fourth international 2018 Fourth International , 2019. Computer Communications IEEE Sensors Journal , vol. 5, no. 3, 2017. , vol. 21, no. 1, pp. 4–21, 2016. Multimedia Tools and Applications , vol. 37, pp. 23–44, 2017. , 2017. , vol. 15, no. 1, pp. 2059–2085, 2015. , vol. 55, no. 10, pp. 78–87, 2012. Briefings in bioinformatics . IEEE, 2017, pp. 473–477. , “Classification and suitability of sensing , “A survey on activity detection and clas- , “A review of ambient intelligence based ac- , “A recent survey for human activity recogini- Sensors , vol. 90, pp. 298–307, 2017. , “Towards a wearable system for predicting the , “A survey of online activity recognition using et al. et al. , “A survey on human activity recognition using , “Increasing trend of wearables and multimodal , “A survey of depth and inertial sensor fusion , “Preferred reporting items for systematic reviews et al. Pattern recognition and machine learning , “Deep learning for healthcare: review, opportuni- International Journal , “Mems sensor technologies for human centred , “A survey of energy-efficient context recognition et al. , “Human body posture recognition—a survey,” in , “Deep learning for health informatics,” et al. et al. et al. et al. Commun. acm et al. . IEEE, 2018, pp. 1–6. et al. et al. et al. et al. et al. , vol. 151, no. 4, pp. 264–269, 2009. et al. ı ` tion based on deep learningconference approach,” in on image information1–6. processing (ICIIP) for human action recognition,” journal of biomedical and health informatics 2006. learning.” tivity recognition for ageing citizens,”Conference on in Advances in Computing,(ICACCA) Communication & Automation preprint arXiv:1701.01887 vol. 76, no. 3, pp. 4405–4425, 2017. smartphone,” systems usingPervasive wearable and Mobile sensors Computing for healthcare2017 International applications,” Conference onApplications Innovative (ICIMIA) Mechanisms for Industry vol. 89, pp. 34–50, 2016. mobile phones,” applications in healthcare, physical activities,mental safety sensing: and a environ- reviewvol. 15, on no. research 3, activities pp. in 6441–6468, italy,” 2015. and meta-analyses:medicine the prisma statement,” freezing of gait in people affected by parkinson’s disease,” interface for human activity monitoring: A review,” sification using wearable sensors,” ties and challenges,” 1236–1246, 2018. and Bioelectronics of biomedical and health informatics no. 2, pp. 386–403, 2016. technologies for activity recognition,” [27] C. Chen [39] C. M. Bishop, [40] P. M. Domingos, “A few useful things to know about machine [23] K. Y. C. Keng Fig. 15. AClassical Systematic Machine Learning Review (CML) of andwearables Human Deep or Learning environment. Activity (DL), We which Recognition surveyedstep make methodologies (HAR) use (noise based of approaches, removal/feature on data extraction published collectedrecognition sensor by technique), accuracy, type, from b) sensors device and the (inertial January type average or finally, 2015 (smartphone, number physiological), their smartwatch, of embedded to DL studied into standalone), September activities, or preprocessing and 2019, CML c) based model. the average on The number results of datasets are used presented to in test the terms methodology. of a) average activity [26] J. K. Dhillon [36] G. Ciuti [38] F. Demrozi [28] V. V. Vyas [29] T. Rault [30] P. Patel [31] P. Kumari [35] M. Shoaib [37] D. Moher [24] R. Miotto [32] D. Rav [25] J. C. B. Gamboa, “Deep learning for time-series analysis,” [33] M. Cornacchia [34] P. Woznowski Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 . . , - 17 Ex- , pp. arXiv Applied Applied , vol. 19, . IEEE, , vol. 18, 2015 IEEE International , vol. 52, no. Proceedings of , vol. 5, no. 6, . IEEE, 2019, Twenty-Fourth . Acm, 2015, , 2015. Sensors , vol. 18, no. 5, p. Future Generation . IEEE, 2018, pp. , vol. 46, pp. 1181– IEEE Pervasive Com- 2018 IEEE 14th Inter- Proceedings of the In- Cluster Computing 2019 IEEE International , vol. 30, no. 3, pp. 22– Sensors 2018 Congreso Internacional de IEEE Internet Computing , 2016. IEEE network Workshops at the Thirtieth AAAI Con- , vol. 59, pp. 235–244, 2016. Measurement and Control , 2016. , “Deep, convolutional, and recurrent Proceedings of the 12th ACM International , “Mobile big data analytics using deep , “Deep activity recognition models with Procedia Computer Science , “Dealing with imbalanced data sets for , “Activity recognition using accelerometer , “Iot system for human activity recognition et al. , “A robust human activity recognition system , “Deep learning to predict falls in older adults , “Human activity and heart rate monitoring , “Automatic annotation for human activity et al. , “Recognizing detailed human context in the , vol. 81, pp. 307–313, 2018. et al. , “Unimib shar: A dataset for human activity , “Smart phone based data mining for human et al. , vol. 62, pp. 915–922, 2018. et al. et al. et al. et al. , “Human activity recognition from smart watch et al. , vol. 16, no. 1, p. 115, 2016. , “Adaptive multiple classifiers fusion for inertial et al. et al. et al. , “Sensor-classifier co-optimization for wearable hu- , “Deep convolutional neural networks on multichan- , “Multimodal wearable sensing for fine-grained ac- et al. ´ es et al. et al. ˜ , vol. 7, no. 10, p. 1101, 2017. nez and D. Roggen, “Deep convolutional and lstm recur- et al. et al. et al. ´ o Sensors , vol. 16, no. 4, pp. 62–74, 2017. . ACM, 2019, pp. 45–48. tivity recognition in healthcare,” recognition using acceleration data from smartphones,” activity recognition,” 1187, 2015. no. 5, pp. 26–35, 2015. nel time series for human activity recognition,” in models for human activity recognition using wearables,” activity recognition based onInternational single Conference accelerometer,” in on2015, Systems, pp. Man, 1488–1492. and Cybernetics Sciences 1–14, 2018. International Joint Conference on Artificial Intelligence preprint arXiv:1604.08880 Conference on PErvasivements Technologies Related to Assistive Environ- wild from smartphones andputing smartwatches,” based on daily-life trunk accelerometry,” 1654, 2018. sensor and machine learningnational classifiers,” Colloquium in on Signal Processing & Its Applications (CSPA) sensor based human activity recognition,” wearable sensors,” in recognition in free living using a smartphone,” sensor data using arandom hybrid forest of algorithm,” principal1-2, component pp. analysis 37–45, 2019. and learning and apache spark,” 29, 2016. IEEE, 2018, pp. 233–238. no. 7, p. 2203, 2018. smartphones using machineJournal for learning Innovative algorithms,” Researchpp. in Science 31–37, & 2018. Technology using bioharness 3ternational Conference and on smartphone,”ACM, Future 2017, Networks in p. and 49. Distributed Systems system in aInnovacióny mobile Tendencias en platform,” Ingenier´ıa(CONIITI) 1–6. in man activity recognition applications,”Conference in on Embedded Software andpp. Systems 1–4. (ICESS) human activity recognition2018. using mobile phone sensors,” in smartphone sensors usingpert deep systems learning with applications neural networks,” Computer Systems rent neural networkstion,” for multimodal wearable activity recogni- triaxial accelerometers,” in ference on Artificial Intelligence celerometer data usingSoft convolutional Computing neural networks,” sensors by deep convolutional neural networks,” in using smartphone sensors and deep learning,” the 23rd ACM internationalpp. conference 1307–1310. on Multimedia [87] D. De [88] J. Yang [81] Y. Chen and Y. Xue, “A[82] deep learningD. approach Micucci to human [86] G. Chetty [89] N. Y. Hammerla [91] Y. Vaizman [83] A. Nait Aicha [84] A. A. Sukor [90] I. A. Lawal and S. Bano, “Deep human activity recognition using [92] F. Cruciani [96] J. Manjarr [85] Y. Tian [93] S. K. Polu and S. K.[94] Polu, “HumanC. Rodriguez activity recognition on [95] S. Balli [97] A. Nk [98] K. T. Nguyen [99] C. A. Ronao and S.-B. Cho, “Human activity recognition with [104] A. Ignatov, “Real-time human activity[105] recognitionM. A. from Alsheikh ac- [100] W. Jiang and Z. Yin, “Human activity recognition using wearable [102] F. Ord [103] M. A. Alsheikh [101] M. M. Hassan . , , , Sensors . IEEE, ESANN PloS one , vol. 140, 2018 IEEE- IEEE Transac- . ACM, 2018, . International Proceedings of the , 2019. , vol. 16, no. 2, p. . ACM, 2019, pp. . IEEE, 2016, pp. Proceedings of the 23rd , vol. 190, pp. 35–49, . IEEE, 2018, pp. 379– Proceedings of the 2nd Sensors Engineering Applications of Arti- 2018 International Conference on . ACM, 2018, p. 72. International Journal of Distributed Procedia Computer Science , vol. 55, pp. 105–115, 2020. Proceedings of the International Con- IEEE Transactions on Biomedical En- Neurocomputing , vol. 17, no. 9, pp. 1991–2001, 2018. 2018 15th Learning and Technology Confer- 2018 IEEE Wireless Communications and Twenty-fourth international joint conference , “Human activity recognition using mul- , 2015. , vol. 15, no. 12, pp. 31 314–31 338, 2015. , “Robust human activity recognition using , “Analysis of multi-sensor fusion for mo- , “Smoking activity recognition using a single , “Multimodal human activity recognition from , “User-independent recognition of sports ac- , “Context-driven active and incremental activ- et al. , “A body sensor data fusion and deep recurrent arXiv preprint arXiv:1906.03033 et al. , “Swimming style recognition and lap counting , vol. 72, pp. 190–202, 2018. , vol. 15, no. 5, p. 1550147719849357, 2019. , “Iot based mobile healthcare system for human , “From user-independent to personal human ac- , “Sensor based human activity recognition using et al. , “Smartphone-based activity recognition for in- Sensors , “Online human activity recognition using low- , “Human activity recognition via smart-belt in et al. et al. et al. et al. , “Action2activity: recognizing complex activities , “Energy efficient human activity recognition using , “Mixed-kernel based weighted extreme learning , “Physical human activity recognition using wear- et al. Information Fusion 2016 International Joint Conference on Neural Networks et al. , “Bi-view semi-supervised learning based semantic , “Deepsense: A unified deep learning framework for . ACM, 2018, pp. 22–26. . IEEE, 2018, pp. 29–34. et al. et al. et al. et al. et al. , vol. 63, no. 4, pp. 788–796, 2015. et al. et al. et al. . IEEE, 2016, pp. 381–388. et al. et al. ˜ nazco et al. et al. wearable inertial sensors using machine learning,” in 189, 2016. EMBS Conference on BiomedicalIEEE, Engineering 2018, and pp. Sciences 402–407. (IECBES) door localization using avol. convolutional neural 19, network,” no. 3, p. 621, 2019. ity recognition,” 2018, pp. 468–473. wearable sensors,”Networking in Conference Workshops (WCNCW) 383. ence (L&T) neural network-based behaviorhealthcare,” recognition approach for robust bile and wearableProceedings sensor based of human theapplications activity international recognition,” in conference on datawireless body processing area and networks,” wearable acceleration sensors attached to the body,” 2016. autoencoders with wearable sensors,” tivities frommatching-based a approach,” singlegineering wrist-worn accelerometer: Aadaboost template- ensemble classifier,” pp. 104–111, 2018. activity recognition using multiplesensors,” accelerometer in and gyroscope (IJCNN) activity recognition,” in tiple smartphoneInnovations sensors,” in Science, in Engineering and Technology (ICISET) Sensor Networks on artificial intelligence human activity recognition usingtions accelerometers,” on Mobile Computing from sensor data,” in vol. 10, no. 7, p. e0130851, 2015. activity recognition on2016 smartwatches IEEE usingCommunication International deep Workshops Conference (PerCom learning,”1–6. Workshops) on in Pervasive Computing and ference on Computer-Aided Design time-series mobile sensing data26th processing,” International in ConferenceWorld Wide on Web Conferences World Steering Committee, Wide360. 2017, pp. Web 351– tivity recognition models using smartphone2016. sensors.” in wrist imuinternational and conference deep onpp. learning digital 48–51. light,” signal processing in machine forwith inertial imbalanced sensor dataset,” based human activity recognition power wearable devices,” in using a smartwatch and deep learning,” in International Symposium on Wearable23–31. Computers smartwatches and smartphones,” ficial Intelligence able sensors,” [56] M. Z. Uddin [72] A. A. Badawi [73] B. Zhou [74] G. Civitarese [80] G. Ding [58] Y. Zhu [76] A. Subasi [79] A. K. M. Masum [57] H. F. Nweke [61] M. Arif and A. Kattan, “Physical activities monitoring using [71] L. Wang, “Recognition of human activities using continuous [75] J. Margarito [77] S. Ha and S. Choi, “Convolutional neural networks[78] for human A. Subasi [59] M. Lv [60] Y. Liu [62] S. Bhattacharya and N. D. Lane, “From smart to deep:[63] Robust G. Bhat [64] S. Yao [65] P. Siirtola [66] E. V. A [67] G. Brunner [70] D. Wu [68] R. San-Segundo [69] F. Attal Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 , , , 18 2018 , vol. 18, . IEEE, IEEE Access Computing in . IEEE, 2018, . IEEE, 2018, Sensors Applied Soft Comput- 2018 IEEE Symposium 2018 15th International , vol. 171, pp. 754–767, . IEEE, 2018, pp. 463– , 2018. Mathematical Problems in . IEEE, 2018, pp. 1–6. - , vol. 2018, 2018. Multimedia Tools and Applica- IEEE Internet of Things Journal . ACM, 2018, pp. 72–75. arXiv preprint arXiv:1810.12176 , 2019. , 2019. , vol. 19, no. 3, p. 458, 2019. Neurocomputing , 2018. , “Human activity recognition by wear- Sensors , 2018, pp. 349–357. , “Transition-aware human activity recogni- , “A robust deep learning approach for , vol. 21, no. 5, pp. 16–25, 2018. Proceedings of the 2018 ACM International 2018 24th International Conference on Pattern , “Human action recognition using triaxial Journal of Sensors et al. . IEEE, 2018, pp. 165–170. 2018 IEEE International Symposium on Medical , “Human physical activity recognition using Multidisciplinary Digital Publishing Institute Pro- , “Human activity recognition from the acceler- , “Improving activity recognition for missing et al. , “Activity recognition with multiple wearable , “Iot wearable sensor and deep learning: An , “Semi-unsupervised learning of human activity et al. , “On the use of ensemble of classifiers for , “Comparison of different sets of features for , “Design of novel deep learning models for real- , vol. 18, no. 11, p. 3726, 2018. , “Deep residual bidir-lstm for human activity , “Sensor data acquisition and multimodal sensor , “Attention-based convolutional neural network et al. , “Adding structural characteristics to distribution- , “From user-independent to personal human activ- , “Human activity recognition from inertial sensor , “Temporal alignment improves feature quality: an , “Efficient human activity recognition solving the et al. et al. , “Activity recognition system for dementia in smart , “Extreme learning machine-based deep model for et al. , “Sequential human activity recognition based on et al. 2018 40th Annual International Conference of the IEEE et al. et al. , vol. 2018, 2018. , “Human activity recognition based on convolutional et al. , “Towards unsupervised physical activity recognition et al. et al. EAI Intl. Conf. on Mobile Computing, Applications and ´ et al. e et al. et al. IEEE Sensors Journal et al. et al. et al. et al. et al. Sensors et al. , vol. 2, 2018, p. 1242. et al. et al. , vol. 19, no. 7, p. 1716, 2019. et al. et al. , vol. 76, no. 8, pp. 10 701–10 719, 2017. , vol. 37, pp. 1018–1022, 2015. for weakly labeledsensors,” human activities recognition with wearable sensors for industrial applications,” in recognition using wearableEngineering sensors,” human activity recognition by wearable sensors,” integrated approach for personalizedin human activity a recognition smart home environment,” time human activity recognition with mobile phones,” in position-independent smartphone-based humannition,” activity recog- deep convolutional network and extremewearable learning sensors,” machine using human activity recognition with wearable sensors,” Series on Computational Intelligence469. (SSCI) no. 12, p. 4189, 2018. vol. 6, no. 5, pp. 8553–8562, Oct 2019. International Joint Conference on2018, Neural pp. 1–8. Networks (IJCNN) smartphone sensors,” Science & Engineering using smartphone accelerometers,” tions accelerometer-based activity recognition,” ing based accelerometer representationsing for activity wearables,” recognitionSymposium in us- on Wearable Computers homes based on wearable sensor data,” in 2016. accelerometer data: selective approach,” in Multi-Conference on Systems, Signals &pp. Devices 491–496. (SSD) ity recognition models exploitingarXiv preprint the arXiv:1905.12285 sensors of a smartphone,” data,” in Services Student Workshop neural network,” in Recognition (ICPR) confusing activities viavol. 7, deep pp. ensemble 75 490–75 499, learning,” 2019. tion using smartphones,” using deep generative models,” fusion forSensors human activity recognition using deep learning,” pp. 1–4. ation data of aby wearable device. activities?” which in features areceedings more relevant 2018. able sensors:applications,” Comparison in ofMeasurements different and Applications classifiers (MeMeA) for real-time experiment on activityProceedings recognition of with the accelerometerRecognition IEEE data,” Workshops Conference in on Computer Vision and Pattern time-series usingworks,” batch in normalizedEngineering in deep Medicine lstm and Biology recurrent Society (EMBC) net- [151] A. Malais [152] Y. Zhao [150] K. Wang [153] J. Sun [145] S. Rosati [146] V. Bianchi [147] M. Nutter [148] R.-A. Voicu [149] B. Almaslukh [154] X. Niu [157] C. Catal [155] J.-L. Reyes-Ortiz [156] Y. Lu [132] A. Mimouna [133] H. Kwon [134] P. Siirtola [141] W. Xu [144] C.-F. Su [135] M. Willetts [140] T. Hossain [142] M. Espinilla [143] R. Zhu [136] G. De Leonardis [137] S. Chung [138] H. Choi [139] T. Zebin , , JMIR arXiv Int. J. , vol. 1, , vol. 19, , vol. 16, , vol. 18, , vol. 4, no. 4, . IEEE, 2017, Sensors Sensors Sensors Proceedings of the Proceedings of the Journal of Parallel Artificial Life and IEEE Sensors Journal IEEE Transactions on Journal of Ambient In- Proceedings of the ACM Information Processing & 2018 41st International Con- . IEEE, 2018, pp. 1–6. , vol. 9, no. 4, pp. 1049–1060, , vol. 7, pp. 9893–9902, 2019. , vol. 18, no. 7, p. 2146, 2018. Signal, Image and Video Processing . ACM, 2018, p. 16. IEEE Transactions on Industrial Infor- . IEEE, 2018, pp. 1126–1131. JMIR mHealth and uHealth , vol. 118, pp. 67–80, 2018. Sensors , 2019. , vol. 17, no. 4, pp. 160–165, 2017. , “Sleep quality prediction from wearable 2017 IEEE 14th International Conference on IEEE Access , “Real time human activity recognition on , vol. 13, no. 6, pp. 3070–3080, 2017. ¨ , “Human activity data discovery from tri- , vol. 7, no. 2, p. e11201, 2019. , “An effective deep autoencoder approach otz, “Ensembles of deep lstm learners for et al. , “Deep neural network based human activity , “Wearable sensor data based human activity , “Complex human activity recognition using Proceedings of the 3rd International Conference on et al. et al. et al. , “Wearable sensor-based activity recognition for , “Empirical study and improvement on deep , “Comparison of data preprocessing approaches , “A novel wearable sensor-based human activity , “Novel approaches to human activity recognition , “Human activity recognition using smartphones,” , “A comparative study on human activity recogni- , “Deep recurrent neural network for mobile human , “Robust human activity recognition using smart- et al. , “Deep transfer learning for cross-domain activity , “Gchar: An efficient group-based context—aware et al. , “Improving sensor-based activity recognition using , vol. 51, no. 2, pp. 204–214, 2015. et al. , “Applying multivariate segmentation methods to , “Innohar: a deep neural network for complex human et al. et al. . ACM, 2017, p. 14. . ACM, 2018, pp. 118–121. et al. et al. et al. , “Comparison of feature learning methods for human et al. et al. et al. et al. et al. et al. et al. , vol. 23, no. 2, pp. 173–185, 2018. , vol. 16, no. 7, p. 1033, 2016. et al. et al. , vol. 15, no. 5, pp. 2691–2699, 2018. et al. 2018 2nd International Symposium on Multidisciplinary Studies recognition for the order picking process,” in no. 2, p. 11, 2017. 4th international Workshop onInteraction Sensor-based Activity Recognition and phone sensors viaIndustrial ct-pca Informatics and online svm,” using smartphone sensors,” matics activity recognition,” on Interactive, Mobile, Wearable and Ubiquitous Technologies data using deep learning,” p. e125, 2016. Comput. Sci. Netw. Secur recognition approachSensors using artificial hydrocarbon networks,” activity recognition using wearable sensors,” activity recognition using wearables,” smartphone and wrist-worn motion sensors,” human activity recognition from wearable sensors’ data,” for applying deepcontext learning of to industry human 4.0,” activity recognition in the no. 4, p. 426, 2016. networks in humantelligence activity and recognition,” Humanized2018. Computing recognition,” in Crowd Science and Engineering preprint arXiv:1905.03809 mHealth and uHealth housekeeping task,” in Wearable and Implantable Body Sensorpp. Networks 67–70. (BSN) transfer learning for human activity recognition,” vol. 16, no. 11, pp. 4566–4578, 2016. for online smartphone-based human activity recognition,” no. 2, p. 679, 2018. based on accelerometer data,” vol. 12, no. 7, pp. 1387–1394, 2018. in and Innovative Technologies (ISMSIT) recognition using machine learning: A new approach,” no. 1, p. 57, 2019. for an upper body exercise classification system,” 2019. axial accelerometer sensor:to Non-supervised feature learning extraction sensitivity Management parametrization,” tion using inertial sensors in a smartphone,” human activity recognitionand Distributed on Computing smartphone,” activity recognitionRobotics with high throughput,” smartphones using lstm networks,” in vention on Informationand Microelectronics and (MIPRO) Communication Technology, Electronics 2018 ACMSymposium International on Joint PervasiveComputers Conference and and Ubiquitous 2018 Computing and International Wearable motion capture as additional information,” in [108] R. Grzeszick [110] ——, “A novel ensemble elm for human activity recognition [109] Z. Chen [111] C. Xu [125] F. Li [112] A. Sathyanarayana [113] M. Shoaib [124] H. Ponce [117] K. Li [106] X. Zheng [107] Y. Guan and T. Pl [114] J. Suto and S. Oniga, “Efficiency investigation of artificial neural [115] J. Wang [118] K.-C. Liu [119] R. Ding [123] B. Almaslukh [126] A. Jordao [127] E. Bulbul [116] H. Nguyen [130] M. Milenkoski [120] P. Nardi, “Human activity recognition: Deep learning[121] techniques I. P. Machado [122] A. Wang [128] L. Cao [129] M. Inoue [131] P. Lago Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 . , . . 19 Pattern 2007 3rd Journal of , vol. 12, 2015 IEEE 2015 22nd 2015 IEEE . IEEE, 2016, . IEEE, 2010, IEEE Transactions on 2010 Seventh interna- International workshop Medicine and science in , 2013. 2015 IEEE International Computational and mathe- Mobile Information Systems Proceedings of the 2012 ACM 2012 16th International Sym- , vol. 14, no. 2, pp. 436–446, Esann . ACM, 2012, pp. 1026–1035. . IEEE, 2015, pp. 144–149. 2016 IEEE-EMBS International . IEEE, 2015, pp. 473–478. Journal of Ambient Intelligence and Formative assessment, learning data . IEEE, 2012, pp. 108–109. , vol. 2016, 2016. . IEEE, 2015, pp. 582–587. . Elsevier, 2016, pp. 217–242. . Springer, 2014, pp. 91–98. , “Human activity recognition in aal en- IEEE journal of biomedical and health infor- , “Activity recognition using cell phone , “A smartwatch-based framework for real- , vol. 89, pp. 29–40, 2019. 2015 Seventh International Conference on Knowl- , “Improving classification of sit, stand, and , 2009. , “Smartwatch-based activity recognition: A et al. , “Activity recognition in youth using single , “Wearable assistant for parkinson’s disease ACM SigKDD Explorations Newsletter , vol. 49, no. 4, p. 801, 2017. et al. , “A public domain dataset for human activity et al. . Elsevier, 2014, pp. 85–104. , “Collecting complex activity datasets in highly , “Pervasive self-powered human activity recog- . IEEE, 2015, pp. 63–68. , “A benchmark dataset to evaluate sensor dis- , “Comparative study on classifying human ac- , “Activity recognition from on-body sensors by et al. , “Tse-cnn: A two-stage end-to-end cnn for human et al. , “mhealthdroid: a novel framework for agile devel- , “Distributed human action recognition via wear- ˇ cius et al. , “Smartphone-based human activity recognition in et al. et al. et al. , “Human activity detection based on multiple smart , vol. 43, no. 10, pp. 3605–3620, 2010. . IEEE, 2007, pp. 281–286. et al. et al. et al. et al. et al. et al. ˇ sevi et al. et al. et al. ˜ , 2019. nos Conference on Ubiquitous Computing accessed: 2019-11-16. able motion sensor networks,” Smart Environments mobile devices,” in edge and Systems Engineering (KSE) placement in activity recognition,” in vironments using random projections,” matical methods in medicine activity recognition,” matics Recognition machine learning approach,”Conference on in Biomedical andpp. Health Informatics 426–429. (BHI) buildings using locality-constrained linear coding,” in accelerometer placed atsports wrist and exercise or ankle,” nition without the accelerometer,” in recognition using smartphones.” in tivities with miniature inertial and magnetic sensors,” 19th InternationalWork Conference in on Design (CSCWD) Computer Supported Cooperative 10th Conference onIEEE, Industrial 2015, Electronics pp. 214–219. and Applications (ICIEA) recognition on smart mobile devices,” vol. 2016, 2016. lie in2015 a IEEE smartphone InternationalApplications Symposium (MeMeA) human Proceedings on activity Medical Measurements recognition and system,” in Conference on PervasiveIEEE, Computing 2015, and pp. Communications 79–86. (PerCom) time and online assessmentbiomedical and informatics mobility monitoring,” Wearable Sensors analytics and gamification no. 2, pp. 74–82, 2011. rich networked sensortional conference environments,” on in networkedpp. sensing 233–240. systems (INSS) for ubiquitous activityProceedings recognition of using theACM, 2012 wearable 2012, ACM sensors,” pp. Conference 1036–1043. in on Ubiquitous Computing classifier fusion: sensorinternational scalability conference and on robustness,”information in intelligent sensors, sensor networksdataset and for activityposium monitoring,” on in Wearable Computers patients with the freezingInformation of gait Technology symptom,” in Biomedicine 2009. opment of mobileon health ambient applications,” assisted in living phone sensors and machine learning algorithms,” in in virtual environments,” in accelerometers,” accelerometer-based human activityInternational recognition,” Conference in Mixed Designtems (MIXDES) of Integrated Circuits & Sys- [206] , “Active miles,” http://hamlyn.doc.ic.ac.uk/activemiles/activemiles.html, [207] A. Yang [208] C. Pham, “Mobirar: Real-time human activity recognition using [191] J. Huang [205] O. Ba [186] Q. Zhu [190] R. Dama [192] S. Khalifa [185] G. M. Weiss [187] J. Lee and J. Kim,[188] “Energy-efficientN. real-time A. human Capela activity [189] A. Mannini [193] M. Kheirkhahan [194] T. Tamura, “Wearable inertial sensors and their applications,” in [197] D. Roggen [198] D. Anguita [199] M. Zhang and A. A. Sawchuk, “Usc-had: a daily activity dataset [200] P. Zappi [201] A. Reiss and D. Stricker,[202] “IntroducingM. a Bachlin new benchmarked [203] O. Banos [204] K. Altun [195] M. Feidakis, “A review of emotion-aware systems for e-learning [196] J. R. Kwapisz [183] P. Lubina and M. Rudzki,[184] “ArtificialX. Yin neural networks in , IEEE . IEEE, . IEEE, . ACM, . ACM, , 2016, pp. 2015 IEEE Automation , vol. 13, no. 1, Journal of Electri- , vol. 75, no. 12, Signal Processing IEEE Transactions 2015 IEEE Confer- . IEEE, 2015, pp. 2016 Ieee International International Journal of 2017 39th Annual Interna- ICT4AgeingWell . IEEE, 2015, pp. 1–3. . IEEE, 2016, pp. 1–5. 2016 IEEE SENSORS , vol. 16, no. 9, pp. 3198–3207, 2015 IEEE International Conference 2016 19th international conference on , vol. 2015, p. 34, 2015. . Atlantis Press, 2016. 2016 IEEE International Conference on . IEEE, 2015, pp. 68–73. , vol. 10, no. 4, p. e0124414, 2015. . IEEE, 2016, pp. 371–378. , vol. 3, no. 6, pp. 1124–1134, 2016. Proceedings of the 2015 ACM International Proceedings of the 2015 ACM International , vol. 29, no. 13, pp. 1981–1991, 2016. , “Feature extraction from smartphone iner- , “Activity recognition based on accelerome- , vol. 17, no. 6, pp. 1353–1368, 2017. , “Feature selection for wearable smartphone- , “The mobiact dataset: Recognition of activities PloS one , “Evaluation of a smartphone human activ- , “Cnn based approach for activity recognition , “Human activity recognition using wearable et al. Multimedia tools and applications , “Human activity recognition with hmm-dnn , “Recognizing human activities from raw ac- et al. IEEE Sensors Journal . IEEE, 2017, pp. 2438–2441. , “Harke: Human activity recognition from kinetic , “A triaxial accelerometer-based human activity , vol. 71, pp. 198–209, 2016. et al. , “Improved activity recognition by using enriched , “Activity recognition based on inertial sensors for , “Human activity recognition with inertial sensors , “On motion-sensor behavior analysis for human- , “Lstm networks for mobile human activity recog- , “Beyond activity recognition: skill assessment from , “Human activity recognition based on transformed et al. , “Multicolumn bidirectional long short-term mem- et al. 2015 IEEE 14th International Conference on Cognitive et al. et al. 2016 International Conference on Artificial Intelligence: et al. et al. IEEE Transactions on Systems, Man, and Cybernetics: et al. . IEEE, 2015, pp. 1–4. et al. et al. et al. et al. . IEEE, 2015, pp. 865–870. et al. et al. et al. et al. Journal of neuroengineering and rehabilitation et al. , vol. 47, no. 12, pp. 3142–3149, 2016. 2015 IEEE 6th Latin American Symposium on Circuits & Systems 2016, pp. 1–6. nition,” in Technologies and Applications ter sensor usingence combinational on classifiers,” Open in Systems (Icos) tional Conference ofSociety the (EMBC) IEEE Engineering in Medicine and Biology of daily living using smartphones.” in using a wrist-worn accelerometer,” in model,” in ambient assisted living,” in Joint Conference on2015, Pervasive pp. 1155–1166. and Ubiquitous Computing feature selection and classificationcal framework,” and Computer Engineering tion workers’ activity recognitionin and Construction classification,” acceleration data,” in 2015, pp. 1011–1015. activity recognition viaConference smartphones,” on Identity, in Security and Behavior Analysis (Isba) 143–151. vol. 120, pp. 359–372, 2016. Informatics & Cognitive192–197. Computing (ICCI* CC) information fusion (fusion) accelerometer data,” in ory for mobile devices-based human activity recognition,” Joint Conference on Pervasive and Ubiquitous Computing tial signals for human activity segmentation,” (LASCAS) Internet of Things Journal accelerometer sensors,” in Consumer Electronics-Asia (ICCE-Asia) based human activity recognition usingin a tri-axial accelerometer,” from smartphone accelerometer datasensing,” for user contextSystems awareness p. 5, 2016. celerometer data14th using International deep Conference on(ICMLA) neural Machine networks,” Learning and in Applications accelerometer data from aCommunication mobile Systems phone,” based human activity recognitionstroke with patients,” able bodied, elderly, and ity recognition applicationpants,” with able-bodied and stroke partici- using a deep learning approach,”2016, in pp. 1–3. energy harvesting dataon in Mobile Computing wearable devices,” using quasiperiodic timeaccelerometer,” series collected frompp. 7257–7270, a 2016. single tri-axial recognition via eemd-based featuresture and selection,” game-theory-based fea- on Engineering and Technology (Icetech) 2016. recognition on smartphone,” in [177] M. S. Zainudin [176] Y. Chen [178] G. Vavoulas [181] L. Zhang [171] Y. Zheng, “Human activity recognition based on the hierarchical [172] D. Tao [173] R. Akhavian and A. H. Behzadan, “Smartphone-based construc- [174] I. Suarez [175] C. Shen [179] R. San-Segundo [180] M. Panwar [182] K. Davis [170] A. Khan [169] C. Torres-Huitzil and M. Nuno-Maganda, “Robust smartphone- [158] J. Wannenburg and R. Malekian, “Physical activity recognition [166] L. Zhang [167] M. Zubair [168] X. Heng [159] N. A. Capela [160] T. Zebin [161] S. Khalifa [162] A. D. Ignatov and V. V. Strijov, “Human[163] activityZ. recognition Wang [165] N. Capela [164] P. Paul and T. George, “An effective approach for human activity Accepted for Publication in IEEE Access DOI: 10.1109/ACCESS.2020.3037715 20 received her PhD in computer MD MS FASN FCCM is a R. Parisa Rashidi science in 2011learning. with She is an currently emphasisat an associate on the professor machine J.Biomedical Engineering Crayton (BME) PruittFlorida at (UF). Family She University Department istrical of also & of affiliated Computer with EngineeringComputer the (ECE), & as Elec- well Information as Science(CISE) & departments. Engineering She“Intelligent is Health theaims Lab” director to of (i-Heal). bridge the theing Her gap and between research machine patient learn- care. She has served on the Azra Bihorac, Glenn DavisAnesthesiology Professor and of PhysiologyGenomics and Medicine, at Functional Surgery, Precision University of andnership Florida. Intelligence (PrismaP), She agroup in leads of multidisciplinary experts research in Medicine datafocused science on Part- the and development informatics, andof implementation intelligent systemsment and clinical technologies decisions and todelivery optimize aug- health in care surgery,nephrology. critical The care team medicine is and developing machine technical program committeereviewer of of numerous several IEEE conferences journals. and has been a learning and informatics tooltion of for hospital-acquired real-time complications risk andthe stratification kidney application disease and of as annota- omics wellof technologies as patients for on with urine critical illness. forhuman-centered Her predictive health vision enrichment is care topatient’s develop that “personal tools for delivers clinical intelligent optimized profile”in using care national digital tailored and data.and to international Through critical a her professional careand work organizations medicine, scientists, in she promotingleadership, nephrology their has research equality advocated andUniversity for and of recognition women Sarajevo, education. Bosnia in physicians dency and She health at Herzegovina, care Marmara internal completed University, medicinefellowships Istanbul, her resi- in Turkey and critical University MD care ofClinical medicine Florida, at Science and at nephrology the University and of Florida. her Masters in , , . IEEE, Sensors , vol. 17, ICC 2019- 2019 IEEE Sensors IEEE journal of , vol. 19, no. 7, p. , 2018, pp. 64–67. . IEEE, 2018, pp. Sensors PhD in computer science, JMIR mHealth and uHealth PhD in computer science, 2018 IEEE 23rd International , vol. 17, no. 11, p. 2556, 2017. . IEEE, 2019, pp. 1–5. Proceedings of the 2018 ACM , vol. 23, no. 4, pp. 1585–1594, 2018. Sensors , 2016, pp. 267–272. , “Combining smartphone and smartwatch , “A comprehensive analysis on wearable ac- , “Accelerometer-based human fall detection , “Wearable sensors for recognizing individuals , “Accuracy of samsung gear s smartwatch for Graziano Pravadelli, IEEE senior member, IFIPfull 10.5 professor WG of member, informationat is processing the Computer systems Science Departmentversity of of the Verona Uni- (Italy)cofounded since EDALab 2018. s.r.l.,the In design an 2007 of he IoT-based SME monitoringmain systems. working interests His focus on simulation on and system-level modeling, semi-formalbedded systems, verification as well ofto as develop em- on IoT-based virtual theirfor coaching application people platforms with special needs. In the previous Florenc Demrozi, IEEE member, receivedgrees the B.S. in andfrom Computer the M.E. University de- Science ofin 2014 Verona, and and Italy, respectively 2016, Engineering andputer the Science Ph.D. from degree University in2020. of Com- He Verona, Italy, is in and currently a Temporary Professor Postdoctoral researcher atComputer the Science, Department Universitywhere of he of is member Verona, oftems the Italy, Design) ESD Research (Electronic Group, Sys- bient working Intelligence (AmI), on Ambient Am- Assisted Living et al. et al. SEKE , “Deep ensemble learning for human activity , “Perrnn: Personalized recurrent neural networks et al. et al. et al. et al. et al. undertaking daily activities,” in International Symposium on Wearable Computers tivity recognition usingbiomedical and health wrist informatics accelerometers,” sensor data inevaluation.” activity in recognition approaches: an experimental activity recognition: Validation study,” vol. 7, no. 2, p. e11270, 2019. human activity recognition,” 2nd Wireless Africa Conference (WAC) for acceleration-based human activity recognition,” in 1644, 2019. recognition using smartphone,”Conference in on Digital1–5. Signal Processing (DSP) using lstm-rnn deep neural network architecture,” in 2019 IEEE International Conference on2019, Communications (ICC) pp. 1–6. augmentation for humanable activity imu sensor classification data using based a on deep lstm wear- neural network,” using convolutional neural networks,” vol. 18, no. 9, p. 2892, 2018. celeration sensors in human activity recognition,” no. 3, p. 529, 2017. contexts, he collaborated in severalhe national published and European more projectsjournals. and than 120 papers in international conferences and (AAL) and Internet of Things (IoT). [209] S. A. Elkader [210] F. B. A. Ramos [211] A. Mannini and S. S. Intille,[212] “Classifier personalizationA. Davoudi for ac- [213] A. Murad and J.-Y. Pyun, “Deep recurrent neural networks for [217] X. Wang [214] G. L. Santos [215] R. Zhu [216] S. W. Pienaar and R. Malekian, “Human activity recognition [218] O. Steven Eyobu and D. Han, “Feature representation and data [219] M. Janidarmian