Sensor Fusion in Human Activity Recognition and Occupancy Detection Sensorfusion in der menschlichen Aktivit¨ats- erkennung und Pr¨asenzdetektion Der Technischen Fakult¨at der Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg zur Erlangung des Doktorgrades Dr.-Ing. vorgelegt von Lars Zimmermann aus Freudenberg Als Dissertation genehmigt von der Technischen Fakult¨at der Friedrich-Alexander-Universit¨atErlangen-N¨urnberg Tag der m¨undlichen Pr¨ufung:5. M¨arz2020 Vorsitzender des Promotionsorgans: Prof. Dr.-Ing. habil. Andreas Paul Fr¨oba Gutachter: Prof. Dr.-Ing. Georg Fischer Prof. Dr. sc. techn. Leonhard Reindl Abstract This thesis investigates the recognition of human activities from the position of the wrist and the feasibility of occupancy detection based on environmen- tal sensors. In preparation to this thesis, a wrist-worn 10-axis human activity tracker combining a 3-axis accelerometer, a 3-axis gyroscope, a 3-axis mag- netometer, and a barometric pressure sensor was developed. Four volunteers participated in collecting a first dataset of 190 measurements by perform- ing ten everyday activities. An approach is introduced by which a person's energy expenditure can be estimated independently of the measurement po- sition. Using the acceleration data obtained at the wrist, various methods for detecting cycling are presented. It is also shown that the barometer allows discriminating activities that involve a vertical displacement, such as climb- ing stairs or riding an elevator, as well as helps to reduce the false alarm rate of a fall detector. The potential of magnetic field sensing is derived from the signal characteristics of the magnetic interference as experienced during cycling and riding an elevator. It is demonstrated that the examined activity set can be discriminated by fusing the data of either the accelerometer and the barometer or the gyroscope and the barometer. To detect and count occupants, an approach using a fusion of environ- mental sensors from an indoor air quality measurement system is presented. Environmental sensors, as opposed to motion detectors, are nonintrusive, easy to install, low cost, detect nonmoving occupants, do not have dead spots, and can even infer the number of occupants. For this study, measure- ments of carbon dioxide, volatile organic compounds, air temperature, and relative air humidity were conducted in four student apartments for a total of 49 days. Features were extracted from the environmental sensor data and subsets selected using correlation-based feature selection. In this thesis, a comparison of the supervised learning models RIPPER, Na¨ıve Bayes, C4.5 decision tree, logistic regression, k-nearest neighbours, and random forest is performed. Furthermore, a method is proposed to greatly reduce time and effort of collecting training data in residential buildings. The results indicate that the predictive power of volatile organic compound sensing is comparable to that of carbon dioxide. With a simple Na¨ıve Bayes classifier, the approach detected occupancy and estimated the number of occupants with an accu- racy of 81.1 % and 64.7 %, respectively. In addition, a gas sensor test setup was developed that is capable of characterizing carbon dioxide and volatile organic compound sensors simultaneously. Zusammenfassung Diese Arbeit untersucht die Erkennung menschlicher Aktivit¨atenausgehend von der Position des Handgelenks sowie die Machbarkeit einer Pr¨asenzdetek- tion basierend auf Umweltsensoren. Als Grundlage diente ein 10-Achsen Ak- tivit¨atstracker bestehend aus einem 3-Achsen Beschleunigungssensor, einem 3-Achsen Gyroskop, einem 3-Achsen Magnetometer, und einem baromet- rischen Drucksensor. Vier Freiwillige haben sich durch Ausf¨uhrungzehn allt¨aglicher Aktivit¨atenan der Aufzeichnung eines ersten Datensatzes mit 190 Messungen beteiligt. Zur Sch¨atzung des Energieaufwands einer Per- son wird ein Ansatz vorgestellt, der auf eine beliebige Messposition an- wendbar ist. Unter Nutzung der am Handgelenk gewonnenen Beschleu- nigungsdaten werden verschiedene Methoden zur Erkennung von Fahrrad- fahren pr¨asentiert. Außerdem wird gezeigt, dass das Barometer die Unter- scheidung von Aktivit¨atenwie Treppensteigen oder Aufzugfahren erm¨oglicht sowie die Fehlalarmrate einer Sturzerkennung reduzieren kann. Das Potenzial der Magnetfeldmessung wird aus den Signalcharakteristika der magnetischen St¨orfelder,wie sie bei Fahrrad- oder Aufzugfahren auftreten, abgeleitet. Es wird demonstriert, dass die hier untersuchten Aktivit¨atenmittels einer Fu- sion aus entweder dem Beschleunigungssensor und dem Barometer oder dem Gyroskop und dem Barometer unterschieden werden k¨onnen. Zur Detektion und Bestimmung der Personenanzahl im Raum wird ein Ansatz basierend auf der Fusion von Umweltsensoren eines Raumluftquali- t¨atsmesssystemspr¨asentiert. Im Gegensatz zu Bewegungsmeldern sind Um- weltsensoren unaufdringlich, leicht zu installieren, kosteng¨unstig,erkennen ruhende Personen, weisen keine Abdeckungsl¨ucken auf, und erlauben Schluss- folgerungen auf die Personenanzahl. F¨urdiese Untersuchung wurden in vier Studentenapartments Messungen ¨uber den Gehalt an Kohlendioxid, fl¨uchti- gen organischen Verbindungen, der Lufttemperatur, und der relativen Luft- feuchte mit einer Gesamtdauer von 49 Tagen durchgef¨uhrt. In dieser Arbeit werden die Algorithmen RIPPER, Naiver Bayes, C4.5 Entscheidungsbaum, Logistische Regression, N¨achste-k-Nachbarn, und Random Forest verglichen. Weiterhin wird eine Methode vorgeschlagen, die den Aufwand an Trainings- daten in Wohngeb¨auden deutlich reduziert. Die Ergebnisse zeigen, dass Sen- soren f¨urfl¨uchtige organische Verbindungen eine vergleichbare Prognosef¨a- higkeit wie Kohlendioxidsensoren aufweisen. Mit einem Naiver Bayes Klassi- fikator wurde eine Genauigkeit von 81,1 % bei der Pr¨asenzund von 64,7 % bei der Personenanzahl erzielt. Ferner wurde ein Pr¨ufaufbauentwickelt, mit dem verschiedenartige Gassensoren gleichzeitig charakterisiert werden k¨onnen. Acknowledgements First and foremost, I would like to thank my supervisor Prof. Dr.-Ing. Georg Fischer for the extraordinary support of my doctorate and the trust placed in me over the years. It was his personal enthusiasm for innovative ideas that only led me to decide to pursue the doctorate. Above all, Prof. Fischer has always given me the necessary freedom to explore and set my own directions in research. Next, I would like to express my thanks to Prof. Dr. Leonhard Reindl for accepting the second review of this thesis. Prof. Reindl is the chair holder of the Laboratory for Electrical Instrumentation at the Department for Mi- crosystems Engineering (IMTEK) and director of the Centre for Renewable Energy (ZEE) of the University of Freiburg. I would like to thank Georg Schmidt for the favourable general conditions at eesy-id, without which my research activities would have been difficult to carry out in the first place. To my former colleagues at eesy-id, in partic- ular Dr.-Ing. Lukas Reuter and Alexander Schmidler, thank you for your professional support during hardware development. Special thanks go to all members of the Institute for Electronics Engi- neering. The chair holder Prof. Dr.-Ing. Dr.-Ing. habil. Robert Weigel, who already supervised me in my diploma thesis, has always taken time for my matters. Furthermore, I would like to thank the whole team of the EjHome-Center for the pleasant time and great cooperation. I thank my former students Jens Pfeiffer, Christian Hesse, Jan Geret, Wolfgang R¨odle,Philipp Pollinger, Kerstin Inkmann, Alexander Hofmann, and Renata de Azevedo Allemand Lopes for their contribution to my re- search. Finally, I would like to thank Thomas Rechenmacher and Daniel Wilhelm for the discussions and their valuable feedback on my written work. Contents List of Acronyms and Symbolsx 1 Introduction1 2 Related Work6 2.1 Fusion of Wearable Sensors in Human Activity Recognition..6 2.2 Fusion of Environmental Sensors in Occupancy Detection... 12 2.3 Contributions of This Thesis................... 20 3 Human Activity Recognition With a Wrist-Worn Activity Tracker 25 3.1 Assessment of Energy Expenditure............... 27 3.2 Wearable Sensors for Human Activity Tracking........ 28 3.2.1 Accelerometer....................... 28 3.2.2 Gyroscope......................... 32 3.2.3 Magnetometer....................... 35 3.2.4 Barometer......................... 39 3.2.5 Other Wearable Sensors................. 42 3.3 Human Activity Recognition System.............. 44 3.3.1 System Overview..................... 44 3.3.2 Hardware Setup...................... 44 3.3.3 Software Description................... 46 3.4 Data Collection.......................... 48 3.4.1 Activity Set........................ 48 3.4.2 Participants........................ 49 3.4.3 Dataset.......................... 50 3.5 Human Activity Recognition Algorithm............. 50 3.5.1 Data Preprocessing.................... 50 3.5.2 Feature Extraction.................... 51 3.5.3 Computational Complexity................ 57 vii viii Contents 3.6 Results............................... 57 3.6.1 Accelerometer-Based Human Activity Recognition and Its Limitations...................... 57 3.6.2 Gyroscope-Based Human Activity Recognition and Its Limitations........................ 62 3.6.3 Magnetometer-Based Human Activity Recognition and Its Limitations...................... 64 3.6.4 Barometer-Based Human Activity Recognition and Its Limitations........................ 68 3.6.5 Human Activity Recognition With Sensor Fusion... 70 4 Occupancy Detection