Responsive IoT: Using Biosignals to Connect Humans and Smart Devices by Alexandre Armengol Urpi Submitted to the Department of Mechanical Engineering in partial fulfillment of the requirements for the degree of Master of Science in Mechanical Engineering at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2018 @ Massachusetts Institute of Technology 2018. All rights reserved. Signature redacted A uthor ..... .................... Department of Mechanical Engineering 141 May 21, 2018 Signature redacted Certified by ................. Sanjay E Sarma Professor Thesis Supervisor Signature redacted A ccepted by .......................... Rohan Abeyaratne Chairman, Department Committee on Graduate Theses MASSACHUSETT INSTITUTE OF TECHNOLOGY JUN 2 5 2018 LIBRARIES ARCHIVES 2 Responsive IoT: Using Biosignals to Connect Humans and Smart Devices by Alexandre Armengol Urpi Submitted to the Department of Mechanical Engineering on May 21, 2018, in partial fulfillment of the requirements for the degree of Master of Science in Mechanical Engineering Abstract The growing Internet of Things (JoT) ecosystem being built today is already af- fecting a great many daily objects, which may share information about their state, location and sensed data among others. Today, humans communicate with IoT de- vices through visual, voice or tactile interfaces. More natural and organic interaction requires more sophisticated communication methods. This thesis explores seamless interfaces between human and IoT devices. In particular, I focus on using biological signals as the interface to directly connect with the smart surroundings. The work is partitioned in two parts. First, I present a wearable sensing system to estimate the thermal comfort level of the user by monitoring skin temperature, blood volume pressure and skin conductivity. This effort is a first step towards connecting room occupants and smart A/C devices, which can enable real-time adjustments of indoor conditions. In the second part of the thesis, brain signals are used as the interface to navigate in a Virtual Reality (VR) environment. We develop Sublime, a new concept of Steady-State Visually Evoked Potentials (SSVEP) based Brain-Computer Interface (BCI). In this technology, brain-computer communication is triggered by imperceptible visual stimuli integrated in the virtual scene and subliminal informa- tion is seamlessly'conveyed to a computer. By monitoring the elicited SSVEPs, the system is able to identify the gaze target of the user, thus enabling a hands-free menu navigation tool. Thesis Supervisor: Sanjay E Sarma Title: Professor 3 4 Acknowledgments First of all, I wish to express my sincere thanks to Professor Sanjay Sarma for his valuable directions and continuous encouragement. He has provided me total freedom and flexibility to satisfy my interests both inside and outside of school, which is an invaluable gift. Research is full of ups and downs, and I have been truly impressed by how he is able to transform frustration into excitement and motivation with just a 5-minute conversation. But above all, he is an excellent person who has always been open to talk both professionally and personally. I also want to thank Dr. Stephen Ho for his guidance and support during these two years at the lab. His critical advice has been very valuable in my first two years as a graduate student. A special mention should also go to all Sanjay's lab members: Nithin, Debbie, Yongbin, Pranay, Pankhuri, Nidhi, Dajiang, Shane, Felipe, Esteve, Alex, Josh, Rahul, Brian and Laura. They provided everyday help and support but, most importantly, they allowed me to be part of this great working environment which made me feel so welcome since the first day. These two years here at MIT wouldn't have been the same without the prince's, tsaprs, swing-dancers, dumpling-makers, phantastics, catalans and spaniards; all bringing meaning to my everyday life. Finally I want to thank my mom, my dad and my sister for always staying by my side, wherever I am and wherever they are. 5 6 Contents 1 Introduction 15 1.1 M otivation ...................... .......... 15 1.2 Empathic buildings: sensing thermal comfort ...... ....... 16 1.3 Brain-Computer Interfaces ......... ........... ... 17 2 Background 19 2.1 Thermal Comfort Sensing ............... ...... 19 2.1.1 PMV - PPD Model . ......... ..... ........ 19 2.1.2 Adaptive Models ..... ........... ........ 23 2.1.3 Personalized Thermal Comfort Sensing .... ........ 23 2.2 SSVEP-based Brain-Computer Interfaces ..... .. ........ 25 3 A Wearable-based Thermal Comfort Sensing System 27 3.1 Potential Biosignals for Thermal Comfort Estimation . ........ 27 3.1.1 Wrist Skin Temperature ........ .... ........ 27 3.1.2 Vasomotion ......... ........ .. ........ 27 3.1.3 Electrodermal Activity ......... .... ... ... 28 3.2 Wearable: Biosignal Monitoring Wristband .... .. ........ 30 3.2.1 Sensors .... ...... ..... ...... ........ 30 3.2.2 Functioning Modes ..... ........ .. ........ 31 3.3 First Experiments . ......... .......... ... ... 32 3.3.1 R esults ....... ........ ....... ... ... 33 3.4 Machine Learning Approach . ......... .... .. .. .. .. 39 7 3.4.1 Feature Selection .... ...... ... .. .. .. 39 3.4.2 Time Windows ............. ... .. 40 3.4.3 Datapoint Extraction .. ... .... .. ... .. .. 41 3.4.4 Single-User Model. .. .... .... .. .... ... 42 3.4.5 Multi-User Model ............. .. ..... 44 4 A Brain-Computer Interface for Virtual Reality 49 4.1 Sublim e ................... ....... ....... 4 9 4.2 Materials and Methods ........... .. .... ..... ... 50 4.2.1 Virtual Reality Display Device . .. ..... .... .... 5 0 4.2.2 Visual Stimuli Generation . .... ....... ....... 5 0 4.2.3 Beating effect ............ .. .... ..... ... 5 1 4.3 Virtual Reality Application ........ ......... .... 53 4.3.1 Main Menu ............. .... ...... .... 53 4.3.2 Movie Playback Menu ....... .. .... ..... ... 53 4.3.3 Real-time Feedback ......... ......... ..... 54 4.4 EEG Recording Equipment ........ ... ......... .. 5 5 4.5 SSVEP Detection .............. ......... ..... 55 4.5.1 Canonical Correlation Analysis . ..... ......... 5 5 4.5.2 Logistic Regression ......... ......... .... 56 4.6 Real-time Data Processing ......... ... ......... .. 5 7 4.7 System Configuration ............ ..... .... ..... 58 4.8 Experiments ................. ... .... ..... .. 58 4.8.1 Subjects ............... ......... .... 58 4.8.2 Experiment 1: Navigation Time . ..... ......... 5 9 4.8.3 Experiment 2: Subjective Experience . ......... .... 59 4.9 R esults .................... ....... ....... 6 0 4.10 Discussion .................. .............. 6 1 5 Conclusions and Future Work 63 5.1 Sensing Thermal Comfort . 63 8 5.2 Brain Waves in Virtual Reality ...... ............. 64 5.3 Future Work ......... ........ ............. 65 5.3.1 Improvements in the comfort models ...... ....... 65 5.3.2 Future work on Sublime ...... ............. 65 9 10 List of Figures 2-1 Predicted Percentage of Dissatisfied versus Predicted Mean Vote. 21 2-2 Body thermoregulation diagram. Extracted from [20] ........ 24 3-1 PPG signal. Extracted from [461. Notice that vasomotion information can be obtained from the signal amplitude. ...... ...... .. 28 3-2 Typical shape of a phasic skin conductance. ...... ........ 29 3-3 Empatica E4 Wristband .... ...... ...... ...... ... 31 3-4 E4 recording mode ....... ...... ...... ...... ... 31 3-5 E4 recording mode ..... ....... ...... ...... .... 32 3-6 Plot of data obtained during one of the experiments. .. ..... .. 33 3-7 Gradient of skin temperature vs. thermal sensation. ....... ... 34 3-8 PPG signal and its envelope during 6 seconds of data approximately. The blue line represents the PPG signal measured by the wristband. The yellow and orange lines define the envelope of the PPG signal computed in M atlab. ... ..... ..... .... ..... ..... 35 3-9 PPG signal and its envelope of 10 minutes worth of data. .... ... 36 3-10 Plot of skin temperature and vasomotion signal obtained from the PPG sign al. .... ..... .... ..... ..... .... ..... ... 36 3-11 Vasomotion signal during the experiment explained above. Orange and green lines are thermal sensation and room type respectively. ..... 37 3-12 Plot of EDA signal (blue). Vertical cyan lines mark the EDA peaks detected........ ................................... 38 11 3-13 Plot of EDA signal (blue) during the experiment. Vertical cyan lines mark the EDA peaks detected .................. .... 38 3-14 Relative frequency of EDA peaks vs. thermal sensation. ..... ... 39 3-15 Mean IBI vs. temperature gradient point scatter for the three different classes. It shows how the distinct class datapoints cluster differently. 43 3-16 Mean vaso vs. std vaso point scatter for the three different classes. Va- sodilation and vasoconstriction in hot and cold states is clearly reflected. 43 4-1 Blue line shows a 44Hz stimulating sine wave sampled at RR = 1/90Hz. The red dashed line shows the beating effect that will be perceived. 52 4-2 Screenshot of the movie covers the user sees in the main menu. .... 53 4-3 Star Wars Episode VIII playing in the movie menu. .......... 54 4-4 Loading bars for two different selectable objects. Figure 4-4b shows the flickering object that allows the user to return to the main menu. 54 4-5 System blocks configuration. ... .................... 58 12 List of Tables 2.1 Predicted Mean Vote scale ........................ 20 4.1 Results for ExperimentI. ..... ........... ........ 60 4.2 Results for Experiment2. ......... ............ ... 60 13 14 Chapter 1