Analysis and Prediction of Emotions Using Human-Robot and Driver-Vehicle Interactions
Total Page:16
File Type:pdf, Size:1020Kb
i Analysis and Prediction of Emotions using Human-Robot and Driver-Vehicle Interactions A dissertation presented By Fatemeh Gandomi to The Department of Mechanical and Industrial Engineering In partial fulfillment of the requirements for the degree of Doctor of Philosophy (PhD) in the field of Industrial Engineering Northeastern University Boston, Massachusetts (April 2018) ii Dedication I would like to dedicate this dissertation to my parents and my family for their unconditional supports throughout these years. iii Abstract This dissertation undertakes the study of human factors in two interconnected fields, the human-robot and driver-vehicle interactions. Our societal systems today contain integrated human and machine entities. When they work together as integrated units, they can address a wide range of problems that are too complex to be addressed by individual machines working separately. The design and implementation of most modern functional systems are to place primary emphasis on technological innovations without much consideration for social components. Today’s humanity is encircled by so many instrumentations and electronics, so it would be really difficult to distinguish as to where the end of the tools and the starting point of humans are. The future of human-machine interaction needs to have a human centered approach to components of human-machine interaction. The most innovative way of solving such complicated problem from engineering and entrepreneurship perspectives is to find the problems that most people have and then creative ways to use technology to solve them. In other words “finding problems worth solving” is a design strategy that human-machine interaction uses in many domains of research. Recent research shows that human-robot interactions can have wide range of applications in the field of therapeutic interventions for children with developmental disorders, such as autism spectrum disorders (ASD). Since these children are in constant communications with humans, as well as showing reactions naturally, they are incapable of stimulating factors. Using a humanoid robot as an intermediary agent, for example, can significantly improve their rehabilitation and psychological treatments. Due to appealing design and configuration of humanoid robots in their appearance, motion and iv sound, they can draw patient’s attention more efficiently. This, in turn, increases aspiration of children to participate in therapeutic activities more effectively. Along this line of reasoning, this research concerns the use of humanoid robots with various communication capabilities, such as speaking and executing hand and foot in interaction with humans. For this, an experimental setup was proposed and designed to recognize human behavior in Gaze-communications with a robot utilizing joint attention and eye contact. The experimental results demonstrated that participant’s favorable feeling for the robot was more than other objects in the experiment. Moreover, the feeling is enhanced when robot started to talk or initiate a social interaction with human. Hence, such robot-human interactions have the capability to make improved joint attention for therapeutic interventions. The second part of this dissertation undertakes the driver-vehicle interactions and discusses ways this interaction can be used to influence driver in a positive way. Driver with anger state cannot influence perceived workload. Moreover, induced anger can reduce driver situation awareness. Previous research studies show that listening to music while driving is a very effective way to change driver’s mood. The Limbic system in human brain is so powerful that emotions can change how human thinks. Music can aid in the production of serotonin, which can make people happy. In this part of the dissertation, our primary objective was to assess the effect of immediate change in music tempo on driver when they get angry via internal or external/environmental causes such as personal issue, traffic, and aggressive drivers. For physiological states measurement, heart rate, skin conductance and electromyography techniques were utilized to recognize driver’s states as well as the effect of music type on driver’s mood while driving. The v results show that music has the capability to influence physiological signs and potential to reduce the impact of negative emotions on human while driving in angry mood. More specifically, the study has demonstrated that tempo music can mediate the effects of anger in a simulated environment and have potential to manage mood states. In summary, this research provides some experimental work to demonstrate the importance of human emotion and effects on the interaction of human-machine systems and system performance. vi Acknowledgments I would like to sincerely thank my PhD research advisor, Prof. Yingzi Lin, for her supports, dedication and guidance during my graduate studies over the past. I would also like to thank my advisory committee members for their supports and feedbacks. I am also thankful to my groupmates, lab mates and friends who have been helping me throughout this important journey. vii Table of Contents Topics Page No. Abstract …………………….………………………………………………………..… iii Acknowledgements …………………………………………………………...……….. vi Table of Contents ………………………………………………………………...…… vii List of Figures ………………………………………...………………………………. ix List of Tables ……………………………..………………………………………..… xii Part I: Human-Machine Interaction with Application to Social Robots ……….…. 1 Chapter 1: Background and Motivations …………………...………………………….. 2 1.1. Introduction and Overview ……………………………………...…………… 2 1.2. Problem Statement …………………………………………………………… 5 1.3. Research Questions ………………………….……………………………….. 6 1.4. Literature Review …………………………………………………………….. 6 1.5. Autism and It Social Implications ……………………………………………. 9 1.6. Social Robots ……………………….........…………….…………….……… 12 1.7. Needs and Facilitation ………………………………………………………. 14 1.8. Summary and Outlook ……………………………………………………… 15 Chapter 2: Human-Machine Interaction using Social Robots …………………….….. 17 2.1. Introduction and Problem Statement ……………………………………….. 17 2.2. Social Robots in Stable Interaction in Autism Therapy ……………………. 18 2.2.1. Millo ………………………………………………………………... 18 2.2.2. Kasper ………………………………………………………………. 19 2.2.3. NAO ………………………………………………………………… 20 2.2.4. Leka ………………………………………………………………… 21 2.2.5. Other Social Robots ………………………………………………… 22 2.3. EZ-Robot as A Social Robot ……………………………………………….. 23 2.3.1. EZ-Robot Specifications …………………………………………… 23 2.3.2. EZ-Robot Capabilities ……………………………………………… 24 2.3.3. MYO Armband Attachment ……………………………………….. 28 Chapter 3: Human Behavior in Gaze Interaction with EZ-Robot …………………… 30 3.1. Introduction and Problem Statement ……………………………………….. 30 3.2. Stimuli and Apparatus ……………………………………………………… 30 3.3. Eye Tracking Process ……………………………………………………….. 31 3.4. Participants …………………………………………………………………. 33 3.5. Experimental Procedure and Methodology …………………..…………….. 34 3.6. Results and Data Analysis ………………………………………………….. 37 viii 3.7. Summary and Conclusions …………………………………………………. 44 Chapter 4: EZ-Robots as Social Mediators …………………………………………… 46 4.1. Introduction and Problem Statement ……………………………………….. 46 4.2. Robots in the Aurora Research Project ……………………………….…….. 48 4.3. Imitation Game Therapy for Autism ………………………………….…….. 50 4.4. Eye Interaction Therapy for Autism …………………….………………….. 51 4.5. Emotion Recognition Therapy for Autism …………………………………. 52 4.6. Dance Therapy for Autism …………………………………………….……. 53 Part I Conclusions …………………………………………………………………….. 55 Part II: Human-Machine Interaction with Application to Driver Distraction Reduction …………………………………………………………………………….. 56 Chapter 5: Background and Motivations ………………………………..……………. 57 5.1. Introduction and Overview …………………………………………………. 58 5.2. Literature Review …………………………………………………………… 60 5.3. Problem Statement and Motivation ………………………………………… 63 Chapter 6: Introduction to Physiological States and Human Emotions ……………… 65 6.1 Music Effects on Human State of Brain …………………………….……… 65 6.2 Physiological States of Anger ………………………………………..……... 67 6.2.1 Anger and Heart Rate ………………………………………..……… 68 6.2.2 Anger and Skin Conductance ……………….……………………… 69 6.2.3 Anger and Electromyography (EMG) ………………………….…... 70 6.3 Anger and Road Rage ……………………………………………………… 72 Chapter 7: Effects of Music on Driver’s Anger Mood ………………………………. 74 7.1. Problem Statement and Research Hypotheses …………………….…….….. 74 7.2. Technical Approach and Methods ………………………………..………… 74 7.3. Music Selection ……………………………………………………...……… 75 7.4. Driving Simulation …………………………………………………..….…... 77 7.5. Physiological Measurements ……………………………………….…….… 78 7.5.1. Facial Electromyography (EMG) Measurement ……………………. 79 7.5.2. Skin Conductance (SC) Measurement …………………………….... 80 7.5.3. Heart Rate (HR) Measurement ……………………………………... 81 7.6. Experimental Setup and Procedure ……………………………………...….. 81 7.7. Results and Analysis of Data …………………………………………..…… 83 7.8. Summary and Discussions …………………………………………..……… 89 Part II Conclusions …………………………… ……………….………………….…. 91 References Cited ………………………...……………………………………………. 92 ix List of Figures Figure 1.1 Riba robot as a robotic nurse bear, [29]. Figure 1.2 Paro robot interacting with elderly people, [30]. Figure 1.3 Romeo robot proving companionship and service, [31]. Figure 2.1 Specifications of Milo robot. Figure 2.2 Milo robot interacting with Autism children. Figure 2.3 Kasper robot. Figure 2.4 Nao robot interacting with an autistic child. Figure 2.5 Leka robot. Figure 2.6 Representative