i
Analysis and Prediction of Emotions using Human-Robot and Driver-Vehicle Interactions
A dissertation presented By Fatemeh Gandomi
to The Department of Mechanical and Industrial Engineering
In partial fulfillment of the requirements for the degree of Doctor of Philosophy (PhD)
in the field of
Industrial Engineering
Northeastern University Boston, Massachusetts (April 2018) ii
Dedication
I would like to dedicate this dissertation to my parents and my family for their unconditional supports throughout these years.
iii
Abstract
This dissertation undertakes the study of human factors in two interconnected fields, the human-robot and driver-vehicle interactions. Our societal systems today contain integrated human and machine entities. When they work together as integrated units, they can address a wide range of problems that are too complex to be addressed by individual machines working separately. The design and implementation of most modern functional systems are to place primary emphasis on technological innovations without much consideration for social components. Today’s humanity is encircled by so many instrumentations and electronics, so it would be really difficult to distinguish as to where the end of the tools and the starting point of humans are. The future of human-machine interaction needs to have a human centered approach to components of human-machine interaction. The most innovative way of solving such complicated problem from engineering and entrepreneurship perspectives is to find the problems that most people have and then creative ways to use technology to solve them. In other words “finding problems worth solving” is a design strategy that human-machine interaction uses in many domains of research.
Recent research shows that human-robot interactions can have wide range of applications in the field of therapeutic interventions for children with developmental disorders, such as autism spectrum disorders (ASD). Since these children are in constant communications with humans, as well as showing reactions naturally, they are incapable of stimulating factors. Using a humanoid robot as an intermediary agent, for example, can significantly improve their rehabilitation and psychological treatments. Due to appealing design and configuration of humanoid robots in their appearance, motion and iv sound, they can draw patient’s attention more efficiently. This, in turn, increases aspiration of children to participate in therapeutic activities more effectively. Along this line of reasoning, this research concerns the use of humanoid robots with various communication capabilities, such as speaking and executing hand and foot in interaction with humans. For this, an experimental setup was proposed and designed to recognize human behavior in Gaze-communications with a robot utilizing joint attention and eye contact. The experimental results demonstrated that participant’s favorable feeling for the robot was more than other objects in the experiment. Moreover, the feeling is enhanced when robot started to talk or initiate a social interaction with human. Hence, such robot-human interactions have the capability to make improved joint attention for therapeutic interventions.
The second part of this dissertation undertakes the driver-vehicle interactions and discusses ways this interaction can be used to influence driver in a positive way. Driver with anger state cannot influence perceived workload. Moreover, induced anger can reduce driver situation awareness. Previous research studies show that listening to music while driving is a very effective way to change driver’s mood. The Limbic system in human brain is so powerful that emotions can change how human thinks. Music can aid in the production of serotonin, which can make people happy. In this part of the dissertation, our primary objective was to assess the effect of immediate change in music tempo on driver when they get angry via internal or external/environmental causes such as personal issue, traffic, and aggressive drivers. For physiological states measurement, heart rate, skin conductance and electromyography techniques were utilized to recognize driver’s states as well as the effect of music type on driver’s mood while driving. The v results show that music has the capability to influence physiological signs and potential to reduce the impact of negative emotions on human while driving in angry mood. More specifically, the study has demonstrated that tempo music can mediate the effects of anger in a simulated environment and have potential to manage mood states. In summary, this research provides some experimental work to demonstrate the importance of human emotion and effects on the interaction of human-machine systems and system performance.
vi
Acknowledgments
I would like to sincerely thank my PhD research advisor, Prof. Yingzi Lin, for her supports, dedication and guidance during my graduate studies over the past. I would also like to thank my advisory committee members for their supports and feedbacks. I am also thankful to my groupmates, lab mates and friends who have been helping me throughout this important journey.
vii
Table of Contents Topics Page No.
Abstract …………………….………………………………………………………..… iii Acknowledgements …………………………………………………………...……….. vi Table of Contents ………………………………………………………………...…… vii List of Figures ………………………………………...………………………………. ix List of Tables ……………………………..………………………………………..… xii
Part I: Human-Machine Interaction with Application to Social Robots ……….…. 1 Chapter 1: Background and Motivations …………………...………………………….. 2 1.1. Introduction and Overview ……………………………………...…………… 2 1.2. Problem Statement …………………………………………………………… 5 1.3. Research Questions ………………………….……………………………….. 6 1.4. Literature Review …………………………………………………………….. 6 1.5. Autism and It Social Implications ……………………………………………. 9 1.6. Social Robots ………………………...... …………….…………….……… 12 1.7. Needs and Facilitation ………………………………………………………. 14 1.8. Summary and Outlook ……………………………………………………… 15 Chapter 2: Human-Machine Interaction using Social Robots …………………….….. 17 2.1. Introduction and Problem Statement ……………………………………….. 17 2.2. Social Robots in Stable Interaction in Autism Therapy ……………………. 18 2.2.1. Millo ………………………………………………………………... 18 2.2.2. Kasper ………………………………………………………………. 19 2.2.3. NAO ………………………………………………………………… 20 2.2.4. Leka ………………………………………………………………… 21 2.2.5. Other Social Robots ………………………………………………… 22 2.3. EZ-Robot as A Social Robot ……………………………………………….. 23 2.3.1. EZ-Robot Specifications …………………………………………… 23 2.3.2. EZ-Robot Capabilities ……………………………………………… 24 2.3.3. MYO Armband Attachment ……………………………………….. 28 Chapter 3: Human Behavior in Gaze Interaction with EZ-Robot …………………… 30 3.1. Introduction and Problem Statement ……………………………………….. 30 3.2. Stimuli and Apparatus ……………………………………………………… 30 3.3. Eye Tracking Process ……………………………………………………….. 31 3.4. Participants …………………………………………………………………. 33 3.5. Experimental Procedure and Methodology …………………..…………….. 34 3.6. Results and Data Analysis ………………………………………………….. 37 viii
3.7. Summary and Conclusions …………………………………………………. 44 Chapter 4: EZ-Robots as Social Mediators …………………………………………… 46 4.1. Introduction and Problem Statement ……………………………………….. 46 4.2. Robots in the Aurora Research Project ……………………………….…….. 48 4.3. Imitation Game Therapy for Autism ………………………………….…….. 50 4.4. Eye Interaction Therapy for Autism …………………….………………….. 51 4.5. Emotion Recognition Therapy for Autism …………………………………. 52 4.6. Dance Therapy for Autism …………………………………………….……. 53 Part I Conclusions …………………………………………………………………….. 55 Part II: Human-Machine Interaction with Application to Driver Distraction Reduction …………………………………………………………………………….. 56 Chapter 5: Background and Motivations ………………………………..……………. 57 5.1. Introduction and Overview …………………………………………………. 58 5.2. Literature Review …………………………………………………………… 60 5.3. Problem Statement and Motivation ………………………………………… 63 Chapter 6: Introduction to Physiological States and Human Emotions ……………… 65 6.1 Music Effects on Human State of Brain …………………………….……… 65 6.2 Physiological States of Anger ………………………………………..……... 67 6.2.1 Anger and Heart Rate ………………………………………..……… 68 6.2.2 Anger and Skin Conductance ……………….……………………… 69 6.2.3 Anger and Electromyography (EMG) ………………………….…... 70 6.3 Anger and Road Rage ……………………………………………………… 72 Chapter 7: Effects of Music on Driver’s Anger Mood ………………………………. 74 7.1. Problem Statement and Research Hypotheses …………………….…….….. 74 7.2. Technical Approach and Methods ………………………………..………… 74 7.3. Music Selection ……………………………………………………...……… 75 7.4. Driving Simulation …………………………………………………..….…... 77 7.5. Physiological Measurements ……………………………………….…….… 78 7.5.1. Facial Electromyography (EMG) Measurement ……………………. 79 7.5.2. Skin Conductance (SC) Measurement …………………………….... 80 7.5.3. Heart Rate (HR) Measurement ……………………………………... 81 7.6. Experimental Setup and Procedure ……………………………………...….. 81 7.7. Results and Analysis of Data …………………………………………..…… 83 7.8. Summary and Discussions …………………………………………..……… 89 Part II Conclusions …………………………… ……………….………………….…. 91 References Cited ………………………...……………………………………………. 92 ix
List of Figures
Figure 1.1 Riba robot as a robotic nurse bear, [29].
Figure 1.2 Paro robot interacting with elderly people, [30].
Figure 1.3 Romeo robot proving companionship and service, [31].
Figure 2.1 Specifications of Milo robot.
Figure 2.2 Milo robot interacting with Autism children.
Figure 2.3 Kasper robot.
Figure 2.4 Nao robot interacting with an autistic child.
Figure 2.5 Leka robot.
Figure 2.6 Representative social robots (top left: Muu Robot, bottom left: Infanoid
robot, top right: Bandit, and bottom right: Romibo Robot).
Figure 2.7 EZ-Robot.
Figure 2.8 Color tracking software used in EZ-Robot.
Figure 2.9 Glyph code.
Figure 2.10 (left) A representative QR code, and (right) QR code used for robot
maneuvering.
Figure 2.11 Myo armband and its application.
Figure 3.1 Eye tracker system (from https://www.tobiipro.com/).
Figure 3.2 Experimental environment.
Figure 3.3 Participant with gaze angle irrespective. x
Figure 3.4 Gaze fixation chart.
Figure 3.5 Percentage of gaze fixation for all four subjects (1: EZ-Robot, 2: toy, 3:
computer, and 4: human).
Figure 3.6 Probability plot for all four subjects (EZ-Robot, toy, computer, and
human).
Figure 3.7 Plot of Gaze fixation comparison.
Figure 3.8 Heat map.
Figure 3.9 Counting eye fixation and size of each fixation.
Figure 3.10 Example of heat map plot and fixation counts.
Figure 4.1 QR Code with “happy word” information.
Figure 6.1 Increase and decrease between heart beats depending on what the person
is doing in a healthy body.
Figure 6.2 Corrugator Supercilious muscle.
Figure 6.3 Heart rate before and after getting angry.
Figure 7.1 Northeastern University Driving Simulator with participant during driving.
Figure 7.2 Connections of different sensors.
Figure 7.3 The sequence of steps of experiment in driving simulation.
Figure 7.4 Screenshot of Tempo SlowMo Original App. xi
Figure 7.5 (A) Interval plots for comparing 3 types of music and distribution of
samples by EMG; and (B) Probability plot for 3 types of music by EMG
signal.
Figure 7.6 (A) Interval plots for comparing 3 types of music and distribution of
samples by SC signal, and (B) Probability plot for 3 types of music by SC
signal.
Figure 7.7 (A) Interval plots for comparing 3 types of music and distribution of
samples by HR signal, and (B) Probability plot for 3 types of music by HR
signal.
xii
List of Tables
Table 3.1 Summary of subject’s data.
Table 3.2 T-test for comparing interactions between robot and human, robot and
computer, and robot and toy.
Table 7.1 Different values and speed of music per minute for different scenarios.
Table 7.2 Descriptive statistics of EMG indicators between three types of music.
Table 7.3 Descriptive statistics of SC indicators between three types of music.
Table 7.4 Descriptive statistics of HR indicators between three types of music.
1
Part I
Human-Machine Interaction with Application to Social Robots
2
Chapter 1
Background and Motivation
In this chapter, an introduction to social robots with application to therapeutic treatment
is given, followed by a literature review on children with autism. A number of proposed
treatments are discussed to improve patients with this disorder. The first few subsections
deal with the disease itself and psychological treatment in the long run. An overview of
the recent applications of robots (both humanoid and non-humanoid) in various
therapeutic tasks used by patients is given in the later subsections in this chapter.
1.1. Introduction and Overview
Social robots are becoming popular in various fields as tools and have been considered by
many researchers in recent years. Recent research shows that human-robot interactions
can have wide-ranging applications in the field of therapeutic interventions for children
with developmental disorders, including autism spectrum disorders (ADS). Utilization of
robots due to features such as the ability to program and fatigue repetition can have many
benefits in dealing with humans. For example, social robots can help improve elderly loneliness and their social skills. Robots can also be used in providing physical strength and the ability to cooperate alongside with the doctors. When a robot performs goal- directed movements, better outcomes can be achieved compared to traditional physical therapy. 3
Autism is one of the serious illnesses in the brain developmental disorders. This
disorder makes it difficult for people suffering from this disease to communicate with
others and their outside world. In some cases, self-harm and aggression are also seen. In
these people, repetitive movements (e.g., touching, jumping), unusual responses to others,
attachment to objects, or resistance to change, and unusual sensations in the five senses
(vision, hearing, odor, and olfaction) are common. The Autism symptoms and severity
vary from person to person, but all types of Autism can affect people’s ability to
communicate with others. While no specific treatment for Autism has yet been found,
there exist some good treatments that can help these patients to increase their ability.
These therapies can not only help children reducing their abusive behaviors, but also teach children the ability to independently fulfills their needs and gradually reach a normal life.
Treatments are generally classified into several categories, which could include:
Behavior and Communication Approaches, Dietary Approaches; Medication, and
Complementary and Alternative Medicine [1-3]. In treating children with Autism with a robot, researchers believe that a robot can be a key to the diagnosis and treatment of
Autism. Dr. Carolyn Garver has been working in Autism treatment center in Dallas with a lot of experience with autistic children over the past 40 years [4]. Carolyn is one of the first people to prototype a robot therapy with autistic children. Her response to social
robot was immediate; she says “they wanted to touch him, they wanted to talk to him, they
wanted to see what he would do, they wanted to take him and put him in their classrooms”.
Children with autism understand physical world much better than the social world. They
also have an affinity for technology. One thing found in working with children and robots
is that children build a relationship with robots and they draw other people into this. Some 4 of the social challenges with autistic children are that they sometimes do not engage in social or emotional interactions with other people. However, they are recruited engaging with robots so they are trying to use a word system to engage in their social and emotional activities to promote an actual reality by practicing the emotional and social interaction with words within a robotic system so they can better cope with other people in reality.
Progress has been made in the field of technology and has opened hope for new solutions in the field of interaction with patients and facilitating their treatment. Social robot is a common point between various scientific fields, including engineering, sociology and psychology. It allows humanoid robots to be intermediary agents playing an effective role in various tasks that interact directly with humans. One of the important applications of this new field is the use of humanoid robots with special capabilities in the therapeutic areas, it is in mutual communication with patients. Recent research in the field of social robotics can be applied to reduce the severity of stress and pain in patients with cancer.
Other applications of these smart robots can be used for companionship with the elder people. The intelligence and programing capabilities of these robots provide the opportunity to use the data as a good companion for these groups.
In addition to therapeutic applications, using robots in education also has achieved a good milestone. Application of humanoid robots in interaction with children and adolescents increase the efficiency and speed of education. Focusing on a single activity for a long time is very difficult for these children, and that is why parents and therapists often use repeated action to teach them. Using these ways, they can teach rules and guidelines to children to help them understand and interact with the world around them.
This makes the child more motivated and engaged because his or her attention is more 5 focused on the toy and the task at hand, and not just the re-enforcer. Simply amazing this could be a great tool for therapists to connect with these kids and keep them more engaged.
The second very difficult thing in this process is to actually getting into the field of technology and how it can work with individuals with autism; how it can empower therapists to work with those individuals and ultimately the most out of his amazing toy.
However, it is really more than just a toy; it is a companion as an interactive robot and a wonderful educational tool.
1.2. Problem Statement
Normal children learn complex behavior by observing other behavior they gain, for example, from observing by imitation from friends or parents. Imitation in social communication and social behavior is widely seen. Children learn more from what they see happening around them rather than being told what to do. However, in Autism children due to lack of communication, lack of understanding perception, and lack of verbal and non-verbal communications, they can’t imitate and learn from other people.
That is, they do not tune into other people in the same way as typically developing babies and children. Joint attention is needed for developing communications and language skills. Many autistic children have difficulty with joint attention which will make it hard for them to learn skills such as taking turns, interpreting facial expressions or keeping to the topic of a conversation, very hard to learn how to behave normally.
Study shows that robot can help children with autism to improve their social and cognitive skills by creating fun and educational games to attract their attention. Games increase motor, cognitive and emotional skills and encourage social interaction to build a 6
relationship and connection. What we do is we actually venture to go into a child’s world to build a connection to help them develop better social skills and having a life with passion, confidence and purpose. The fundamental treatment to educate children with autism is trying to push them to learn new skills to be independent and being able to pay back to the society. No matter why it happens, ASD is being diagnosed at a higher rate every year so it is important to find a way to help this indivisible to be part of our normal society. As a result, the goal of this dissertation is to investigate the potential of using robots as therapeutic or educational tool specifically for children with Autism.
1.3. Research Questions
Our research in social interaction between robots and people with Autism has led to the
following questions that we would like to explore through performing this research.
Q1: Do individual have more tendency to interact with robots?
Q2: Is the robot intended as a catalyst for social interaction, socialization of ASD
children, rather than a teacher for a specific social skill?
Q3: What methodologies and techniques need to be developed and used for
children with ASD through the use of social robots?
1.4. Literature Review
In the past couple of years, the idea of a machine that can interact with the world around
has been one of the motives behind artificial intelligent [5]. Recently, the use of robots in
the field of training and diagnosis and interactive in the therapy or education of people
with autism has grown dramatically [6, 7]. Most of available robotic systems base for 7
Autism children use a toy-base as a therapist and social assistance. However, recent studies found that robotic systems, capable of having complex interaction patterns and feedback for the treatment, are more useful in therapeutic intervention [8]. Researchers who use robots in the field of treatment of Autism often report increase enthusiasm and engagement, increase level of attention, and exquisite social behaviors such as attention, sharing and self-imitating individuals in their observations. Research indicates that children with Autism are naturally involved with such techniques.
Implementation of the application for teaching language and academic skills to children and adult with Autism who work with computer is very effective method for helping them [9]. Children with Autism typically play with a toy repetitively and monotonously, and mostly do not play with other kids. Data show that many individuals with ASD demonstrate a preference for robot-like characteristics over non-robotic toys
[10, 11]. The method of teaching interactive imitation is a naturalistic way, and its major emphasis is on the social role of imitation. In classical behaviors and other educational programs, Autism is especially receptive to a lot of inputs and so it is actually a sensory overload issue; that is, these children are getting more inputs than normal people.
Somehow, when they can hyper focus on a robot even though it is busy and limited, it shuts out all the other noises for them, and hence, they find that somewhat comforting or even relaxing [12]. The results show that ASD children did not lose interest in robots and their performance improved over time [13].
ASD children are all different, they all have different challenges and they all need to be approached differently but one thing most of them do not have in common is that they would rather play by themselves. Children with ASD speak more in general, while 8
also willing to interact with a social robot than with another adult or computer game [14].
Scassellati worked on social communications in robot by imitation, [15]. He believes that social functioning, based on the complex interplay between multiple people and the environment, when utilizing a robot has more advantages over direct experiments on human subjects. Humanoid robots have been also used in many research centers therapy in the world. The notion and attempt of using robots to improve imitation and turn-taking skills is very useful for parents and therapist [14, 15].
Robot features repeatability and high flexibility, and do different things without fatigue. The feature of non-humiliation and ridicule by robots has led to a reduction in the anxiety and increase in the willingness of these people to attend training and learning to learn more [16, 17]. Kohima and Yano (2001) built a child-like humanoid, infamous, and a small creature-like robot [18. They investigated human social development, especially of interpersonal communication. There are some robots that researcher designed to make a self-esteem for individuals with Autism. Werry worked with pairs of autistic children and showed that the humanoid robot’s ability to provide a focus of attention and shared attention [19]. The report shows that each robot, having particular characteristics, allowed to create interesting interactions with each child [19].
Nao is a humanoid robot developed by Aldebaran Robotics, a French robotics company. Nao, with teachers guidance, can teach ASD children to focus, converse, imitate, even pick up on the subtleties of human emotions that these kids normally struggle to identify. Another robot that is especially used for individuals with autism and designed to be interesting and approachable for learners with ASD is Milo. This
humanoid robot can walk, talk and even model human facial expressions. Michaud and 9
Théberge-Turmel designed robots with different appearance such as elephant, a spherical
robotic ‘ball’, and a robot with arms and a tail with different interactive capabilities that
could engage children with autism by playing different games, [20]. The benefits of
using robotic systems for learning social behavior and communications have been
demonstrated over by many researchers.
1.5. Autism and Its Social Implications
Autism, referred to as Autism spectrum disorder (ASD), is a complex developmental
disability typically appear during early childhood before three years old and affect an
individual’s ability to communicate, and interact with others. No specific cause for
autism has been identified but the fact is that abnormal brain function could create
Autism states [21]. Leo Kanner was the first person to explain autism, [22]. According
to Kanner, underlying feature of autism is the inability to establish relationships with
others and situations that begin from the beginning of life. Some of the early signs of
autistic disorder are described by Borden [23]. Autism is one of the five diseases that are
generally referred to as “penetrating growth disorders”, or “PGD”.
Children and adults with Autism have problem in verbal and non-verbal communications, social interactions, and game-related activities. Generally, they have three main categories of failure; inadequate social interaction, inadequate communication, and repetitive behaviors [24]. Autism is a type of developmental disorder (social relationships) characterized by abnormal verbal communications, social behavior, and communication skills. For these children, there is no difference if they are hugging or not. They have weak muscles and crying very little, but they may be highly 10 irritable. In the first six months of life, they cannot pay attention to their parents, they do not ask for anything, do not smile or delay in this regard, and are not interested in toys. In the second six months, they are not interested in social games, they are cold and insensitive, do not appear to communicate verbally or nonverbally, and are less active or more active against stimulations.
ASD affects how people communicate with others. This is not the same as being shy or not knowing what to say. Most people with Autism do not understand some of the basic social conventions that others take for granted. They might have trouble making eye contact, holding a conversation, or recognizing gestures. About one-third of people with ASD are nonverbal, meaning they do not use speech. Along with communication issues, people with Autism often like to follow certain patterns or repeated behaviors.
Many have sensitivity to bright light or loud noises, and others have physical problems such as troubles walking or picking up small objects. Some have intellectual disabilities, but about half have an average or above average IQs. It is also common for people with
Autism to have a great long-term memory for certain details, and many excel in math, science, music or art. With such a wide variety of symptoms, no two people with ASD are alike. The behaviors vary so much that they used to be classified as different disorders. One was Asperger Syndrome, where people obsess over topics, miss nonverbal social cues, and may not understand appropriate social behavior.
In 2013, scientists realized that the boundary between Asperger Syndrome and some of the other disorders was fuzzy, so they decided to put them all under one name.
This made it easier to diagnose Autism, which is important. The younger someone is diagnosed, the earlier they can get help. In fact, many children who have Autism will 11 show signs by two years old, including not responding to their name, avoiding eye contact, and flapping their hands or rocking repeatedly. Even though there is no cure for
ASD, therapy and medication can help people adjust. Scientists are also doing clinical trials to find other solutions. They have learned that one out of every 68 children in the
US has the disorder, but they still are not sure what causes it. Autism is over four times more common in boys than girls, and most scientists think genes play a role because it often runs in families. Some people with ASD have abnormal chromosomes, but this cannot be the whole story, as there are people with the same gene changes who do not have Autism.
Other possible causes include having older parents, being exposed to high levels of testosterone in the womb, and having complications during pregnancy or birth. It may even have something to do with gut bacteria, since many autistic people have gastrointestinal issues. However, it is most likely a combination of many factors. The one thing scientists know for sure is that vaccines do not cause Autism. The most popular signs of Autism are delayed learning to speak or do not speak at all; repetitive behavior; inability to make eye contact; inability to say the names and avoidance of cuddle, or asking for help; inability to start conversation and talk to others and keeping the conversation; unusual responses to people; sensitivity to light, sound or touch, and strange food habits.
Self-control behaviors such as hitting head to ground or wall or hitting with hands make it difficult for them to communicate with others and the world around them. When these children grow older, it is possible that some of them become more interested in relationships and less misunderstood. Some of them are pretty much upbeat and 12
sophisticated normal. The rest of these children still have problems in skilled linguistic
and social and adolescence and their maturity in the illness is very intense. Most children with Autism are slow in learning science and skills and some of them have signs of low intelligence; while other children with autism have problems in learning but not in a social, verbal and linguistic relationship. A small number of children with Autism
potentially have high skills to learning special skills such as art, music or mathematics.
1.6. Social Robots
Social robots are referred to the use of robots in direct communication with humans that
involve verbal or motor interaction, while the sociality between robots is not part of this
field. The term of social robot was considered first by Billard and Dautenhahn, [25, 26],
Breazeal and Scassellati [27], as well as Fong and Nourbakhsh [28]. This is the area
where it employs the robot to help human beings in daily life. To use it, a smart tool is
used to help educate, treat as well as perform social services. Such applications and
services in public places and amusements are the notable items in the field of social
robotics. Because the main subject of this dissertation is on the use of social robots in
therapeutic applications, we will discuss several examples of the work and research that
have been done in the interaction between the robot and the human for therapeutic
purposes.
Social robots in interacting with elderly are used as assistive devices to give a
service to elderly people in homes or hospitals. Functionalities are related to the support
of independent living by supporting basic activities (e.g., eating, bathing, toileting and
getting dressed) and as a navigation, monitoring of those who need continuous attention 13
and maintaining safety. The robots that have been used in this field, whether in research
or in the executive, are explained and described briefly next.
Riba: The labs in Japan has created a robotic nurse bear, called Riba. This is one of the
ways that Japan is preparing for its growing elderly population. It has a height of about
140 cm, weight of about 180 kg (including battery), [29].
Figure 1.1 Riba robot as a robotic nurse bear [29]. Paro: This robot is manufactured and developed by a leading Japanese company, called
AIST. Paro has five types of sensors; tactile, light, audition, temperature, and posture sensors, with which it can perceive people and its environment. With the light sensor,
PARO can recognize light and darkness, [30].
.
Figure 1.2 Paro robot interacting with elderly people, [30].
14
Romeo: This robot is made by a French company called Aldebaran. It can help elderly with household tasks and even become their friends. The humanoid robot movement is driven by maxim’s motors. The robot is about 146 cm tall with soft facial features that give him a friendly appearance [31].
Figure 1.3 Romeo robot providing companionship and service, [31].
1.7. Needs and Facilitation
Studies have shown that children with Autism prefer simple action and predictable
environments [32]. Behaviors and disorders in autistic children are different and
variable; therefore, this should be the case in the treatment of autistic children. The
treatment method for each child should be proportional to abilities of his or her disorder.
These children should be evaluated according to their growth and learning time. This
type of training can reduce behavior problems and improve adaptation skills. With the
robot, the game can be designed to suit the child’s ability and enhance his/her motivation
for learning. 15
Continuous treatment for this disorder is very important. Some humanoid robots
are made to teach and educate autistic children as they can have repeated single action for
several times without exhaustion. This technology is more efficient for treatment, it is
predictable and simple in its facial and verbal expressions, but also more constant in its
behavior as a human being. Autistic children have short memory, so they need simple
movement. Robots can do action simpler than human without any change and will help
improve children learning skills. This makes their reaction faster, in addition to making
them enjoy working with robots like toys [33, 34].
1.8. Summary and Outlook
Robotic technology has progressed substantially over the past decades and used in many products. Countries facing the challenge of increasing population can use social robots as one way to address labor shortages. Such robots can become a personal companion for older children and adults to accompany them at home or outdoors. Social robots can even be used as a platform for future health services.
Use of robotic system to engage ASD children in their social and emotional activities is therefore very important to promote an actual reality by practicing emotional and social interactions. Such interactions with robotic systems will help them better cope with other people in reality. The goal of this dissertation is to investigate the potential of using robots as therapeutic or educational aids, specifically for children with Autism. As a result, our approach may lead to the following questions that we explore through this dissertation. Do individuals would prefer more interactions with robots? Depending on 16 the answer, different methodologies and techniques can be developed and used as outlined in the following next chapters.
17
Chapter 2
Human-Machine Interaction with Social Robots
In this chapter, social robots in stable interaction in Autism therapy are reviewed and discussed. Several commonly-used social robots such as Millo, Kasper, NAO, Leka and finally EZ-Robot are presented. A special attention is placed on EZ-Robot with details on its specifications, and capabilities along with the MYO armband attachment.
2.1. Introduction and Problem Statement
It has been very common over the past few years to utilize humanoid robots to interact with individuals with Autism. The benefits of such utilization of robots can be briefly summarized as follows.
1. Due to great effectiveness of humanoid robot for individuals with Autism, it has
attractive properties and significant effectiveness on improving communications
as well as on their treatment. Children with Autism are attracted to robot’s
characters and can make an effective emotional connection.
2. Among the suitable methods for more effective communication with children, the
use of such robots can be more effective in interacting with children. Robot’s
movements and his dialogues can encourage children to engage in the discussions,
answer to questions and give emotional expressions.
3. Humanoid robot, in addition to possess the ability to move and talk, have other
features such as the ability to play interactive sounds, music and feelings the robot 18
may have in his face using the LEDs. These capabilities can be used to improve
interactions with children and, as a consequence, necessary treatments of selected
factors for autistic children.
2.2. Social Robots in Stable Interaction in Autism Therapy
In this part, we will discuss several social robots in order to identify the necessary
capabilities needed to be used for autistic therapy with low cost and more efficiency.
According to the required capabilities and the desired therapy in this research, a
humanoid robot, called JD-Robot manufactured by EZ-Robot was purchased. Because
the underlying objective of this project is on the use of social robots in autism therapy,
we will list and provide some working functionalities of some select social robots used
commonly for Autism. The concept of stable interaction in this section refers to a
regular and repeated communication between the robot and the human being. This communication can be a daily relationship or weekly relationship between a human and a
humanoid robot at home, therapy center or in the hospital (for a relatively long time).
Throughout this connection, humans and robots get used to each other's behavior.
2.2.1. Milo
Milo is a two-foot-tall humanoid robot designed by Robotkind® to interact with Autistic
people using vocal and facial expressions (see Figures 2.1 and 2.2). Milo takes what
diligence psychologists have been doing with real evidence-based intervention
approaches and skills. It teaches approaches and adds that level of engagement and 19 motivation. This robot is really engaging for these patients and helps them pay attention and learn information so quickly.
Figure 2.1 Specifications of Milo robot. Figure 2.2 Milo robot interacting with Autism children
Milo speaks slowly, he plays games and teaches trivia to kids. He even throws dance parties but this is not just another talking toy. Milo is designed to help children with
Autism opening up and engaging with him and consequently improve their interaction with other individuals. Milo serves as a bridge and his expressive face helps teach social skills. The research has shown that using Milo in therapeutic interventions for children with Autism has significantly increased eye contact, body language and friendliness.
2.2.2. Kasper
Kasper is a British-made robot developed by the University of Hertfordshire. It has been used in helping autistic children. A child-sized humanoid robot, Kasper can talk and is equipped with skin sensors on various parts of his body including on his cheeks, torso, arms, palms and feet. Such arrangements allow him to respond to touch and other interactions, as his face was designed to be simple with easy-to-read expressions, [35]. 20
He can also comb his hair, imitate eating, play the tambourine and sing along with children.
Figure 2.3 Kasper robot.
2.2.3. NAO
NAO, a French-made robot by Aldebaran Robotics, is a programmable kid-sized humanoid robot with a height of 22.5 inches, weight of 4.5 kg and with 25 degrees-of- freedom (DOF), see Figure 2.4). Autonomy is 60 minutes in active use and 90 minutes in normal use.
Figure 2.4 Nao robot interacting with an autistic child.
21
This robot is fully autonomous and is available for education, research, companies
and hobbyist developers. The features of this robot include the full planning capabilities,
the presence of a variety of sensors, the presence of a local computer, remote control and its lightweight body. NAO listens, talks, recognizes knocking, connects to different devices in various ways, communicates with another robot in different ways, balances, moves agile. It can also pick up different objects and plan in different ways. This robot was not purposefully designed for autistic children but it has been used in this area of research.
2.2.4. Leka
Leka is a cute robotic and interactive smart toy the shapes like a ball. It can run around
and come back to the child and responding to calls (see Figure 2.5), [37]. Parents who
have been using this toy have been supportive of its use and claim that it a grate smart toy
and their children were immediately able to connect to this robot.
Figure 2.5 Leka robot.
22
2.2.5. Other Social Robots
There are other smart toys and robots for communicational supports and therapies. These social robots are intended to make connection between doctors and patients in medical environments. Figure 2.6 depicts some of these social robots; some may look like toys, others are smart toys while some of them look like a real humanoid robot.
Figure 2.6 Representative social robots (top left: Muu Robot, bottom left: Infanoid robot, top right: Bandit, and bottom right: Romibo Robot).
23
2.3. EZ-Robot As A Social Robot
The social robot used in this research is an EZ-Robot, manufactured by EZ-B and created
by Tony Stark [38]. It is a small size humanoid robot, with 16 degrees-of-freedom. The
concept has taken many years of development in both hardware and software to achieve
the EZ-Robot’s innovation. EZ-Robot is a programmable robot in the sense that therapist have the option of adding customized actions and movements. The EZ-Builder is a software that runs the robot and very easy to use. This software allows creation of movement animations for robot to walk, dance and exercise. It has the following features and attributes; speech recognition, vision tracking, joysticks, Wii controllers, and many more. EZ-Robot can be pre-programmed for certain sequence of motions, in addition to
possessing the capability to add a more customized action as well as the being controlled
by therapists or children in a tele-operation mode.
2.3.1. EZ-Robot Specifications
The EZ-Robot is 15 cm-wide, 13 cm-long and
33 cm-tall. It weighs about 1.33 Kg. The
following is a list of its specifications and
capabilities. These features are all detailed in
the EZ-Robot original website [38]. Figure 2.7. EZ-Robot.
1 EZ-B v4/2 Wi-Fi Robot Controller
1 Humanoid head with camera and RGB eyes
6 Lever HDD servo motors 24
2 Servo-motor grippers
1 Humanoid 2 Servo-motor Foot and Ankle (Left)
1 Humanoid 2 Servo-motor Foot and Ankle (Right)
1 200MHz 32B Processing
1 WiFi Connectivity (Ad-Hoc/Infrastructure/WEP/WPA/WPA2)
1 Amplified Speaker for Speech and Music
All ez-robot information comes from the original website of ez-robot [32].
2.3.2. EZ-Robot Capabilities
All social robots used in modeling and enhancing the social skills as assistance to therapist or parents to interact with children should have the special capability to encourage in social behavior when dealing with children with ASD. In the following, these capabilities are described for this special social robot.
Color Detecting and Tracking: Children with Autism have less accuracy and memory
in the diagnosis of color than healthy children. One of the tutorials in the ABA technique
is color identification. EZ-Robot, using its camera, can track colors. For this, the camera
head of the robot moves to follow an object of a specific color. Using this capability, we
can teach the robot to detect and track the colors of our choice.
25
Figure 2.8. Color tracking software used in EZ-Robot.
Voice/Speech Recognition and Voice Training: To make more natural communication
with human, the recognition of the sound is necessary for human-robot interaction. Most
individuals prefer to use a robot that has fewer capabilities than to use a robot that has a lot of functionalities. One of the most important capabilities is the voice recognition in order to have a better relationship with humans. By recording the motion and gesture noises in advance, the noises are easily estimated and changed to movement doing the job we want by voice recognition. Human-robot conversation is one of the most difficult aspects of developing a robot capability that most people really love this part of the robot,
[27].
In EZ-Robot, by using speech recognition option we can create and add customized speech recognition commands and have different actions. Alternatively, the robot can speak in response to whatever it is commanded to perform. Speech recognition is a fun way of interacting with robot especially for children. More specifically, EZ- 26
Robot has a Voice Training option, wherein the robot can be by trained for better
recognition of voice.
Face Detection Emotion: Social Robots have progressed over the last few years. These
robots must recognize human emotions to have better communication with them, and
hence detecting people faces is of great importance. Non-Violent Communication is referred to states in which it is necessary to know well and understand how every event has influence in us [39, 40]. Human emotions have been identified through image analysis (changing facial expressions), physical situations analysis, change in physiological states of the body such as movements of the hands and feet, as well as speech analysis, [41]. The EZ-Robot uses image analysis to recognize human emotion by using Microsoft cognitive vision. This software has a service to describe contents or read images from the robot camera through the web. The EZ-Robot uses this feature to recognize emotions like happiness, sadness and anger. The algorithm for processing face emotion during this process is composed of the following steps: taking an input image, applying a series of preprocessing, and finally categorizing the image in one of the facial
expressions. The facial expressions include six modes; anger, discomfort, joy, surprise,
hatred and fear.
Glyphs: Glyphs is an “elemental symbol within an agreed set of symbols, intended to
represent a readable character for a specific purpose [Wikipedia] in the robotics industry,
see Figure 2.9. Glyphs can be used to control movement of robot with the camera. When
glyph is far away, the robot will move forward, and consequently, when the glyph is
close, the robot will move in the reverse direction. 27
Figure 2.9 Glyph code representing a readable character for a specific purpose for use in robotics industry.
QR Codes: QR Code is the trademark for a type of matrix barcode. “A barcode is a machine-readable optical label that contains information about the item to which it is attached”, [Wikipedia]. Being aware of the locations and navigating the robots are some of the challenging issued in robotics industry. In short, how does a robot know where it is and where to go, as well as how to get there? QR codes, sometimes called robot vomit, can be used to help resolving this challenging problem. By sticking QR codes in a desired location, the robot, using its video camera, can read the QR code information and set the variable to realize the target location (see Figure 2.10). The QR Code is a tracking type device for most robots, especially for EZ-Robot.
Figure 2.10 (left) A representative QR code, and (right) QR code used for robot maneuvering. 28
2.3.3. MYO Armband Attachment
Myo uses muscle EMG activities to grab and release gesture detection. In Myo armband technology, electrical activities from muscles are measured to detect movement and sense motion, orientation and rotation of forearm. This information is then transmitted over a
Bluetooth-enabled smart connection to communicate with compatible devices, [42]. Myo could theoretically allow users to control tons of things on computer, drones, robot smart phones and even smart TVs. Through the use of arm muscles for movement, the device includes small teal Bluetooth adapter and teal micro USB cable. The Myo itself looks very sci-fi inspired, with its overall design and band expandable between 7.5 and 13 inches depending on forearm circumference using the sizing clips. It weighs about 93 grams and contains a three-axis gyroscope, a three-axis accelerometer and a three-axis
XS magnetometer.
Figure 2.11 Myo armband and its application.
Myo gesture control armband interprets muscle movements in human arm and turns them into signals. A computer or smart phone can understand these detectable signals and can turn them into favorable actions. Myo understands five main gestures making a fist, open fingers swiping left, swiping right, and double tapping thumb and middle finger together. 29
EZ-Robot has an option to interactive with Myo armband. It uses gestures and an accelerometer to move servos for moving or grabbing objects. The servos can mimic human movement using its built-in accelerometer.
30
Chapter 3
Human Behavior in Gaze Interaction with EZ-
Robot
In this chapter, our focus is on ways in which autistic children can be engaged with humanoid robot in simple interactive activities and investigating the interaction of human to robot. Due to limited access to human samples, the experiments are performed on normal adults. Using the results of this study, we can expand it to other areas of research.
3.1. Introduction and Problem Statement
This section provides a quick overview of analysis and discussion about occurrences of joint attention in social robots that emerge spontaneously in natural interactions between individuals and robot. In this study, a humanoid robotic serves as a focus of attention in the environment that mediates these interactions. The emphasis here is on the interactions amongst the participants, as well as the four objects used (i.e., robots, computer, toys and human). Throughout this chapter, we would like to explore how adult initiate and orientate joint attention during interactions that involve a robotic device.
3.2. Stimuli and Apparatus
The major apparatus in the experiment is a social robot. The social robot used in this research is the EZ-Robot. As mentioned in the preceding chapter, this humanoid robot is capable of displaying human-like body gestures, speaking, playing sound effects and 31
music, and dancing. All of the aforementioned capabilities can be designed and
programmed in order to display the desired actions. It can be controlled via a computer interface (Windows 8) or smartphone (iPhone 7 or higher). We also use Woolen doll as a toy, a 13-inch monitor Lenovo laptop equipped with an eye tracker information to play the video, as well as a human seating beside these objects.
The aim of this study is to compare these four objects and measure and assess participants’ reactions to the objects/human. The reason for choosing these samples in the experiment is that most children during therapeutic works may interact with human as a therapist, and toys as they also like to play with toys. The research also shows that these children are interested in electronic devices such as computers and mobile devices.
3.3. Eye Tracking Process
Eye tracking systems have been used in many fields including rehabilitation, psychology,
cognitive science, and human-computer-interaction, and marketing. This is considered as
one of most common ways to measure eye movements. Duchowski’s eye tracking
methodology shows that Gaze analysis can be used for many applications, [43]. In recent
years, the precision of eye tracker systems has increased dramatically without the
requirements of chinrest and headgear. This technology can perform eye checkup
routines for infants, children and adults very easily and effectively.
Eye tracking can help businesses and scientific organizations to gain real insights
into human behavior, [44]. Tobii® Pro Insight, a research consultancy, conducts
attention based studies helping organizations make better business-critical decisions (see
Figure 3.1). In this study, the Pro2 Tobii glasses are used because they provide the 32 ability to store the perception of the environment by the person in a real environment.
Pro2 glasses are effective tools for determining how vision and behavioral gaze during interaction with an object can be assessed. It is expected that these qualitative data will provide the ground for creating new techniques, optimizing the Autism classroom, and improving the overall quality of training and even therapeutic treatments. Tobii Pro2 glasses are also great tools for video quality analysis to detect children’s attention to robots are different objects. An eye tracker can be used to check when an individual can look at several objects at the same time and focusing exactly on which points (see Figure
3.1).
Figure 3.1 Eye tracker system (from https://www.tobiipro.com/)
Why eye tracking? Eye tracking systems provide a special opportunity to see what happens behind the scenes. Through these systems, items such as the total number of fixation, the average time of fixation, their density, and so on can be determined. Using 33
eye tracker system, one can realize each object and the frequency and duration of steering
at them. This system is used in our research in order to determine how long and how
many times each participant looks at each of the object. Eye tracking systems can be
specifically used in places where traditional practices such as simple question and answer
techniques are restricted and cannot be effective.
Why Tobii? Tobii Pro2 eye glasses allow to track eye movement without contact. The
Tobii system provides an easy and fast approach to transport analytical data. Also, the
Tobii Studio platforms such as heat map, gaze plots and clusters can quickly provide data
that can be used for more accurate analysis.
3.4. Participants
Our participants sample consists of 28 (19 males and 9 females) Northeastern University
students, randomly selected with an average age of 24.3 years. All individuals had
normal vision, and were volunteered to participate in the experiment. The experiments
were conducted at the Northeastern University Human-Factors research facility. The
experimental procedure consists of behavioral data collection and video recording steps.
The experimental procedure did not call for invasive or potentially dangerous alteration;
instead it provided a fun platform for participants to enjoy. All data were stored and
analyzed anonymously, so no sensitive information was collected.
34
3.5. Experimental Procedure and Methodology
The main objective of this research is to compare the attractiveness between robot,
human, computer and toy. In order to evaluate the interaction between humanoid robot
and other objects, an eye tracking system to detect gaze was designed, developed and
implemented. The most important goal in this experiment is how these ideas can be
applied in practice through designing and building the required interface. More
explicitly, we target robot’s use for individuals with Autism working within social contexts, and try to observe people working and interacting with our social robots implementations.
For evoking communications, we propose a gaze-communicative stuffed-toy
robot system that is aware of the user’s conscious and subconscious gaze. In this section
we detail our approach of using these experiences in order to demonstrate the
attractiveness of social robots. This research utilizes the unique properties of social robots
and how they relate to interaction with people especially autistic individuals. It also
presents a detection mechanism into a wider social context interaction with robot. It also
provides design, development and implementation for evaluating interfaces to validate
and further develop our theories.
A controlled experiment was designed to compare the effects of interactions with
a social robot, the EZ-Robot of Figure 2.7. The interactions between a human and robot,
toys, and computer were investigated. Each participant in our study completed a
sequence of 2-min interactional conditions. The experiments run in a large room, while
participants were conducted in another observation room, different from the room where
interactional conditions were administered. Before participants entered the experiment 35 room, all conditions and instructions were explained to them, while they were not aware of the aims and objectives of the experiment. Upon entering the room, participants wear the eye tracking system for tracking their gaze. For each participate, the eye glass must be calibrated in order to get the best and most precise result.
Once participant enters the room, independent from their height, they stand 1.5 to
2 meters away from the experiment table where objects were placed (see Figure 3.2).
The objects on the table include a number of regular toys without any movement, a computer playing a video clips that were taken from the same robot, the EZ-Robot itself, and a human. The video stimuli shown in each scene on the computer include robot dancing, signing, going forward and moving hands in order to cause different distractions.
The EZ-Robot was showing a number of activities such as exercising, dancing, moving forward and backward, and hand and leg movement. Gaze position and natural viewing behavior for both eyes, with Tobii Pro2 glasses, were captured. Tobii Pro software was installed and used on a Lenovo laptop (running Windows 8.1 Pro, with
Intel® Core ™ i7 processor, 8 Gb RAM, and 64-Bit system type). This was the optimum system provided by LC Technologies. Other hardware was optimized to fulfill the minimum hardware requirements of each system. The pattern gazes for each participant were captured and analyzed. The amount of attention paid by each participants on the objects was further studied. Each activity session involves recording the eye tracker, social robot movement, playing the video, human gesture with facial movement and the fixed toys for about 120 seconds.
36
Figure 3.2. Experimental environment.
EYE TRAKING HUMAN ROBOT
COMPUTER
TOYS
Figure 3.3 Representative participant with shown gaze angles.
Figure 3.3 shows the system structure of our gaze-communicative robot, toys, computer and human. A camera is placed on the eye glass to detect user’s gaze. The recording unit holding by participant hand is connected to eye glasses with an HDMI 37 cable and connected to computer trough WiFi. The eye tracking system server sends the angles of user’s gaze to a multimodal reaction processing server prepared for the reactive subjects.
3.6. Results and Data Analysis
The following analytic observations focus on area of interest (AOL) including robot, computer, toys and human to test whether participant had higher attention to the social robot or not. Each result was summarized corresponding to the user’s gaze at a longer- duration looking at the robot, toy, computer or the human. All data recording for each participant was executed using Tobbi Pro2 glasses, and data were collected by Tobii Pro
Glasses Controller (64). The Tobii Pro Lab (64) data capture software was used to analyze heat map and eye gaze plot in order to see the point of interest for each of the participant. The AOI and counts of fixation to the subjects for all participant were collected and summarized by multi-dimensional scale.
Figure 3.4 Gaze fixation chart. 38
Figure 3.5 Percentage of gaze fixation for all four subjects (1: EZ-Robot, 2: toy, 3: computer, and 4: human).
When participants gaze to each object, the Tobii Pro Lab (64) software record such gaze fixation and has an option to count this fixation as a fraction of a second. Figure 3.4 demonstrates the total of these fixations, while Figure 3.5. shows the percentage of these fixation count.
Figure 3.6 Probability plot for all four subjects (EZ-Robot, toy, computer, and human). 39
Table 3.1 Summary of subject’s data.
SUMMARY Groups Count Sum Average Variance Robot 27 1353 50.11111 745.1795 Toy 27 387 14.33333 68.76923
Computer 27 454 16.81481 79.849 Human 27 189 7 28.07692
ANOVA ource of Variatio SS df MS F P‐value F crit
Between Group 29723.80556 3 9907.935 42.99038 3.75E‐18 2.691979 Within Groups 23968.74074 104 230.4687
Total 53692.5463 107
Fixation duration each subject: The average of the subjects’ total eye fixation is shown by each object (i.e., robot, toys, computer and human) in Figure 3.7. Table 3.1 gives the summary of data. One-factor repeated measures of ANOVA (α=0.05, φ= 20, 2) resulted in the probability p<0.01. Based on our model of gradual gaze-communication between robot and toys, computer, and human, we could verify that the effect of joint attention and eye-contact for the interest of participants to robot. As compared in Figure 3.6, the probability of four objects shows the distribution of data that follow a specific distribution.
Comparison of the area of interest (AOL): The results shown in Figure 3.7 and Table 3.2 show there is significant differences between robot and computer, toys, human, our hypothesis is supported by these results, which indicates that the subject looked more at the robot toward computer, toys, human. 40
Figure 3.7 Plot of Gaze fixation comparison.
By observing the eye tracking data using heat map and Gaze plot captured during experiment, one can determine the possibility of statistical analysis of those particular elements to the research. This is done by extracting the results from heat map by choosing the AOL for four objects in the experiment as desirable AOI areas.
Table 3.2 T-test for comparing interactions between robot and human, robot and computer, and robot and toy.
robot humna robot toy robot computer Mean 50.11111 7 50.11111 14.33333 50.11111 16.81481 Variance 745.1795 28.07692 745.1795 68.76923 745.1795 79.849 Observations 27 27 27 27 27 27 Pearson Correlation ‐0.00771 0.492886 0.017905 Hypothesized Mean 0 0 0 df 26 26 26 t Stat 8.044217 7.648511 6.055565 P(T<=t) one‐tail 7.97E‐09 2.03E‐08 1.07E‐06
Heat map are visualizations data representing important perspective of visual
behavior. Such powerful representation can clearly and effectively illustrate trends and
patterns in the data. Figure 3.8 shows the heat map data and how the location of looking
at objects is highlighted on the stimulus. 41
Figure 3.8 Heat map.
Gaze plots show the visual behavior, used to describe the sequence of fixations
during the simulation. In this experiment, gaze plots were used to show the visual path of
each participant and not for multiple participants as they can be unclear.
Figure 3.9 shows gaze plot fixation duration and time spent looking at,
respectively, the robot, toys, computer and human. This fixation duration and time are linked to the diameter of the fixation circles; that is, the size of fixation markers on a gaze plot is proportional to the duration of the fixation. The larger the dots, the longer the fixation and hence the larger the longer look at the object. 42
.
Figure 3.9 Counting eye fixation and size of each fixation.
Hot spots with red color indicate interest in looking at that object. In some cases, the hot spot can mean that the participant was confused and spent a long time trying to make sense of an experiment object. However, in this experiment and due to the clarity of the elements, looking back at the object simply means the object is interesting and participant would like to spend more time. By observing heating map data of the 28 participants as shown in Figure 3.10, the results demonstrated that 56.78% looked at the robot, 16.2% at toys, 19.05% at computer, and 7.93% at human. 43
Figure 3.10 Example of heat map plot and fixation counts.
The number in each circle shows the order of looking at each element. The results indicate that at the beginning of the experiment and before the robot start moving, the participants were not attracted to the robot and spent more time watching the videos playing in computer or the toys on the table. When the robot started moving, participants have immediately responded by looking at the robot. The interest of participant was 44
increased when the robot started to communicate with them. This indicates a positive
interaction between the social robot and participants. The results also demonstrated even
a small humanoid robot can provide an enjoyable interaction and focal joint attention
between other objects in experiment.
3.7. Summary and Conclusions
The studies reported here revealed the details and qualities of joint attention skills in adult with robot. In many respects, the analysis of the results for adults’ interaction with social robot was very impressive. The interaction between autistic child and robot plays a major role in the possible therapeutic interventions. Before designing the scenario for therapy, we should assure that the robot can make joint attention of children. Through performing this experiment, one can find the answer of the question we posed earlier; that is, can the robot get joint attention more than other objects usually used for therapy? The answer is now becoming very clear as shown in the heat map data of 28 participants in this experiment (see Figures 3.8 and 3.10).
By observing gaze plot results, almost all participants initially paid attention to toys just for a few seconds. As shown in this plot (see Figure 3.10), the numbers show the sequence of looking at the computer because the video shows the movement of the robot. At the start of the experiment, the EZ-Robot just played music, followed by making movement, dancing and turning, talking, then the attention went to the robot.
The robots’ body orientation did get the adult’s attention to the region of the robot’s body. Furthermore, the conspicuous attention to the robot will be understandable as getting joint attention for children with Autism in order to design therapeutic games. The 45 methods used in conducting the trials and the results of these trials show the potential benefits of robots as assistance for both educational and therapeutic interventions.
46
Chapter 4
EZ-Robots as Social Mediators
This chapter provides a brief overview of EZ-Robot as a social mediator and expands the
results obtained in the preceding chapter to propose a number of clinical therapies that
could help in Autism treatment. This includes a review of robots used in the Aurora
Research Project and targeting several therapeutically-used remedies for autistic children such as imitation game therapy, eye interaction therapy, emotion recognition therapy and finally dance therapy.
4.1. Introduction and Problem Statement
Considering the results obtained in the previous section and the attention of the individual
to the robot, and according to the results obtained in this field, we can use the robot as a
friend in the therapy stage or at home with parents to help autistic children. Normal children gain complex behavior by observing others’ behavior and by hearing the voice
of others. As a result, children learn most of the issues from observing by imitation.
That is, they learn simple behavior from peers or elderly before they can impose very
complex skills. Game is the most natural form of the child’s desire to deal with the world
around him or her. Playing is a useful tool in developing the child’s emotional and social
skills, [32, 45 and 46]. 47
Games have the benefits of being used as learning and training tools in order to
balance the subject matter according to ability of the player [28]. One of the most
common and important ASD therapies is the behavioral therapy because behavior is the
diagnostic of the criteria. A major problem for people with Autism is the so-called social
disorder. Children with Autism have many problems communicating with others,
especially with their peers. These children often use toys instead of imaginative or
symbolic games. They use objects in the form of a pattern, or to stimulate themselves.
Game therapy is a kind of treatment that is used for children, in which children reveal
their problems at an imaginary level by puppets, sculptures and other means. Playing game allow children the opportunity to work with other children in a creative and highly social way. This is a great benefit to children with Autism who struggle with peer interactions. Game is a great tool for helping children to take them away, bringing them to the real world and helping them in collaborative engagements.
Moreover, suitable games can help autistic children discover their emotions, environments and make good relationships with parents, brothers/sisters and friends.
Studies show that children’s performance was significantly better with the computer
game as opposed to the non-computer game”, [45-47]. Children with Autism like to
imitate actions with objects, sensory effect like flashing lights and sounds, [34]. A
computer game was designed for autistic children for increasing their fluency in speech,
[48]. A therapeutic game can provide parents with an opportunity to play an active role
in the development of a child with autism. The therapy game can be hard for parents or
therapist because they must spend more time and be patient to repeat an action more and
more and over time. Hence, a robot can act like an assistance to help them. The games 48
designed for children should be welcomed into the community and can be used like a
Theatre Therapy for autistic children where they learn from one another.
4.2. Robots in the Aurora Research Project
The concepts used in this dissertation are motivated in part based on the Aurora project,
an assistive humanoid robot interaction called AURORA (2005), [26]. This study uses
robots as tools for education and therapeutic assistance beside the parents or therapist.
This project started in 1998 by Prof. Kerstin Dautenhahn and many other PhD students
and postdoctoral researchers around the globe have contributed to this research. The research method of the project is by adopted case study in a small group, for long-term engagement of children with Autism in a variety of ways to develop and increase their communication and social interaction skills. Long-term effects are important to demonstrate therapeutic objectives.
Kindergarten and preschool children can learn simple tasks such as recognizing colors, letters or giving “yes” or “no” answers to simple questions by observing friends and healthy classmates who participate in these tasks. Children with Autism, however, cannot be trained through the modeling of their friends and classmates. Many of these children have problems with the coordination of emotions, stimuli, and other social attributes which are among the rest of the children, totally involuntary and natural.
Children with Autism only need one convenient way to receive the information. Many children with Autism are interested in playing with mechanical toys or computers and learning simple tasks through these. AURORA project aims in helping children who have problems in developing general communication and interaction skills that are 49
required in human-human contact interactions. The researchers in this project have
developed new robotic technology and software for the robot to learn and adapt during
the trials. They also investigated and developed new methodologies and played different
scenarios for robot-assisted play and therapy for children with special needs, in particular
for children with autism, [26].
One methodology consisted of conducting an exploratory study following a single
case protocol, [33]. The research was based on ABA method and floor time, as explained next.
ABA Method: ABA therapy was developed in the 1970s by researchers UCLA. The
basis of this therapy is on a “reward” mechanism. One of the variety of treatments for
autism is behavioral and educational treatments. Researchers have reported various
impacts of interventions behavioral in the treatment of autistic children. These results
included increasing child’s growth rate, gaining superior grades IQ, growth of social
behaviors and communication skills as well as reduced autistic behaviors.
Floor Time: This method resembles a therapeutic game. In this method an attempt is
made to communicate the child with another person through detailed game and playing
more and more effective. This method consists of six steps; and the child is mimicking
the stages during which they are taught and how to learn from one another. In other
words, the child must have six steps in the roof to go through. For example, due to sensory impairments in understanding some of them, the exercises are difficult to find, or are unable to do some physical exercises are not included in the program. In this method,
parents and therapists try to make child’s progress through the paths that the child likes
and directs them, so that the child can achieve the six stages of development in this 50 healing method. This method does not help to get spoken and physical skills emotionally and psychologically. However, if in this method a robot can help parents and therapists, it should be more efficient.
4.3. Imitation Game Therapy for Autism
One of the most important techniques in floor time is the technique that is designed to follow the child’s lead as a model and to relate to the child on his or her developmental level. In AURORA research, this role model can be a robot. Children by imitation from the robot have personal attachment to the situation; and through watching robot action, retaining the information, and then later replicating the behaviors had improved interactions. Imitation of body language of a robot is more interesting and easier and fun for children. Imitation can be used to help autistic children learn and develop cognitive skills.
Research shows that modeling techniques are widely used in helping children with autism to learn appropriate behaviors. How to start playing with robot? The best recommendation is to start slowly by simple game, then increasing the difficulty and intensity. Children are beginning the program with very low skill, comfort, and tolerance levels. As they gain skills and confidence; however, they can endure more time spent and more challenging tasks can be handled. For example, if the child is flapping his arms, the robot is also flapping its arms. It is very important to bring pleasure and comfort while encouraging the child to interact with robot. Therapists or parents must be able to ensure that the child would feel safe and supported in what is happening and is able to feel comfortable with taking a risk in his or her play. By using this relatively 51 inexpensive social robot (i.e., the EZ-Robot) at home, for example, the child is able to see these skills for daily life, and not just for use in the therapist’s office. This approach allows the development of a very personal and unique relationship.
1. Imitating child from the robot: gestures and body movements such as clapping hands,
weaving hands, moving arms up, moving head from left to right, standing in front of
child, and performing a movement first then asking the child to do the same
movement.
2. Imitating robot from child: by using Myo armband in child’s arm, when child moves
his/her hand up, the robot will do the same movement. This is very beneficial for
child to learn from imitation and remembering simple movements.
3. “Do this”: A common way to teach children with Autism to having the child respond
to the adult’s prompt to “Do this”. By using EZ-Robot and saying “Do this” and
performing a simple action, child can show response to doing that act.
4. Pointing to an object: If the robot is pointing to something, it will create some
excitation in child to pay attention at that point and practice actually by copying from
the robot arm.
4.4. Eye Interaction Therapy for Autism
Eye interaction is the ability of the child to follow the look, the head position or other gestures of another person, in order to understand the subject matter of his attention.
Generally joint attention skill in children appear between about 9 and 18 months of age and impairment in these skills are among the earliest abnormalities noticed in Autism,
[39, 40 and 45]. When we are seeing or saying something to an individual with Autism, 52 he/she is not making eye contact with us and it bothers us because it may appear that they may not be listening to us. Since they have been listening in whole energy field and have been listening to whatever is happening around themselves, they cannot just listen with their ears. Even though they are not making eye contact, they are absorbing everything that is happening around themselves and so they become very much aware of their surroundings.
One of the first symptoms of children with ASD is what is referred to as disadvantages in the growth of commitment. There are many research efforts on robots and their gestures for communication in connection with gaze. Castellani and his group have developed a control method for a remote robot using user’s gaze, [46]. This research did not exactly refer to the social effectiveness of gaze but instead focused on controllability of using gaze. During this game, the child is able to make eye contact with robot more easily than making eye contacts with others. The reason why the child is able to make eye contact with a robot but he finds it difficult to make eye contact with parents or his teacher or even his peers, is because he feels very comfortable with robot as robot makes him feel very comfortable and relaxed.
4.5. Emotion Recognition Therapy for Autism
One of the factors to have a good communication with others is to recognize their emotion. Social emotional is one of the types of thing that children need to learn. To create a meaningful and responsive social interaction with other children, there are two ways to use EZ-Robot for teaching facial emotions to children with Autism. 53
1. QR Code: In this method, a picture that shows the emotion (e.g., sad, happy, angry,
..) is coded by the name of emotion to generate a QR barcode. The barcode is put
under the picture. When the robot sees the barcode, it can read it and relate to the
emotion that this code resembles. Figure 4.1 shows an example of a barcode with the
word “Happy”. When the robot reads this code, it says happy.
Happy QR Code
Figure 4.1 QR Code with “happy word” information.
2. Microsoft cognitive option: EZ-Robot can detect emotion by using Microsoft
cognitive emotion. This service has an emotion plugin so the robot can actually tell
the words, happy or sad, for example.
4.6. Dance Therapy for Autism
Children with ASD are at risk of inactivity and are less physically moving, [41].
Involvement in physical activity is often challenging and hard for children with ASD because of weakness motor skill, [47]. Dance therapy is one of the best ways to help people with Autism to raise their fullest potential. Dance movement therapy is one 54 approach that is gaining attention for its unique capability to work directly with the core deficits of autism as a form of creative arts therapy. Dance movement therapy is utilized within a therapeutic relationship with a credentialed therapist and uses the expressive elements of dance and movement as a method of assessment and intervention.
Creative movement and dance are practical and feasible options for children with
ASD [48]. Dance/movement therapy can certainly address this deep human effect of
Autism by helping parents to learn how to understand their child’s nonverbal communications and signals. Dance movement therapists can help parents and families in building warm and satisfying relationships with their children.
How to play: EZ-Robot has the potential to be programmed to dance and perform a lot of movements with head, hand and feet while at the same time playing the music. The therapist or parents can use EZ-Robot to work with an individual to make them exiting to move. They trust the robot and follow each of rhythm as much as they can. When the robot starts dancing and playing music, children can start imitation from the robot. 55
Part I Conclusions
The demonstration of gaze-communication with robot using eye contact and based on gaze-tracking experiments show that, i) the participant’s favorable feeling for the robot was more than other objects in the experiment, and ii) the feeling is enhanced when robot starts to talk or doing a social interaction with human. Various reactions of the EZ-Robot have made more effective emotional communications, and have the capability to make joint attention in order to get attention (see Figures 3.9 and 3.10).
Based on the results presented here, robots are more attractive for children and adults. They have been used for social interactions, so they have potential for using as therapeutic interventions. Such social robotic application can improve the joint attention skills because joint attention is fundamental in social communication skills of the ASD children, [49]. By recognizing human behavior of Gaze-communication with social robots through joint attention and eye contact, we can create a friendship between robot and autistic children. This can be used to mediate sharing and imitation to encourage imitative behavior, [50]. As a result, robots for Autism therapy can have different roles such as friends, toys, therapists or peers.
Robots can be a leader in social interactions as they can verbally ask a child to perform some movements and actions, [51]. Playing is an important principal for all children as part of their social learning and social development. For autistic children, this role is more highlighted, and hence, robots that are designed for therapy often can act as plaything during therapy. The advantages of using the EZ-Robot for therapeutic interventions were also presented and descried throughout this part of the dissertation.
56
Part II
Human-Machine Interaction with Application to Driver Distraction Reduction
57
Chapter 5
Background and Motivations
Driver distraction, road rage, anger, drowsiness and fatigue serve as examples of the most
common dangerous moods that could lead to serious motor vehicle road accidents and
death. Often, drivers may fail to recognize their emotional situations or moods when they
are distracted, or upset for any reason during driving. Studies show that if one can prevent negative feelings and unpleasant moods while driving, it can prevent the occurrence of many accidents. In fact, this is the main motivation for this study in which we are looking for ways to enhance the usefulness of the level of security while driving.
Driver with anger state cannot influence perceived workload and moreover induced anger can reduce driver situation awareness.
Previous research studies show that listening to music while driving is a very effective way to change driver’s mood. For this, many researchers have studied ways by which the music can change and/or influence the drivers’ mood. Inter-correlation between music tempo and change of driver’s anger is much needed. In this part of the dissertation, this inter-correlation is addressed to identify drivers at risk of distraction due to anger by physiological measuring for emotional indicators. Consequently, use of such findings is emphasized to see the effect of music on changing the mood when drivers are at risk of distraction due to anger. For physiological states measurement, heart rate, skin conductance and electromyography techniques are utilized to recognize driver’s states as well as the effect of music type on driver’s mood while driving. The driving simulation experiments show promising results as detailed and described next. 58
5.1. Introduction and Overview
According to the ABC news, America is a nation on wheels. To get where they need to go, 90% of Americans say they usually drive, reporting an average of 87 minutes a day behind the wheel. The average household owns two cars, trucks or sport utility vehicles – and one in four owns three or more. Emotions behind the wheel often make you feel independent (74%), relaxed (48%), occasionally frustrated (62%), nervous (56%) and even angry (43%). the average person drives approximately 12,000 miles per year that equates to about 33 miles a day spending this much time behind the wheel can be safe or dangerous based on several factors. When driving, driver will have to recognize how to properly handle many things such as traffic reading signs checking speed calculating, following distance whether reacting to dangerous conditions controlling the vehicle and many more. It is driver responsibility to know and obey all traffic laws and to perform the necessary actions to keep another driver and themselves safe. Driving safe is a tremendous responsibility that requires 100% focus to properly handle all the factors going on.
Distracted driving includes physical or mental action that takes driver mind off of
the road. There are some types of distraction during driving including texting, using
cellphone, eating and drinking, talking to passenger, grooming, using and reading maps,
using navigation system, adjusting the radio, CD player or MP3 player, as well as
distraction by emotions such as anger, fatigue and depression. Distracted driving is the
number one killer of the American teens. Distracted driver is 100% preventable. Getting
angry is one of the factors which distracts driver. Drivers must control their emotions as
not to endanger themselves or others on the road. 59
The most obvious symptom of anger is a reduction in driver ability to concentrate
and focus on what is going on around. It will take the driver a longer time to get the
grasp of the information which they would interpret immediately when they are fully
alert. Road rage is an intense overreaction that occurs to people driving vehicles. It can
occur to men, women and people of all ages and it could happen anywhere. Road rage
can result in someone having a really bad day but tragically on the other end of the
spectrum road rage can result in death. The fact is that road rage is a problem that occurs
all over the spectrum.
One of the reasons of road rage is the fear. When somebody is driving down the
road and someone nearly cuts the road off, fear shoots through the human, it literally
looks like this extra adrenaline cortisol going through the bodies and there is bunch of
energy that make the driver angry. Fear is one big reason why people have intense road
rage responses. Sometime road rage occurs because the driver has a tendency on the road
to take things personally especially things that are just simply not personal. When the
driver accidentally cuts someone off or makes a mistake driving down the road, it is
certainly not intentional toward another person. However, when someone does it to that
driver or just happens to be their car going front of the car all of a sudden, it becomes
personal like how dare that person disrespects me, or how dare they disrespect me and
then lash out and they kind of go hard at somebody which is where the road rage comes
in. When people get angry, they lose up to 80% of the intelligence and people get angry
on the road where they are surrounded by strangers but separated by. Since they don’t have the level of accountability inside of the car, it makes the driver more angry. 60
Loss of intelligence, loss of accountability and loss of respect and rage come with very primitive emotion that people have. Everyone likes listening to music in the car, and every year radios and CD players become more and more advanced. Many of the functions and features of these new systems are incredible. Listening to music is still under investigation for a possible driving distraction. However, there is no question that it is dangerous to be handling the systems themselves as the driver takes their eyes off the road for more than a quick glance for change the volume or track or radio station. In this research, we consider only the physiological effects of music on driver’s mood while driving.
5.2. Literature Review
Recently, a new focus by car manufacturers has been placed on driverless cars for increased safety of driver and passengers as well as roads. However, these driverless cars are still in their infancy and require human supervision. It is believed that if the supervisor/driver distractions are kept at minimum, the likelihood of accidents decreases significantly. Along this line of reasoning, the more states of supervisor/driver are known, the easier to control the driving for increased safety. The road accident statistics shows that 66% of traffic fatalities are caused by aggressive driving. These drivers attempting actions such as speeding, tailgating, running red lights, yield signs and other traffic signals, ignoring signals from other drivers. As reported by the AAA Foundation for Traffic Safety, out of over 106,000 fatal crashes, 55.7% involved during a recent four- year period driver who committed one or more aggressive driving actions. Aggressive driving was also a factor in many non-fatal accidents that caused serious injuries. 61
Driving occurs in a complex environment of pedestrians, cyclists, cars, and traffic signals, so there is a little opportunity to the driver to personally minimize frustrating events, [52, 53]. Distraction from anger is one of the ways to control or reduce intensity of anger. By diverting attention to other things, anger moment can be coped. “The function of anger is to regulate body processes related to self-defense and social behaviors”, [54, 55]. Feelings of anger classified as unhealthy emotions, are associated with increased cardiovascular reactivity and a heightened response from the sympathetic nervous system. Driving performance generally becomes better when emotion changes from very negative to positive, [56]. Emotions are relatively brief, intense, and rapidly changing reactions to potentially important events (subjective challenges or opportunities) in the external or internal environment – often of a social nature, which involve a number of subcomponents (cognitive changes, subjective feelings, expressive behavior, and action tendencies) that are more or less ‘synchronized’ during an emotional episode, [57]. “Via music, physiological arousal and driving behavior can be influenced, and thus music can be used as a tool to mediate driver state to improve road safety”, [58].
Several factors significantly affect driver responses to voice interfaces in cars, including perceived voice gender, emotion, and even age, [59]. Listening to music in many people helps them reduce their normal daily stresses. Increase in arousal measures such as SC, HR and EMG have been reported to ‘exciting’ music, as well as more specifically ‘emotional’ music, such as ‘happy’ or ‘sad’ music, [60]. Hearing the music in the car on a typical commuting can affect one’s driving. If one listens to the right music while driving, he or she could drive better and more secure. What tunes are most effective? People have different reactions with listening to music as some tend to help 62 other people, or encourage overt actions in the listener (e.g., rock concerts, music in shops), [61]. Music has effects on changes in heart rate, increased adrenaline and dopamine, eye movements and other body reactions. Each of these physical reactions can affect the behavior of drivers while driving.
Emotional experiences play an important role in increase or decrease of heart rate.
The heart and brain are connected bidirectional, that is, efferent outflow from the brain affects the heart and afferent outflow from the heart affects the brain, [62]. Residuals anger have positive correlations with residuals skin conductance level; that is, those who experience more anger also are more arousal, [63]. One of the most external negative reactions of anger is in the face by frown. The facial EMG technique can detect anger by measuring facial muscle movement. Facial muscle activity, detected as phasic facial
EMG reactions, is a general component of the emotional response, [64]. Research into the phenomenon of aggressive driving has often utilized a measure of driving anger.
These measures are commonly used to assess the factors that may influence a person to act aggressively on the road.
Music helps put our bodies in a relaxed state; it helps to slow the heart rate which lowers the blood pressure, [65]. However, there are some studies about the music that not only cannot help to relax the driver but only make driver more distracted. “Music listening makes people cry, smile, laugh, and furrow their eyebrows – as indicated by observations and electromyography (EMG) measures of facial muscles”, [66, 67].
“When a listener reports that he felt this or that emotion, he is describing the emotion which he believes the passage is supposed to indicate, not anything which he himself has experienced”. He dismisses the physiological changes in response to music, such as heart 63
rate and skin conductance, he claims that physiological changes do not correspond with
musical patterns. moreover, physiological changes, if they occur, may result simply from listeners’ beliefs about the affective power of music, [66].
5.3. Problem Statement and Motivation
Driving is a common work from an independent driver to a collaboration with other drivers. So, study about interaction between a driver and a car has more benefit in daily life. Listening to music is one of the most popular activities while driving. This is because it is an easy access to devices in the car that can help driver getting distracted.
If we prevent negative and unpleasant feelings, driving can prevent many crashes.
In fact, in this project, we are looking to find ways to increase the level of security while driving. There are two groups of research about the effect of music on driver distraction.
In one group, they have found that music makes driver more distracted, [67], while other group finds that music helps drivers to have a safe driving and possess better performance, [68]. The current study was designed to determine how music selection can have an effect on deriver emotion while driving. Anger has negative impression on driving performance, and increases risky behaviors to violate the driving rules, [69].
Driving performance will worsen when the driver has a bad emotional condition than in the neutral condition. We try to find a way to make best situation in the car in order to help the driver to have a safe driving experience.
This study focused on changing performance strategy in driving in relation to
feeling of anger. Driving simulator provides the possibility of study, and the effect of
music in highly controlled situation. This may not seem to be a long drive, but the 64 experiment shows that sign of symptoms of anger and inaccuracy of controlled vehicle appear earlier than expected. In this study, the primary objective is to assess the effect of sudden music effect on drivers when they get angry by internal or external causes. The results show that music has a significant capability to influence physiological signs and potential to reduce the impact of negative emotion effect on human while driving.
Our observations with respect to psychophysiology signal provided demonstrate the effect of varying music tempo as an explanation of music effects on behavior. Music with low to medium tempo levels reduces the sympathetic response to anger. This study has demonstrated that tempo music can mediate the effects of anger in a simulated environment and have potential to manage mood states. Through a series of extensive measurements and data collections, the proposed hypothesis that different types of music have different influences on human emotion was proved. The study further showed interactions between high, medium and low tempo music.
65
Chapter 6
Introduction to Physiological States and Human
Emotions
This chapter provides an introduction to physiological states and human emotions, relating music effects on human state of brain. It covers the physiological states of anger that could be measured and associated to, for example, heart rate, skin conductance and
Electromyography (EMG). The chapter summarizes the study by relating anger to road rage and provides remedies and suggestions to reduce such unpleasant and unwelcomed human behavior.
6.1. Music Effects on Human State of Brain
In this study, we will focus on music to see changes in the driver mood when they get angry and help them to be calm. It is focused on three types of music tempos; low, medium and high to realize the effect of each one of them. Music might have a direct effect on emotions as well, and affect mood states, [70]. Research studies show that increase in arousal would improve efficiency in easy tasks and less so in difficult tasks,
[71, 72]. Music has the power to influence human mind, thoughts emotions and behavior subconsciously music operates on a vibrational level to understand the vibrational level of the music. Human understand the state it operates through energy frequency and vibration when a song play. Emotional state tries into the recording this influences the vibration in the music and then affects. 66
Vibration has a significant effect on the subconscious mind when human is unconsciously unaware of a vibration the mind is greatly impacted far more than normal because it is vulnerable to outside vibrations. Human brain perceives and feels emotions when listening to a song. What happens in the brain when a song plays? The limbic system is composed of structures in the brain that deal with human emotions that deals with three key functions: emotions, memories and arousal. The limbic system is taught to control emotion and other brain functions related to human instincts and memories, [73].
At the peak of emotional time during a song, dopamine is released in the limbic system in the brain. This is the same neurotransmitter involved in more tangible pleasures such as food, sex and drugs. Dopamine is also released during the anticipation of these times so basically listening to music is amazing and human brain is encouraging to keep doing it through the release of dopamine, [74, 75].
Briefly, brain is wired to look for threats or rewards if one is detected; that is the
feeling region of the brain alerts us through the release of chemical messages. Emotions
are the effect of these chemical messages traveling from our brain through the body.
When human brain detects a potential threat, brain releases the stress hormones
adrenaline and cortisol which prepare human for a fight or flight response. Alternatively,
when human detects or experiences something rewarding such as someone doing
something nice, brain releases dopamine oxytocin or serotonin. These are the chemicals
that make human feel good and motivated to continue on the task or behavior. Brain is so
strong that it dominates human behaviors, and in most cases, one is unable to think
rationally in the moment of their emotions. Human thinking can influence emotions and
sometimes this can be unhelpful just thinking about something threatening can trigger an 67
emotional response. This is where we want to play powerful role in the way to manage
driver’s emotions during driving while playing music and achieve our goals.
6.2. Physiological States of Anger
Awareness of many important physiological parameters of the body, using special
sensors, can be effective in improving health. Moreover, knowing the physiological changes associated with changes in thoughts, feelings and behaviors can be very beneficial in managing physiological states of anger. Some of these parameters are: skin conduction, heart rate and electromyography.
Emotions are complex processes involving multiple response channels, [76].
Anger and heart are very connected. Incise heart rate is a common symptom for anger, which is because of the fact epinephrine and norepinephrine constrict blood vessels making heart pump harder. These two hormones also increase the amount of glucose and fatty acids in the blood. The increased levels can lead to damage in artery walls and speed up the process of atherosclerosis. When the fatty plaque builds up in the arteries, it narrows them and decreases the flow of oxygen-rich blood to the body. 68
6.2.1. Anger and Heart Rate
Heart rate variability is the space between two beats of the heart. There should be a
variation between the two, so having this space is not meant to be like a ticking clock and
always on the exact same beat at the same time. It is more of a pump that has little
variations in a healthy body as shown in Figure 6.1. The increase and decrease between
the heart beats depend on what the person is doing.
Heart rate variability is the space between two beats of the heart. There should be
a variation between the two, so having this space is not meant to be like a ticking clock
and always on the exact same beat at the same time. It is more of a pump that has little
variations in a healthy body as shown in Figure 6.1. The increase and decrease between
the heart beats depend on what the person is doing.
The average heart rate of a person is 80 beats per minute. When somebody gets
angry, the heart rate goes up and raised to about 180 beats per minute.
Figure 6.1 Increase and decrease between heart beats depending on what the person is doing in a
healthy body (https://www.medtach.com/about-ecg--hrv.html).
69
6.2.2. Anger and Skin Conductance
Skin Conductance (SC) provides information on the activity of sweat glands that is
closely associated with the activity of the sympathetic nervous system, this variable is
called SCA (skin conduction activity), EDA (electrodermal activity), or classic word
GSR (Galvanic Skin Response). In 1849, Dubois-Reymond in Germany discovered that human skin becomes a better conductor of electricity when external stimuli are presented.
This increased connectivity was related to increased activity of the sweat glands of the
skin.
The highest density of the sweat gland can be found in hands and feet. This sweat gland is more responsive to psychological stimuli on other parts of the body. There is a high correlation between sympathetic nerve activity and skin conductance responses.
The branch of the autonomic nervous system is responsible for the fight response of the
body. Emotion can raise the physiological arousal and organ’s activation, [77]. Both
skin conductance and electrodermal activity can be used as indicators of general arousal
or alertness.
Many researchers have showed a relationship between emotional reactions and
SCR, [78, 79]. Skin conductance measured by electromyography sensor attached to
respondent’s hand, it can basically pick up the levels of sweat. The filling of sweat ducts
results in many low-resistance parallel pathways. The sweat system is controlled by
sympathetic nervous system so the more emotionally aroused, the more human sweat.
For example, if someone has been on a first date their hands sweat because their emotions
are being readied up and quite emotionally charged. 70
The reason this is useful for our research is that because we need to know the emotion. As a result, by using SCR it can give us an indication of people’s emotional arousal and response to the stimuli. During our driving simulation, the connected devices response to these micro changes in sweat and it really gives an indication of participant’s experience of their emotional journey. The great thing about this experiment is that it gives us a sense of the extent of that emotional experience. However, it does not tell us necessarily what emotions often felt. Instead, it does show how emotions charged a particular respondent during that experience. Measuring the SCR is a way to perceive the intensity of personal feelings, but cannot recognize the positively or negatively by SCR alone. When the emotion goes to calm down, the tension decreases and parasympathetic activity increases.
6.2.3. Anger and Electromyography (EMG)
Electromyography (EMG) is a technique used to evaluate and record the electrical activity produced by skeletal muscles. The information is transmitted along nerves as a series of electrical discharges, carrying information in pulse repetition frequency and that is what is detected by an EMG. For this, a needle electrode is inserted through the skin on the face into the muscle. The insertional activity is measured of a contracted muscle.
Anger is one of the basic emotions with specified facial expression. 71
One of the most common indications of an anger in face is the eyebrow.
Corrugator supercilious (corrugator muscle) is a small muscle near the eye that is shaped like a small narrow pyramid (see Figure 6.2). It is also known as the frowning muscle as it controls eyebrow location on the face. The corrugator muscle activity is correlated to negative emotions including anger and also has a linear correlation with anger and stress,
[80].
Figure 6.2 Corrugator Supercilious muscle http://exploreplasticsurgery.com/tag/corrugator-muscle-resection.
Generally, anger experience automatically gives rise to the eyebrow and triggers
muscle movement. Research shows that high and low anger cores displayed different
long and short term cardiovascular responses that might reflect different behavior, [81].
The activity of this muscle can be measured with a Facial Electromyography (EMG)
sensor. The use of surface electrodes detects muscle activity that starts to contract from
the lower skeletal muscle. By using one or more active electrodes located on a specific
muscle and comparing their potential difference with the reference point specified in the
target muscle range. 72
6.3. Anger and Road Rage
Anger is an emotion characterized by a strong feeling of offence, displeasure, or hatred toward someone or something. According to Dr. Harry Mills [74], we did not born with anger emotion we learned how to become angry. Experiencing anger is one of the earliest defense mechanisms that we have. The word of emotion encompasses a broad range of feelings behavior and changes in the body and mind. There are six primary or main types of emotions; fear, joy, love, sadness, surprise and anger. Anger is an act to resist injustice, conflict humiliation, negligence or betrayal. If the anger is active, the individual attacks the target verbally or physically. If the anger is passive, then the person silently sulks and feels tension and hostility.
Angry person shows some physical signs such as teeth grinding, flushed (red and
hot), filling hot in the neck and face, trembling, headache, increased and rapid heart rates,
dizziness, and becoming hyperactive, [74]. Because of anger has physiological sign, it
can be detected by special sensor. Road rage according to definition in DMV [75] is
defined as” aggressive or violent behavior stemming from a driver’s uncontrolled anger
at the actions of another motorist”. Angry and upset driver on the road can do some
behaviors such as tailgating, speeding, cutting others off, braking, checking, and verbally
cursing other drivers.
Anger has three components; i) physical reaction that usually starts by adrenalin
rush, ii) cognitive that people think about what’s making the emotion, and iii) behavioral
action that made up the range of possible actions. Figure 6.3 shows the heart rate before
and after getting angry.
73
Figure 6.3 Heart rate before and after getting angry
(https://www.heartmath.org/research/science-of-the-heart/health-outcome-studies/).
74
Chapter 7
Effects of Music on Driver’s Anger Mood
Using the foundations established in the preceding two chapters, the effects of music on
driver’s anger mode is studied in detail in this chapter. For this, an experimental setup
consisting of a driving simulator and human subjects instrumented with physiological
measurements (e.g., facial Electromyography (EMG), Skin Conductance (SC) and Heart
Rate (HR) are utilized. The experimental setup and procedure are explained, followed by
results and analysis and discussion of the data obtained.
7.1. Problem Statement and Research Hypotheses
The experiment proposed here is designed to test the following hypothesis: What kind of tempo music will be influencing the drivers to make them calm when they are angry?
There are some studies about the music effect on emotion and mood states in long term.
Based on this, it is planned to know about the immediate effect of music to change the driver mood to avoid making dangerous decisions by the affected driver.
7.2. Technical Approach and Method
When people get angry, a range of physiological sense increases or decreases. In this study, we focus on Heart Rate (HR), Facial Electromyography (EMG) and Skin
Conductance to realize the state of anger. Through playing the music, we are trying to explore the effect of the music by measuring these three physiological states. For this, 18 75
Northeastern University graduate students were volunteered in our study. Out of these, there were 12 males (age M=23.7 years, SD=1.1 years) as well as six females (age
M=24.3 years, SD=2.0 years) but the gender was not an important factor in this experiment. Music change strategy tempo was administered as a within-subject factor.
Our study is based on the underlying principle that different emotions stimulate the autonomic nervous system and subsequently physiological changes. This is really important because in most cases it is not clear what activity the autonomic nervous system reflects or relate to other activities. The nervous system builds up automatically,
[82]. Also, emotional signals are affected by many factors some of them are internal factors and some include a wide range of stimuli and outputs. That is why they are inherently noise-free and this is a problem that other conditions get worsen.
7.3. Music Selection
Human emotions can be greatly influenced by music. The limbic system that is involved in our emotion in brain is very powerful to change how we think; that is, music can help us feel happy by generating serotonin. “Something people already listen to and identify with, as a way to help people heal. More specifically, we would like to know how music can be used to help those who have difficulty with their anger”, [83]. Imagine humanity without music is very hard to all cultures. Music is important part of the human construction. Music studies have shown that music affects the heart rate, adrenaline hormone and dopamine, eye movements, and other body reactions. Each of these physical responses can affect the behavior of drivers while driving. 76
Hakan Lidbo, a Swedish music producer, produced a music album for car drivers.
He has made special musical pieces that include 7 singles. Each of these singles is 80 beats per minute and only a bit faster than the heart rate. This music will be broadcast at a wavelength of about 70 dB, thus blurring the noise of the car's wheels by asphalt. Each single-piece design can overcome negative emotions in different situations while driving.
Hakan says that the purpose of these musical pieces is not to concentrate and attract the attention of the driver from driving and music, instead the motive is to keep the driver alert and to keep him from driving when driving because of different emotions, intelligence and loss of senses.
Music and anger, along with many other emotions, have a high correlation. It is a proven fact that anger is caused or subdued by the music. In this study, songs are chosen from the Soul by Ilayaraja from the Nothing but Wind Album. Three significant music characteristics have been examined in this experiment, low and medium and high tempo.
Tempo is the speed of beat usually in beats per minute. There are 3 music stages combined with different tempo with same loudness. The order of the 3 combinations of music type change strategies was counterbalanced over the participants. Tempos were changed using a smartphone application, the “Tempo Slow MO Original” by developer
Martian Storm LTD. As tempo is measured according to beats per minute (BPM), the
BVP is adjusted as 150% of the original song’s BVP as high tempo, keep the original song’s BVP as same to work as median tempo and adjust the BVP as 50% of the original ones’ BVP as low tempo. From the Car Interior Noise Decibel Database, the interior noise while driving is roughly 60 decibels (dB) when assumed that the speed is between
10 to 40 mph. Therefore, we set up the levels of music loudness based on car interior 77
noise decibel and considering the comfort zone of people’s auditory sensation area. The
music level was 60 dB which is close to the interior noise.
7.4. Driving Simulation
This section discusses our driving simulators to investigate driver behavior and
psychophysiological responses. Driving simulator is a great tool to study driver behavior
and driver safety, all the experiment information is useful because it offers a completely controlled experiment or can actually stage anything we want to see. In the simulated environment, the driving simulator can provide a lot of opportunities for doing better experiment.
This study employs a Northeastern University driving simulator with fixed-base vehicle mock-up and functional steering wheel, indicators, break and accelerate pedals
(see Figures 7.1 and 7.2). The speed limit between 10 to 40 mph, road including red lights, stop signs, pedestrians and simulating other cars that cut the driver way are used to build the driving situation for participants feeling in the driving condition. This driving simulation simulates different driving conditions: bad drivers such as overtaking, making collisions, blocking the way at extreme low speed, traffic jam or huge roadblock.
These three conditions are most common situations that lead to road rage of
drivers. Each participant had the same initial conditions with the vehicle pulling out
based on the participant’s current speed and distance from the obstacle. The traffic light
switched from green to yellow at a simulated distance of 1000 feet away and stayed that
color for 2 seconds before turning red. I n other words, the initial time-to-contact for
pedestrians was variable based on the driver’s speed while the initial time-to-contact for 78
cars was invariable as the road rage is difficult to trigger in our laboratory condition. We
use a video to induce the anger emotion of the participants as a means of simulating the
road rage. With big screen projector for simulation, the participants were seated in the
car chair and the physiological sensors were attached to measure their heart rate, skin
conductance and facial electromyography.
Figure 7.1. Northeastern University Driving Simulator with participant during driving.
7.5. Physiological Measurements
Physiological measurements were done continuously during the experiment using Nexus-16 equipment connected to measure facial electromyography, skin conductance and heart rate (see Figure 7.2).
79
Figure 7.2 Connections of different sensors.
7.5.1. Facial Electromyography (EMG) Measurements
For recording EMG signal, there are two kinds of electrodes; surface EMG and intramuscular EMG. In this experiment, surface EMG was used. For this, the assessment of muscle function was used by recording muscle activity from the surface above the muscle on the skin. EMG activity was continually recorded from specific muscles above the eyebrow. The electrodes were placed using the guidelines of Fridlund and Izard, [84].
Electrodes’ specification used to in the experiment are as follows: Type H124SG, Ref
31.1245.21, Ag/AgCI (silver chloride) sensor with diameter of 24 mm, Latex free,
Patented gel formula, AAMI standards, and short-term monitoring system. The H124SG has a unique, patented pre-gelled adhesive side with non-irritating gel, especially developed to prevent allergic reactions. The Snap-On connector can easily be pushed on or removed from the electrode lead. 80
Signals between 90 and 250 Hz were used and signals below 90 Hz and above
250 Hz were filtered out and canceled. The impedance of electrode was lower than 5 kΩ.
The electrode was placed by three sensors above the eyebrow. Before the electrodes
were attached, the skin was cleaned and slightly rubbed in order to better record the
signal. Anger emotions can change EMG signal and consequently recording of this
signal can be used to analyze the results.
7.5.2. Skin Conductance (SC) Measurement
SC is a nonlinear and non-stationary measurement. In this study, nonlinear dynamical
methods and complexity measures were used to analyze the data. The skin conductance
signal is always short. For this and in order to get better observation of angry emotion,
we used short-term SCR analysis. The sensors were connected to two fingers or two parts of the palm to provide feedback in a variety of ways. In this experiment, we connected the sensor to two fingers with the following sensor’s specifications: Size
(approx.) 37 mm by 37 mm by 12 mm; Input Impedance: 10¹² Ω in parallel with 10 pF;
Operating Input Bias: ~1.0 to 2.0 V above sensor ground; Signal Input Range: ± 40 mV;
Channel Bandwidth: 0.05 Hz to 1 kHz; Signal Output Range: ± 2.0 V (+ 2.8 V if used
with sensor isolator); Input/Output Gain: 50; Supply Voltage: 7.26 V (±0.05 V); Current
Consumption: < 1.5 mA; Accuracy: ± 5%, [85].
Music can reduce skin conductance levels during low energy music. Data were
sampled at 5 Hz rate. Using a Continuous Decomposition analysis, the galvanic skin
response or skin conductance, as an electro dermal response, was measured. The skin
becomes a better conductor of electricity due to external stimulation between skin 81 conductance and anger. For this, skin conductance measurements were taken by using finger electrodes, which were attached to finger by tape and the electrodes to both the index and middle fingers.
7.5.3. Heart Rate (HR) Measurement
HR is the number of heart beats per unit of time. Heart Rate Variability (HRV) is a specialized technology developed and used in cardiovascular pulse caused by the compression of the heart, [86]. To send blood around the body, heart rate variability are variables that have significant correlation with defensive reactions [87].
Heart rate is directly related to the cardiovascular system. The signal was measured by a finger-clip. Heart Rate Sensor was placed on pointer finger. Sensor
Specifications include; Voltage: 5V; Control Mode: IIC; Operating Temperature: -20 ~
+60°C. Heart rate data can be useful for monitoring the anger. When someone gets angry, stress mechanism is activated in his/her body followed by many biological changes. Heart rates have significant and positive correlations with excitement of emotions.
7.6. Experimental Setup and Procedure
During the session in the lab, participants were seated behind the wheel for driving simulation at Northeastern University lab and drive with a speed limit between 10-40 mph. Simulated road includes cars, stop signs, intersections and pedestrians. During simulation, subjects were connected to the physiological sensors, hear rate (HR), skin 82
conductance (SC), facial electromyography (EMG). All these sensors were captured with
a sampling rate of 256 Hz using a Biograph infinity software. For HR measurements,
electrodes were taped to pointer finger. For SC measurements, 2 electrodes were
connected to the fingers, and finally for EMG, 3 electrodes were connected to the face above the eyebrow.
We asked participants to sit down while the experimental procedure was
explained to them. It was made sure that they are relaxed and sit in a comfortable
fashion. Emotion such as anger is usually elicited through watching movies, [88]. In
order to evoke anger, we asked the subjects to watch various clips geared to elicit anger
(e.g., one clip was about hitting children in street,
www.youtube.com/watch?v=KWpd1Wy1tZQ). The subjects would be prompted to
answer questions about their anger mood while watching the scene. Moreover, during
the driving simulations, the drivers were suffered by someone cutting their road via
another car or pedestrians to elicit anger. The data was collected every 60 seconds for
bassline and then the participants were required to focus on driving and listening to a
selected high-level music for about 130 seconds.
They were then stopped and watched the second video and started driving while
required to listen to a medium tempo music for about 130 seconds. This process was
repeated for low tempo music. Biograph software was used to collect the physiological
signals from the participants. As mentioned in the music selection part, the music stage
changed after every stop and new video was played at the start of driving. Thus, there
were a total of 3 type music stages. For reliability, the sequence of the three videos and
three music change strategies were played randomly (see Figure 7.3). For validity 83
between two trials of music change, the parameters of music characteristic only changed
once.
Bassline Stage 2 Stage 1 Stage 3 & anger Medium tempo Low temp High tempo inducement
Figure 7.3 The sequence of steps of experiment in driving simulation.
7.7. Results and Data Analysis
The aim of the analysis is to test our hypothesis. First, subjects who were getting angry during watching videos and during simulation will be utilized to do future analysis.
Then, the values of physiological signals during driving and playing music are calculated and compared to determine the rejection or acceptance of the hypothesis. There are 3 types of music. The mean values of each signal during different time when playing different types of music will be calculated. This step is to test the simultaneity of three physiological sensors. Physiological data that include HR, EMG and SC for each subject in each condition were entered separate measures of ANOVA (see Figure 7.4).
84
Music Effect: Music was started at the beginning of each test run with music and
continued until the end of the simulation. During driving simulation, we used 3 levels of
tempo music with different beats as shown in Table 7.1. As tempo is measured according to beats per minutes, we selected the music by the favor of subjects but in this range of value and speed.
Table 7.1 Different values and speed of music per minute for different scenarios.
Music Value Beats per minute (bpm)
High Tempo <70 bmp± 85 dB
Medium Tempo 85-100 bpm at ± 85 dB
Low Tempo >120 bpm at ± 85 dB
Figure 7.4 Screenshot of Tempo SlowMo Original App. 85
Results and Analysis of Facial Electromyography: EMG amplitudes and effective
response magnitudes not only strongly vary between individuals because of differences in affective processes but also due to anatomical and biophysical differences, [89]. We selected muscle activity as it changes significantly between the presentation of the natural
state and the anger state. Hence, by comparing simple effect of tempo it can be seen
when tempo changes between high and medium and low, there are significant differences
in EMG readings.
The results shown in Figures 7.5 to 7.7 compare the EMG, skin-conductance and
heart rate in three levels of music tempo to demonstrate the significant differences. The
ANOVA results show the signals for Facial Electromyography, skin-conductance and
heart rate. There are significant differences between three factors. Also, there is a
significant interaction between them as seen from the graphs and discussed in the text
next.
A B
Figure 7.5 (A) Interval plots for comparing 3 types of music and distribution of samples by EMG; and (B) Probability plot for 3 types of music by EMG signal. 86
Table 7.2 Descriptive statistics of EMG indicators between three types of music.
Music Mean stDev P
High 11.11 1.96 <0.005
Medium 10.47 1.28 <0.005
Low 10.16 0.94 <0.005
By using these normalized EMG data for the corrugator activity, lower valence rating during the low tempo is attained when compared to the high tempo music as shown in Table 7.2; that is, high tempo (M=11.10, SD=1.96, σ2=3.874), Medium tempo
(M=10.46, SD=1.28, σ2=1.652), Low tempo (M=10.16, SD=0.94, σ2 =0.9). We can see that there is a significant difference in EMG activity between high tempo and low tempo music. This result was tested through a repeated measure using ANOVA on EMG as within-subject factor. As results show, there was no significant difference between low tempo and medium tempo.
Skin Conductance Level: The skin conductance level was measured continuously before
inducing anger and throughout the simulation in 3 levels of music tempo. The data were
corrected for the baseline measurement. Most of music types have significant differences except one; that is the condition when tempo is high does not have obvious effect (see
Figure 7.6).
There is a positive correlation between skin conductance and anger. Result show that emotional events induce skin conductance responses varying according to arousal.
When someone gets angry because of sweat, the skin conductance valance goes up.
Based on this, we extracted the SCL value during the driving and playing music with the 87
following results; high tempo (M=8.15, SD=0.28, σ2=0.08), medium tempo (M=7.31,
SD=0.25, σ2=0.06), low tempo (M=7.02, SD=0.23, σ2=0.06). This shows significant differences between these 2 factors but shows no differences between the low tempo and medium tempo.
A B
Figure 7.6 (A) Interval plots for comparing 3 types of music and distribution of samples by SC signal, and (B) Probability plot for 3 types of music by SC signal.
Table 7.3 Descriptive statistics of SC indicators between three types of music.
Music Mean stDev P
High 8.15 0.28 <0.005
Medium 7.31 0.25 <0.005
Low 7.02 0.23 <0.005
88
Heart Rate: Heart rate signals were captured continually during the driving simulation.
The data from systolic HR were obtained during the simulation. The data were collected
from the baseline to measure after the induction and during the driving. Positive reaction
increased heart rate during the driving. These reaction scores were collected for each participant subjected to ANOVA analyses (see Figure 7.7).
A B Figure 7.7 (A) Interval plots for comparing 3 types of music and distribution of samples by HR signal, and (B) Probability plot for 3 types of music by HR signal.
Table 7.4 Descriptive statistics of HR indicators between three types of music.
Music Mean stDev P
High 8.15 0.28 <0.005
Medium 7.31 0.25 <0.005
Low 7.02 0.23 <0.005
There is positive correlation between heart rate and anger when anger goes up,
heart rate goes up as well. For the effect of music on heart rate during driving with anger 89
mood, there is a small difference between medium tempo and low and high tempo with
almost no significant different between high tempo and low tempo; that is, high tempo
(M=34.98, SD=0.62, σ2=0.39), medium tempo (M=34.85, SD=0.61, σ2=0.38), and low
tempo (M=34.97, SD=0.60, σ2=0.36).
7.8. Summary and Discussions
Generally, anger creates increased activities on cardiovascular system, skin conductance
and muscle interaction. We found that the music group with low tempo has performed
better than the high tempo group in the same critical situations. These so-called negative
mood states cause driver to do undesirable reactions that may consequently lead to
dangerous acts while driving. The study showed that music is calming and keeping the
driver focused on driving. In fact, this kind of music adds to the concentration of most
drivers. Listening to music or radio is a common activity for most drivers especially
when they are waiting behind a red light, heavy traffic and or driving for a long distance.
High tempo music has the potential to increase physiological arousal, [90-95].
Recent studies show that about 76% of drivers are distracted by happy music as
they are trying to follow the melody. Interestingly, many drivers get distracted with sad
music excerpts. The results also demonstrate that these musical effects are dependent on
the emotional valence of the driver, [96]. It has been shown that the level of physiological sensation is higher during high-energy negative moods such as anger where no music can have positive health benefits in the short run. However, the music can help mediate the state of anger in the long run. It is a proven fact that music generally has the 90 power to challenge cognitive and emotional behaviors so it will be important as how and when to use it during driving.
Driving is a multiple-task act in the sense that physical, attentional, and emotional feelings must function at the same time [52]. For example, if a driver gets negative feedback, it could well adversely affect one of these 3 sensations and consequently dangerous to operate a vehicle in the road. Past research show that drivers with anger exhibit more violation and cursing; and thereby, reduce anger will greatly reduce the violation [97]. The measure of combined HR, SC and EMG is one of the most common and effective approach to recognize the anger. By increasing HR and SC and change of
EMG from the bassline state, one could recognize the anger easily. Heart Rate (HR) increases more in response to simulative music that aroused feelings of vigor and tension than to sedative music [98].
A quick review of the relevant literature shows that it is very difficult to correlate how music can reduce or alter emotional statuses, [99]. However, the important conclusion from this study is that the same music will not be effective for all drivers even if they are in the same age. Contrary to this, for low and medium tempo music performance, anger drivers have a benefit to be calm while driving. The suggestion is to choose the music to match the driver characteristics and verify their emotions before testing as for real-life implementation. 91
Part II Conclusions
In this part of dissertation, our primary objective was to assess the effect of sudden music effect on driver when they get angry by internal or external causes. The results show that music has capability to influence physiological signs and potential to reduce the impact of negative emotion effect on human while driving in angry mood. Our observation with respect to psychophysiology signals provided show that tempo music has an explanation for music effects on behavior. Music with low to medium tempo levels reduces the sympathetic response to anger. The study has demonstrated that tempo music can mediate the effects of anger in a simulated environment and has the potential to manage mood states. The results for Electromyography and skin conductance signals show that there are significant differences between high tempo and low tempo music but no significant differences were observed when low tempo music was used.
92
References
[1] Levy, Susan E, Mandell, David S, Merhar, Stephanie, Ittenbach, Richard F, Pinto-
Martin, Jennifer A. (2003). “Use of complementary and alternative medicine among
children recently diagnosed with autistic spectrum disorder.” Journal of
Developmental & Behavioral Pediatrics, 24(6): 418-423.
[2] National Research Council. “Educating Children with Autism.” Washington, DC:
National Academy Press, 2001.
[3] Gupta, V. B. (2004). “New York Medical College and Columbia University”., New
York, New York, USA. “Autistic Spectrum Disorders in Children, 193.
[4] Carolyn R. Garver, PhD., Director of Dallas Program. Health Studies; licensed
Child Care Administrator, National/International speaker on autism.
[5] A. M. Turing (1950) “Computing Machinery and Intelligence.” Mind 49: 433-460.
[6] Powell, Walter W, Koput, Kenneth W, Smith-Doerr, Laurel (1996).
“Interorganizational collaboration and the locus of innovation: Networks of learning
in biotechnology.” Administrative science quarterly: 116-145.
[7] Moore, David, Cheng, Yufang, McGrath, Paul Powell, Norman J. (2005).
"Collaborative virtual environment technology for people with autism." Focus on
autism and other developmental disabilities 20(4): 231-243.
[8] De Silva, Ravindra S, Tadano, Katsunori, Higashi, Mastake, Saito, Azusa,
Lambacher, Stephen G (2009). “Therapeutic-assisted robot for children with
autism.” Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International
Conference on, IEEE. 93
[9] Hershkowitz, V. (1997). “How adults with autism utilized their computers.”
Advocate-Newsletter of the Autism Society of America, Inc.
[10] Pierno, A. C., et al. (2008). “Robotic movement elicits visuomotor priming in
children with autism.” Neuropsychologia 46(2): 448-454.
[11] Conti, D. (2016). "Robotics and intellectual disabilities: models and treatment."
[12] Ingersoll, B. and Gergans, S. (2007). “The effect of a parent-implemented
naturalistic imitation intervention on spontaneous imitation skills in young children
with autism.” Research in Developmental Disabilities, 28, 163-175.
[13] Bird, Geoffrey. Leighton, Jane. Press, Clare. Heyes, Cecilia. (2007). “Intact
automatic imitation of human and robot actions in autism spectrum disorders.”
Proceedings of the Royal Society of London B: Biological Sciences 274(1628):
3027-3031.
[14] Kim, Elizabeth S. Berkovits, Lauren D. Bernier, Emily P. Leyzberg, Dan. Shic,
Frederick. Paul, Rhea. Scassellati, Brian. (2013). "Social robots as embedded
reinforcers of social behavior in children with autism." Journal of autism and
developmental disorders 43(5): 1038-1049.
[15] Scassellati, B. M. (2001).” Foundations for a Theory of Mind for a Humanoid
Robot.” Massachusetts Institute of Technology.
[16] Scassellati, B., et al. (2012). "Robots for use in autism research." Annual review of
biomedical engineering 14: 275-294.
[17] Ali Mashhadi, Fatemeh Mirdoraghi, Batol Bahrami, Reza Soltani Shal (2012)
“Effect of Demographic Variables on Children Anxiety.” Iranian J Psychiatry 7:3. 94
[18] Yano, K. and T. Lanusosang (2013). "Globalisation and its impact on agriculture:
an overview of Kohima District, Nagaland, India." International journal of Bio-
resource and Stress Management 4(4): 651-654.
[19] Werry, I., et al. (2001). “Investigating a robot as a therapy partner for children with
autism.” Procs AAATE 2001.
[20] Chattoe, E., C. Conati and B. Cooper (2006). “Socially Intelligent Agents: Creating
Relationships with Computers and Robots.”, 3.
[21] http://www.autism-society.org/what-is/, accessed 1.8.2018.
[22] Kennar, An Austrian-American psychiatrist, physician, and social activist best
known for his work related to autism (June 13, 1894 – April 3, 1981).
[23] Baron-Cohen, S. Mind blindness (1995). “An essay on autism and theory of mind.”
Cambridge, MA: MIT Press/Bradford Books
[24] Association, A. P. Washington, DC: American Psychiatric Association; 2000.
“Diagnostic and Statistical Manual of Mental Disorders.” Text Revision (DSM-IV-
TR).
[25] Billard, A. and K. Dautenhahn (1997). “Grounding communication in situated,
social robots”. Proceedings Towards Intelligent Mobile Robots Conference, Report
No. UMCS-97-9-1, Department of Computer Science, Manchester University.
[26] Dautenhahn, K. (1999). “Robots as social actors: Aurora and the case of autism.”
Proc. CT99, The Third International Cognitive Technology Conference, August,
San Francisco. 95
[27] Breazael, C. and B. Scassellati (2006). “Infant-Like Social Interactions Between a
Robot and a Human Caregiver.” Massachusetts Institute of Technology, Cambridge
Artificial Intelligence Lab.
[28] Fong, T., et al. (2003). "A survey of socially interactive robots." Robotics and
autonomous systems 42(3-4): 143-166.
[29] http://rtc.nagoya.riken.jp/RIBA/index-e.html, accessed 1.8.2018
[30] http://www.parorobots.com/, accessed 1.8.2018.
[31] https://www.maxonmotor.com/maxon/view/application/Romeo-A-helpful-friend-
for-the-future, accessed 2.8.2018
[32] Kordt-Thomas, C. and I. M. Lee (2006). "Floor time: rethinking play in the
classroom." YC Young Children 61(3): 86.
[33] Kazdin, A. E. (1977). "Assessing the clinical or applied importance of behavior
change through social validation." Behavior modification 1(4): 427-452.
[34] Ingersoll, B., et al. (2003). "Effect of sensory feedback on immediate object
imitation in children with autism." Journal of autism and developmental disorders
33(6): 673-683.
[35] http://www.herts.ac.uk/kaspar, accessed 2.8.2018
[36] https://www.ald.softbankrobotics.com/en/robots/nao, accessed 2.8.2018
[37] https://leka.io/, accessed 2.8.2018
[38] https://www.ez-robot.com/, accessed 2.8.2018 96
[39] Charman, T. (2003). “Why is joint attention a pivotal skill in autism?”
Philosophical Transactions of the Royal Society B: Biological Sciences 358(1430):
315-324.
[40] López, B. and S. R. Leekam (2003). “Do children with autism fail to process
information in context?.” Journal of child psychology and psychiatry 44(2): 285-
300.
[41] Pan, Chien-Yu, Frey, Georgia C, 2006, “Physical activity patterns in youth with
autism spectrum disorders.”, Journal of autism and developmental disorders, page
597.
[42] https://www.myo.com/, accessed 2.8.2018.
[43] T. Duchowski, “Theory and Practice.” Third Edition, Chapter 5 (19)
[44] https://www.tobiipro.com/, accessed 3.8.2018
[45] Siller, M. and M. Sigman (2002). “The behaviors of parents of children with autism
predict the subsequent development of their children's communication.” Journal of
autism and developmental disorders 32(2): 77-89.
[46] Galvan, S., et al. (2006). “Innovative robotics teaching using lego sets. Robotics
and Automation.” ICRA 2006. Proceedings 2006 IEEE International Conference
on, IEEE.
[47] Koegel, R. L., et al. (2001). "Pivotal areas in intervention for autism." Journal of
clinical child psychology 30(1): 19-32. 97
[48] Behrends, A., et al. (2012). “Moving in and out of synchrony: A concept for a new
intervention fostering empathy through interactional movement and dance.” The
Arts in Psychotherapy 39(2): 107-116.
[49] Kasari, C., et al. (2010). “Randomized controlled caregiver mediated joint
engagement intervention for toddlers with autism.” Journal of autism and
developmental disorders 40(9): 1045-1056.
[50] Contaldo, A., et al. (2016). “The social effect of “being imitated in children with
autism spectrum disorder.” Frontiers in psychology 7: 726.
[51] Michaud, F., et al. (2005). “Autonomous spherical mobile robot for child-
development studies.” IEEE Transactions on Systems, Man, and Cybernetics-Part
A: Systems and Humans 35(4): 471-480.
[52] Harris, H. and C. Nass. (2011) “Emotion regulation for frustrating driving contexts.
“in Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems. ACM.
[53] Nass, C., et al. (2005) “Improving automotive safety by pairing driver emotion and
car voice emotion.” in CHI'05 Extended Abstracts on Human Factors in Computing
Systems. ACM.
[54] van der Zwaag, M.D., et al. (2011) “The impact of music on affect during anger
inducing drives.” in International Conference on Affective Computing and
Intelligent Interaction. Springer.
[55] Lemerise, E.A. and K.A. Dodge, (2008), “The development of anger and hostile
interactions. Handbook of emotions.”, 3: p. 730-741. 98
[56] Cai, H. and Y. Lin, (2011)” Modeling of operators' emotion and task performance
in a virtual driving environment.” International Journal of Human-Computer
Studies, 69(9): p. 571-586
[57] Juslin, P.N. and J. Sloboda, (2011) "Handbook of music and emotion." Theory,
research, applications.: Oxford University Press.
[58] van der Zwaag, M.D., et al, (2013) Using music to change mood while driving.
Ergonomics, 56(10): p. 1504-1514.
[59] Takayama, L. and C. Nass, (2008) Driver safety and information from afar: An
experimental driving simulator study of wireless vs. in-car information services.
International Journal of Human-Computer Studies, 66(3): p. 173-184.
[60] Rickard, N.S., (2004) Intense emotional responses to music: a test of the
physiological arousal hypothesis. Psychology of music, 32(4): p. 371-388.
[61] Fried, R. and L. Berkowitz, (1979) Music Hath Charms… And Can Influence
Helpfulness1. Journal of Applied Social Psychology, 9(3): p. 199-208.
[62] Thayer, J.F., et al., A meta-analysis of heart rate variability and neuroimaging
studies: implications for heart rate variability as a marker of stress and health.
Neuroscience & Biobehavioral Reviews, 2012. 36(2): p. 747-756.
[63] Zinner, L.R., et al., (2008) Anger and asymmetrical frontal cortical activity:
Evidence for an anger–withdrawal relationship. Cognition and Emotion, 22(6): p.
1081-1093.
[64] Dimberg, U., (1990) Facial electromyography and emotional reactions.
Psychophysiology,. 99
[65] Meyer, K., et al., (1956) The acid mucopolysaccharides of connective tissue.
Biochimica et biophysica acta, 21(3): p. 506-518.
[66] Krumhansl, C.L., (1997) An exploratory study of musical emotions and
psychophysiology. Canadian Journal of Experimental Psychology/Revue
canadienne de psychologie expérimentale, 51(4): p. 336.
[67] Brodsky, W. and Z. Slor (2013). "Background music as a risk factor for distraction
among young-novice drivers." Accident Analysis & Prevention 59: 382-393.
[68] Ünal, A. B., et al. (2013). "Driving with music: Effects on arousal and
performance." Transportation research part F: traffic psychology and behaviour 21:
52-65.
[69] Deffenbacher, J. L., et al. (2003). "Anger, aggression, and risky behavior: a
comparison of high and low anger drivers." Behaviour research and therapy 41(6):
701-718.
[70] Juslin, P. N. (2000). "Cue utilization in communication of emotion in music
performance: Relating performance to perception." Journal of Experimental
Psychology: Human perception and performance 26(6): 1797.
[71] McGrath, J. (1963). Irrelevant stimulation and vigilance performance. Vigilance: A
Symposium. New York: McGraw-Hill.
[72] Beh, H. C. and R. Hirst (1999). "Performance on driving-related tasks during
music." Ergonomics 42(8): 1087-1098.
[73] (https://study.com/academy/lesson/what-is-the-limbic-system-in-the-brain-
definition-functions-parts.html 100
[74] Mills, H. (2005). Health cost of anger. Retrieved from
http://www.mentalhelp.net/poc/view_doc.php?type=doc&id=5809&cn=116
[75] (https://www.dmv.org/how-to-guides/road-rage.php)
[76] Barrett LF, Bliss-Moreau E (2009) Affect as a psychological primitive. Adv Exp
Soc Psychol 41: 167–218.
[77] Yang, Z. and G. Liu (2014). "An Entropy Measure of Emotional Arousal via Skin
Conductance Response." Journal of Fiber Bioengineering and Informatics 7(1): 67-
80.
[78] Edelberg, R. (1972). "Electrical activity of the skin: Its measurement and uses in
psychophysiology." Handbook of psychophysiology 12: 1011.
[79] Lanat`a A, Valenza G, Scilingo EP., (2012) A novel eda glove based on textile-
integrated electrodes for affective computing. Medical & biological engineering &
computing: 50(11); 1163-1172.
[80] S. D. Kreibig, Autonomic nervous system activity in emotion: A review, Biological
psychology, Vol. 84, No. 3, pp. 394-421, 2010.)
[81] Dykens, E. M., et al. (1992). "Adaptive and maladaptive behavior in Prader-Willi
syndrome." Journal of the American Academy of Child & Adolescent Psychiatry
31(6): 1131-1136.
[82] Kreibig, Autonomic nervous system activity in emotion: A review, Biological
psychology, Vol. 84, No. 3, pp. 394-421, 2010. [8] R. W. Picard, E. Vyzas, J.
Healey, Toward machine emotional intelligence: Analysis of affective physiological
state, Pattern Analysis and Machine Intelligence, IEEE Transaction 101
[83] Hakvoort, L., et al., (2015) Influence of music therapy on coping skills and anger
management in forensic psychiatric patients: an exploratory study. International
journal of offender therapy and comparative criminology, 59(8): p. 810-836.
[84] Fridlund, A. and C. E. Izard (1983). "Electromyographic studies of facial
expressions of emotions and patterns of emotions." Social psychophysiology: A
sourcebook: 243-286.
[85] http://www.thoughttechnology.com/sciencedivision/
[86] N Chirakanphaisarn, T Thongkanluang,(2016) Heart rate measurement and
electrical pulse signal analysis for subjects span of 20–80 years, Journal of
Electrical Systems and Information Technology.
[87] Picard, W. and J.A. Healey, (2000) Wearable and automotive systems for affect
recognition from physiology.
[88] Lisetti, C.L. and F. Nasoz, (2004) Using noninvasive wearable computers to
recognize human emotions from physiological signals. EURASIP Journal on
Advances in Signal Processing, p. 929414.
[89] Sahli, H., "Objectifying Human Facial Expressions for Clinical Applications", MS
Thesis.
[90] BH Dalton, DG.Behm, A. Kibele,(2007) Effects of sound types and volumes on
simulated driving, vigilance tasks and heart rate, Occupational Ergonomics.
[91] Husain, G., et al. (2002). "Effects of musical tempo and mode on arousal, mood,
and spatial abilities." Music Perception: An Interdisciplinary Journal 20(2): 151-
171. 102
[92] Adrian C. North , David J. Hargreaves ,(2008) Music and driving game
performance First published: 9 October https://doi.org/10.1111/1467-9450.404128
[93] Konecni, V. J., and Sargent-Pollock, D. (1976). Choice between melodies differing
in complexity under divided-attention conditions. journal of Experimental
Psychology: Human Perception and Performance, 2(3), 347-356. -1523.2.3.347
[94] Craig W. Fontaine, Norman D. Schwalm ,(1979) Effects of Familiarity of Music on
Vigilant Performance First Published.
[95] McNamara, Linda; Ballard, Mary E.(1999)Genetic, Social, and General Psychology
Monographs; Resting arousal, sensation seeking, and music preference Washington
Vol. 125, Iss. 3.
[96] Pêcher, C., C. Lemercier, and J.-M. Cellier,(2009) Emotions drive attention: Effects
on driver’s behaviour. Safety Science, 47(9): p. 1254-1259.
[97] Biassoni, F., et al. (2016). "Hot or cold anger? Verbal and vocal expression of anger
while driving in a simulated anger-provoking scenario." SAGE Open 6(3):
2158244016658084.
[98] Iwanaga, M. and Y. Moroki (1999). "Subjective and physiological responses to
music stimuli controlled over activity and preference." Journal of Music Therapy
36(1): 26-38.
[99] Scherer, K. R. and M. R. Zentner (2001). "Emotional effects of music: Production
rules." Music and emotion: Theory and research: 361-392.