Emotional Design

• D. Norman (Emotional Design, 2004) • Model with three levels – Visceral (lowest level) – Behavioral (middle level) – Reflective (top level) Emotional Intelligence (EI)

• IQ is not the only indicator of intelligence (Emotional Intelligence book by Daniel Goleman, 1995) • EI: Awareness and ability to manage one’s emotions in a healthy manner. • EI: Ability to sense, perceive, understand, and assess own and other people’s emotions

Is Mr. Spock intelligent?

• Spock is only rational

• Descarte’s Error (Damasio, 1994)

• Artificial intelligence searches unlimited search space to make a rational decision

• Missing ‘somatic markers’ that associate feelings with decisions

Damasio’s Somatic Marker Hypothesis

• Originated from the observation of individuals who had sustained damage to the ventromedial prefrontal cortex. • Normal intellectual function • Normal Neuropsychological function • Normal on tests sensitive to frontal lobe function • However, severe impairment in personal and social decision making and conduct. – Difficulty with planning in the immediate, and future. – No longer able to make personally advantageous decisions – Often sustain social, personal, economic losses • The only deficit that could be detected was one in which these individuals failed to display emotion in situations in which emotion would be normatively expected.

• This led Damasio to posit that these individuals manifest a deficit in reasoning that is secondary to deficits in emotional processing. What is Affect?

• The type and degree of emotion a person displays • The experienced, subjective, and conscious aspect of feeling or emotion – Positive – Negative – Neutral

Affect Theory

• Developed by Silvan S. Tomkins in 1962 • Tomkins book Affect Imagery (3 vols.) • Believed that the affect system is the motivating force in human life. • Organized affect into 3 main categories: – Positive, negative, and neutral – Each has a low/high intensity label Tomkins nine affects

• Positive: • Enjoyment/Joy - smiling, lips wide and out • Interest/Excitement - eyebrows down, eyes tracking, eyes looking, closer listening • Neutral: • Surprise/Startle - eyebrows up, eyes blinking • Negative: • Anger/Rage - frowning, a clenched jaw, a red face • Disgust - the lower lip raised and protruded, head forward and down • Dissmell (reaction to bad smell) - upper lip raised, head pulled back • Distress/Anguish - crying, rhythmic sobbing, arched eyebrows, mouth lowered • Fear/Terror - a frozen stare, a pale face, coldness, sweat, erect hair • Shame/Humiliation - eyes lowered, the head down and averted, blushing

Affective Computing

• Computing that relates to, arises from, or deliberately influences emotions

• Coined by Rosalind Picard – Founder and director of the Affective Computing Research Group at the MIT Media Lab. – Her book, Affective Computing (1997) lays the groundwork for giving machines the skills of emotional intelligence.

Affective computing is related to other computing disciplines:

Artificial Intelligence (AI), Virtual Reality (VR) and Human Computer interaction (HCI).

Questions that need to be answered:

What is an affective state (typically feelings, moods, sentiments etc.)?

Which human communicative signals convey information about affective state?

How various kinds of affective information can be combined to optimize inferences about affective states?

How to apply affective information to designing systems? Affective Computing

• Recognize emotions • Express emotions • ‘Have’ emotions Recognize Emotions

• Bio-signals (wearable sensors) • Brain Signals, skin temperature, blood pressure, heart rate, respiration rate • Facial Expressions • Speech/Vocal expressions • Gestures • Limbic movements • Text Recognition

• we need an emotion model that allows us to differentiate between emotional states

• we need a classification scheme that uses specific features from an input signal to recognize the user’s emotions Text

Sentiment Analyzing Discussion Board http://socialxyz.com/SAD/

The Natural Language Toolkit, or more commonly NLTK, is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for the Python programming language. Speech

 Paralinguistic Features of Speech – how is it said?  Prosodic features (e.g., pitch-related feature, energy-related features, and speech rate)  Spectral features (e.g., MFCC - Mel-frequency cepstral coefficient and cepstral features)  Spectral tilt, LFPC (Log Frequency Power Coefficients)  F0 (fundamental frequency of speech), Long-term spectrum Studies show that pitch and energy contribute the most to affect recognition  Speech disfluencies (e.g., filler and silence pauses)  Context information (e.g., subject, gender, and turn-level features representing local and global aspects of the dialogue)  Nonlinguistic vocalizations (e.g., laughs and cries, decode other affective signals such as stress, depression, boredom, and excitement) • Accuracy rates from speech are somewhat lower (35%) than facial expressions for the basic emotions .

• Sadness, anger, and fear are the emotions Speech Signal that are best recognized through voice, while disgust is the worst.

Pre-processing Feature Extraction Classification

Audio recordings collected in call centers Classified Result and, meetings, Wizard of Oz scenarios interviews and other dialogue systems

]M. Pantic, N. Sebe, J. F. Cohn, and T. Huang. Affective multimodal human-computer interaction. In ACM International Conference on Multimedia (MM), 2005. Rafael A. Calvo, Sidney D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

Facial Expressions

Example: Active Appearance Model (AAM)

(AAM) based system which uses AAMs to track the face and extract visual features. Support vector machines are used (SVMs) to classify the facial expressions and emotions.

Lucey, P.; Cohn, J.F.; Kanade, T.; Saragih, J.; Ambadar, Z.; Matthews, I.; , "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression," Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on , vol., no., pp.94-101, 13-18 June 2010 Bio-signals 22

• Physiological signals derived from Autonomic Nervous System (ANS) of human body. – Fear for example increases heartbeat and respiration rates, causes palm sweating, etc. • Psychological Metrics used are: – GSR - Galvanic Skin Resistance – RESSP - Respiration – BVP - Blood Pressure – Skin Temperature • Electroencephalogram (EEG), Electrocardiogram (ECG), Electrodermal activity (EDA), Electromyogram (EMG) • Skin conductivity sensors, blood volume sensors, and respiration sensors may be integrated with shoes, earrings or watches, and T-shirts

Huaming Li and Jindong Tan. 2007. Heartbeat driven medium access control for body sensor networks. In Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments (HealthNet '07). ACM, New York, NY, USA, 25-30. Gestures

 gesture and body motion information is an important modality for human affect recognition; combination of face and gesture is 35% more accurate than facial expression alone.

 Two categories of Body-Motion-based affect recognition  Stylized  The entirety of the movement encodes a particular emotion.  Non-stylized  More natural - knocking door, lifting hand, walking etc.

Fusion Frequently used Detection and Estimation Techniques

• Neural Networks (NN)

• Hidden Markov Models (HMM)

• K-Nearest Neighbors (KNN)

• Linear Discriminant Analysis (LDA)

• Support Vector Machines (SVM)

• Gaussian Mixture Models (GMM)

• Discriminant Function Analysis (DFA)

• Sequential Forward Floating Search (SFFS)

Express emotions

• Kismet (Breazeal and Scassellati, 2002) • Emotional expression for communication and social co-ordination • Emotion for organisation of behaviour (action selection, attention and learning) • Arbib and Fellous (2004)

• More effective expression than humans: • Human expression identified 50% of the time. Computer expression identified 70% of the time (Elliott, 1997). https://www.youtube.com/watch?v=PtCIbGjJV4c 1 2

3 4 1 2

3 4

1. Listening 2. Understand 3. Confused 4. Waving goodbye EXPERIMENTS AND RESULTS Basic Emotions Complex Emotions Stress Level

Feature Estimator

Alert the Driver

Speed, ABS, Driving Aid Agent Traction Control Audio Feature Detector Linguistic / Non-linguistic Navigation Agent Route Selection Feature Detector Facial Expression

Notify …………... Bio-signals Safety Agent in case of Emergency Inter agent Actions …… communication •Steering Movement to aid decision Music, Climate

•Interaction with Gas / Break Paddle making Control

Affective Multimedia Agent 35Seat Pressure Feature Detector 35 Have emotions

• Can machines feel? • How would we know? Criteria for having emotions

• System has behavior that appears to arise from emotions • System has fast ‘primary’ emotional responses to certain inputs • System can cognitively generate emotions • System can have emotional experience • System’s emotions interact with other processes (e.g. memory)