Lecture 2: Theories of While we wait Outline

. Review why emotion theory useful – Give some positive and negative examples . Introduce some features that distinguish different theories – as discrete or continuious – Emotions as “atoms” or “molecules” – Emotions as a consequence or antecedent of emotin . Review some specific influential theories

. Break

. In-class “experiment”

. Evidence for and against dual-process models of emotion . Appraisal Theory . Review a mental health application of affective computing Why should we care about emotion theory? What is a Theory

. Theory explains how some aspect of human behavior or performance is organized. It thus enables us to make predictions about that behavior. – Provides a set of interrelated concepts, definitions, and propositions that explains or predicts aspects of human behavior by specifying relations among variables. – Allows us to explain what we see and to figure out how to bring about change. – Is a tool that enables us to identify a problem and to plan a means for altering the situation. – Create a basis for future research. Researchers use theories to form hypotheses that can then be tested. – Creates a basis for building software: suggests what variables are important to measure and how they relate to each other Example of dangers of atheoretical approaches (e.g., Data Mining)

. We’ll learn about machine learning approaches – Collect bunch of data – Look at lots of features and try to predict some outcome

Input Predicted features output

. Enables us to make predictions about that behavior

. Does not typically allows us to explain what we see and to figure out how to bring about change

. This can easily lead us astray Famous example

. In ‘80s, the Pentagon wanted to harness computer technology to make their tanks harder to attack.

. The research team went out and took 100 photographs of tanks hiding behind trees, and then took 100 photographs of trees - with no tanks.

. They trained a neural network. It reached near-perfect accuracy

. Independent testing showed all “no tank” photos taken on sunny day and all “tank” photos taken on sunny day

. Because neural network was “black box”, this not easy to discover Affective Computing Example

. Last week, showed you system that tIn tries to recognize nonverbal signs of depression and PTSD

. We collected data from two populations – Craigslist (and online job-recruiting service) – US Vets: organization that provides mental-health service for former soldiers

. Tried machine learning approach

. Discovered vocal pitch a strong predictor of depression – Lower pitch predicted depression severity – Not predicted by existing theory

. Turns out there was big imbalance in our data – US Vets had highest rates of depression – US Vets also had highest rates of Males (most soldiers are male) – We actually “discovered” that men speak with a lower pitch Advantages of building on theory

. Theory makes explicit the mechanisms that (are claimed to) underlie some behavior – Allows us to explain what we see and to figure out how to bring about change

. Theories (typical) have good empirical support – The theories we will discuss are supported by dozens of empirical studies – They may still be incorrect of insufficient but are unlikely to suffer the sort of mistakes we just discussed Example: Appraisal Theory

World Events Mental State (beliefs, goals)

Argues for importance of three If we know two of these interrelated concepts variables, we can make • World events Body predictions about the third • Mental state (e.g. goals) Expression • Emotional Response Response= f(Env., Mind) Action Tendency

Physiological Response

10 E.G.: Generating Emotional Response R=f(E,M)

Environment Beliefs, Goals

COMPUTER PREDICITONS: Emotional Response • Computer could predict what Expression emotion a person might hold Action Tendency • Computer could generate a Physiological Response believable emotion to user

11 E.G.: inferring emotional antecedents M=f-1(E,R) Reverse Appraisal

Environment Beliefs, Goals

COMPUTER PREDICITONS: Emotional Response • Computer could predict what Expression goal person has (i.e., what Action Tendency team are they rooting for

Physiological Response

12 Another Example: Galen’s 4-process model of emotion

Posits 4 “prototypical emotions”. Emotions organized in 2-dimensional space (valence, arousal) Argues emotions tend to transition along arrows.

“Angry” “Happy” “Arousal”

“Apa- “Sad” thetic”

“Valence” Galen’s 4-process model of emotion

Prototypical emotions associated with 4 specific physiological systems

YB Bl

“Angry” “Happy” “Arousal” BB Ph

“Apa- “Sad” thetic”

“Valence” Popular 2-dimensional model of emotion

Each prototypical state associated with a characteristic expression

YB Bl C S

Angry Happy “Arousal” BB Ph

M P

Sad Apathetic

“Valence” Again, this theory affords implementation and prediction

. Recognition “language” YB Bl – 4 “prototypical” emotion labels but C S – 2 dimensions

Angry Happy .

Dimension 1 Dimension Predictions BB Ph – If we recognize Anger expect YB is active M P – If recognized Anger, don’t expect transition to

Apathetic Apathy Sad – If BB active, expect sad expressions and self- Dimension 2 report of Sadness

. Control – Can create Apathy by activating Ph system or suppressing other systems – Can’t control Ph by activating Apathy A Hippocratic physician would prescribe treatment to void the body of Theory of the imbalanced humor. if it was a fever -- a hot, dry disease -- the culprit was yellow bile. So, the doctor would try to increase its opposite, phlegm, by Four Humours prescribing cold baths. If there were obvious symptoms of excess phlegm by Galen of Pergamun (c. 180AD) production, the regimen would be to bundle up in bed and drink wine.

Yellow Blood Hot Bile (Fire)

Choleric Sanguine

Black Phlegm

Bile Temperature

Melan- Phleg- choly matic Cold (Air) Dry Wetness Wet (Earth) A good theory is “falsifiable” (Water) Why should we care about emotion theory

. Provides a definition of “emotion” and other related concepts that influence, or are influenced by emotion, and thus a starting point for affective “computing”

. Unfortunately, hasn’t sorted it all out yet – Different theories suggest different concepts and relationships between them

. E.g., Say we want to recognize emotion – Give labeled data to machine learning algorithm – But what are the labels? . Discrete Emotion Theory: focus on discrete labels: Joy, Hope, Fear . Dimensional Emotion Theory: argues discrete emotions do not exist. Instead should focus on broad dimensions: valence and arousal

. Affective computing researchers must make educated guess about which theory to use – But their success or failure can help inform research in the social sciences Scientific approach

Theory Data • e.g.,Psychology Rapport (positive, Human • contingent,Linguistics nonverbal feedback) Behavior • facilitatesNeuroscience conflict • resolutionEconomics

Embed capability within interactive Rapport virtual human Integrated testbed “Test bed”

MRE SASO-ST Gunslinger RapportDCAPS

19 Scientific approach

Inform theoretical Verify Implementation debate in social • Consistent with prior science Human findings? Behavior • Treated “as if” real Test theoretical predictions

Human Integrated Studies “Test bed”

MRE SASO-ST Gunslinger RapportDCAPS

20 For us, A theory should answer “What is emotion?”

. Emotion is a feeling

. Emotion is a state (of physiological arousal)

. A brain process that computes the value of an experience --- Le Doux

. A word we assign to certain configuration of bodily states, thoughts, and situational factors – Feldman Barrett.

. God’s punishment for disobedience -- St Augustine What is emotion NOT?

From Scherer What is an emotion? What is an emotion?

. Components of emotion Emphasizes that emotion potentially impacts several aspects – Cognitive: influences or influenced by thinking – Physiological: related to hormones, heart-rate, sweating… – Expressive: relates to facial expressions, posture, vocal features – Motivation: relates to goals and drives – Feeling: relates to conscious awareness being in an emotional state What is an emotion?

. Phases of emotion: Emphasizes that emotions have “stages” – Low-level: automatic cognitive processes (e.g., reflexes) – Hi-level: deliberate, conscious cognitive processes – Goals/need setting – Examining action alternative: decision-making/action-selection – Behavior preparation – Behavior execution – Communication with other What is an emotion?

. Different theories emphasize different aspects: – Appraisal theories emphasize cognitive antecedents of emotion – Discrete emotion theories emphasize physiological and expressive consequences of emotion

. Affective computing researchers tend to draw on different theories depending on the aspects they focus on – E.g : emotion recognition techniques often draw upon discrete emotion theory and avoid appraisal models What is an emotion: theoretical disagreements

. Different theories can be distinguished by how they chose to define emotion with respect to the previously-mentioned components and phases

– Is emotion discrete or continuous?

– Is emotion an “atom” or “molecule”? (Barrett)

– Is emotion an antecedent or consequent of cognition? Emotions as discrete categories, Emotions are a combination of several biologically fixed, universal to all humans psychological dimensions and many animals

Basic Emotions: Anger, , fear, happiness, sadness, surprise

Rene Decartes, Silvan Tomkins, Paul Wilhelm Wundt, James Russell, Lisa Ekman Feldman Barrett Some discrete models

McDougal 1919

. Ekman – Sadness, happiness, anger, fear, – disgust, and surprise, – Sometimes includes contempt

. Tomkins – Excitement, joy, surprise, distress, – anger, fear, , dissmell, disgust

. Izard & Malatesta ’87 – Happiness, surprise, sadness, anger disgust, contempt, fear

E.g. Le Doux fear circuit Some Dimensional models Russell & Mehrabian’s ‘77 PAD model (pleasure, Russell’s ‘80 circumplex model arousal, dominance) Implications for classification / measurement

Discrete Continuous

Disgust Fear Surprise “Atom” “Molecule”

Emotion components are tightly-coupled Emotions are defined by loose and can be treated as a circuit linking configuration of different components stimuli and response

Jaak Panksepp, Joseph LeDoux, Paul Phoebe Ellsworth, Klaus Scherer, Lisa Ekman Feldman Barrett Implications for classification / measurement

. If emotion is an “atomic” circuit, then all “components” should be aligned – i.e., Facial expressions, physiological response and felt emotion should be consistently-aligned with each other – “Emotion” can refer to the overall circuit but can be measured by any of the components – Expressions should accurately reflect physiology and felt emotion – “Atomic” theories tend to draw on discrete emotion theories Implications for classification / measurement

. If emotion is an “atomic” circuit, then all “components” should be aligned – i.e., Facial expressions, physiological response and felt emotion should be consistently-aligned with each other – “Emotion” can refer to the overall circuit but can be measured by any of the components – Expressions should accurately reflect physiology and felt emotion – “Atomic” theories tend to draw on discrete emotion theories

. Alternative theories – Allow that components influence each other but may be out of sync – Expressions need not accurately reflect physiology and felt emotion – Constructivist Theories (Feldman Barrett): Emotion is a label we assign to our sensed physiological state – Appraisal theories (Scherer & Ellsworth): Emotion is a label a scientist might apply when different components align in a prototypical way Emotion drives behav Behav. Drives emotion (e.g. Appraisal Theory) (e.g., constructivist theories) Emotion precedes and motivates Behavioral response precedes our behavioral response labelling the situation as emotional

Walter Cannon, Phoebe Ellsworth, Klaus William James, Stanley Schachter, Lisa Scherer Feldman Barrett Example (Constructivist Theory)

. Argues first step in the experience of emotion is physiological arousal – Seeing the bear triggers low-level automatic reactions such as arousal and running away

. We next try to find a label to explain our feelings, usually by looking at what we are doing (behavior) and what else is happening at the time of arousal (environment)

. Thus, we don’t just feel angry, happy, etc. We experience general feeling and then decide what they mean (a specific emotion) Schachter 2-factor theory

Appraisal Models

. Constructivist theories argue “seeing the bear” produces arousal

. What if we knew the bear was friendly?

. What if we knew the bear was chained up?

Magda Arnold Appraisal Models

. Appraisal models emphasize the prior beliefs and goals can shape emotional responses

. Explain this by arguing that cognitive processes essentially in initiating emotional responses

World events are “appraised” along a number of dimensions: – Is the event good or bad with respect to my goals – Did I expect the event – Can I control the event – Who do I blame for the event

Different patterns of appraisal will lead to different emotions – I blame someone else for something bad  Anger Some Appraisal Models

. Ortony, Clore and Collins (OCC) Appraisal Variables • desirability •appealingness •praiseworthyness •certainty Some Appraisal Models

. Scherer sequential checking theory

Appraisal Variables • Relevance • Implication • Coping potential • Normative significance

Lesson: Definitions matter

. Geocentricity – Placing earth at center of universe makes it difficult to predict motion of the planets

. Alchemy – All substance can be decomposed into earth, water, air and fire making it difficult to predict consequences of chemical reactions

. Point: – Theory important: allows us make specific predictions and explain variance – Important steps on way to deeper understanding – Recognize that technological choices depend (implicitly or explicitly) on (folk or scientific) theoretical assumptions and failure of the technology may reflect problems with theory, not software 10min break In-class exercise

Split class into 3 groups

• Need group of four volunteers to watch a video • Most of class will stay put and watch this group • Need one more group of four to watch the class

I’ll give out some handouts

• First group will mark down how they feel watching the video • Class will guess what the first group is feeling based on their reactions • Last group will guess what the first group is feeling based on class’s reactions

NO TALKING

Discussion

. Classification – What featured did you use to identify the felt emotion

. Dimensions vs. Basic emotions – Which framework best captured the “meaning” of the interaction

. Observers – Why were (or weren’t) the 3rd group able to infer what is going on

. Mirroring

. How do you think a computer would do? Emotion Theory (continued) Most emotion theories are dual-process theories

Intellect lover of honor and modesty and temperance, and the follower of true glory; he needs no touch of the whip, but is The Allegory of the Chariot guided by word and admonition only

Emotion crooked lumbering animal, … the mate of insolence and pride, shag-eared and deaf, hardly yielding to whip and spur.” Constructivist Models Body Mind

Mind Body

Appraisal Models Dual process models

. Long tradition of separating emotion from cognition

“Emotion” (System 1) “Cognition” (System 2)

. Parallel . Sequential

. Associative . Rule-based

. Intuitive . Rational

Of course, all theories touch on both mechanisms

Specifically, I’m referring to view: • Large independent, loosely coupled processes • Evolved separately

• Represent two “modes” of thinking René Descartes 1641 Descartes René Maybe dual processes are a cognitive illusion

. Argue against this view (Descartes's Error) – One CAN separate emotion and cognition

– Its seductive . Reflect longstanding theoretical and folk distinctions . Consistent with some data – But I’ll argue this data has limited ecological validity (e.g., see also Gigerenzer) . It is fun (and publishable) to show people are irrational

– But this leads to impoverish understanding of both . Cognition w/o emotion is a broken thing Not new idea

. Simon, H. A. (1967). "Motivational and emotional controls of cognition." Psychological Review 74: 29-39. Posited emotion to explain “how a basically serial information processor endowed with multiple needs behaves adaptively and survives in an environment that presents unpredictable threats and opportunities.”

. Several cognitive models seamlessly integrate System 1 and System 2 functions – E.g., Soar, Act-R – I’ve shown emotion can be modeled in these architectures w/o modification Emotional cognition: an example

Emotion about: • Acting in a dynamic and evolving world • Juggling multiple goals and preferences • Confronting opportunities and threats

Emotion can be rational (e.g., Simon; Frank)

Emotion can be sequential (e.g., Schrer)

Emotion follows “rules” (e.g., Frijda)

Marsella and Gratch (2009), “EMA: A process model of appraisal dynamics,” Journal of Cognitive Systems Research, 10(1), pp. 70-90. Sequential Emotions Decision-making

Surprise Orient

Fear Retreat

Anger Attack

Empathy Protect

Marsella and Gratch (2009), “EMA: A process model of appraisal dynamics,” Journal of Cognitive Systems Research, 10(1), pp. 70-90. Dynamics

Surprise

Environment “Working Memory”

Fear Appraisal

Action Inference

Appraisal Affective Frames State Anger

Coping

Empathy Control Signals Dynamics Dynamics in the world

Surprise

Environment “Working Memory”

Fear Appraisal

Action Inference

Appraisal Affective Frames State Anger

Coping

Empathy Control Signals Dynamics Dynamics in Dynamics in perceived the world world relationship

Surprise

Environment “Working Memory”

Fear Appraisal

Action Inference

Appraisal Affective Frames State Anger

Coping

Empathy Control Signals Dynamics Dynamics in Dynamics in perceived the world world relationship

Surprise

Environment “Working Memory”

Fear Appraisal

Action Inference

Appraisal Affective Dynamics Frames State Anger through action

Coping

Empathy Control Signals emotion AND cognition

. The majority of everyday “thinking” involves – Acting in a dynamic and evolving world – Juggling multiple goals and preferences – Confronting opportunities and threats

. Emotion evolved hand and hand with cognition – Two sides of the same system

. Attempts to separate them leads to anomalous behavior

. Yet that is what much of emotion psychology implicitly or explicitly strives to do Mechanically Separate

. Ventral Medial/Orbital Prefrontal Cortex damage – Able to do simple laboratory cognitive tasks

BUT show serious deficits – Abnormalities in emotion Severe impairments in judgment and decision-making in real-life

– Sequential decision-making preserved but becomes unfocused, non-goal- directed Phineas Gage Empirically separate Em & Cog “in the moment” (Clore, Schwarz)

. Emotions inform decisions

. But many experiments separate emotion from decision – Induce an emotion: . Play happy/sad/angry music . Read happy/sad/angry stories – Make people perform an irrelevant task . Buy something . Play ultimatum game – Show logically irrelevant emotion biases decision making

. A common misinterpretation of this data: – Emotion separate from cognition – Emotion “bleeds” over and creates “biases” . i.e., model emotion as rational decision making + bias term Empirically separate Em & Cog over time

. Emotions unfold sequentially hand-in-hand with cognitive processes

. But many experiments break such sequences – Explore “one-shot” decision tasks . Lotteries (Reisenzein) . Ultimatum games

. A common misinterpretation of this data – Emotion is parallel and “unthinking” Some appraisal theories join both perspectives (Arnold, Lazarus, Frijda, Scherer, Ortony et al.)

Desirability Goals/Beliefs/ Environment ExpectednessAppraisal Intentions Controllability

Causal Attribution

Emotion Action Physiological Tendencies “” Response

Problem-Focused Emotion-Focused Coping Strategy (act on world) (act on self)

67 Consequences for Cognitive Architectures (Arnold, Lazarus, Frijda, Scherer, Ortony et al.)

Desirability Goals/Beliefs/ Environment Expectedness Cognition Intentions Controllability(System1)

Causal Attribution

Emotion Action Physiological Tendencies “Affect” Response Emotion (System 2) Resignation TakeProblem action-Focused Emotion-Focused Coping Distancing Seek support Strategy (act on world) (actWishful on self) Thinking

68 Appraisal theory

. For much of the rest of the class I’ll emphasize appraisal theory (esp. work of Smith and Lazarus) – Emphasizes emotion as both an antecedent and consequence of cognition . Provides detailed description of factors that elicit emotion (appraisal) . Provides detailed description of how emotions can shape sequent cognition . Thus can unify appraisal and constructivist approaches – Relatively easy it translate into a computer program – Serves as the theory underlying my own work on affective computing – Can help explain several aspects of emotion . Why a given situation might produce a given emotion . Why an emotion might influence subsequent decisions . Why an expression might shape another’s decision Theoretical Perspective:Appraisal Theory

(Arnold, Lazarus, Frijda, Scherer, Ortony et al.)

Magda Arnold

– Emotion arises from an evolving subjective interpretation of person’s relationships to their environment – Well-suited to computational realization • Emotion arises from series of judgments (appraisals) of how some event impacts an agent’s goals • Artificial intelligence good at doing this sort of thing

70 Appraisal

Desirability Goals/Beliefs/ Environment ExpectednessAppraisal Intentions Controllability

Causal Attribution

Emotion Action Physiological Tendencies “Affect” Response

71 Ortony, Clore and Collins (OCC)

72

Coping shapes beliefs, desires and intentions Coping

Goals/Beliefs/ Environment Intentions

Emotion Action Physiological Tendencies “Affect” Response

Problem-Focused Emotion-Focused Coping Strategy (act on world) (act on self) . Rational models decouple preferences and beliefs – Desires shouldn’t change beliefs (and vice versa) . e.g., Just wanting something shouldn’t make it true – Preferences fixed over time

Intention(play) EU=4

p=.8 Lose Play Game Utility= 20 p=.2 Win Resignation

Wishful Thinking Fear

Intention(play) Sad

p=.6p=.8 Lose Play Game Utility= 1020 p=.4p=.2 Win Distancing Joy Hope

• Coping serves to “confound” beliefs and desires . Emotion-biases on decision making (Loewenstein & Lerner, 2003) . Cognitive dissonance (Festinger57) . Motivated inference (Kunda87) – Little attempt to computationally model . (Marsella&Gratch; Dias) Coping shapes beliefs, desires and intentions

. Appraisal  Emotion: – I’m afraid because I might lose

. Emotion  Coping – I don’t care about winning anyway

. Coping  Re-appraisal – I’m much happier now that I don’t care about wining

. This is core idea behind most therapies for depression (e.g., Cognitive Behavioral Therapy) – Teach people better coping strategies Application Example: Carmen’s Bright IDEAS (CBI)

 Help mothers of pediatric cancer patients cope  Teaching problem solving skills  Bright IDEAS (Identify, Develop, Evaluate, Act, See)  Reach a larger audience  Mothers learn through problem solving scenarios  Story unfolds based on mother’s choices  Learning as a result of experiences with story’s characters  Agent-based interactive characters  Model emotion, personality, dialogue, nonverbal behavior

University of Southern California, Information Sciences Institute CBI Goals

 Teach Bright IDEAS  IDEAS - Rational, Methodical technique  Bright attitude - Critical Emotional Factors  Self-efficacy, healthy coping styles  Problems appraised as challenges, not threats  Provide safe environment that facilitates exploration  Low pressure  Manage stress, facilitate coping  Freedom to explore  Provide engaging, revealing experience  Promote identification  Concretize and highlight pedagogy

University of Southern California, Information Sciences Institute CBI Photo Gallery Act 1: Backstory of Carmen Act 2: Gina and Carmen talk

Act 2: User Interaction Act 3: Diana and Jimmy

University of Southern California, Information Sciences Institute CBI: Act 2 Excerpt (screen capture)

University of Southern California, Information Sciences Institute CBI Interaction: Rubber-band model

Carmen User Gina

This isn’t going to help. Emotions/ Dialog Thoughts

Story Progression

Failure Success Success

Key Pedagogical/Dramatic Junctures

University of Southern California, Information Sciences Institute Realizing the tug-of-war: Gina’s role

 Gina “directs” socially aware dialog:  Goal: Guide Carmen thru BI  Her dialog strategies & sub-strategies structure conversation  “Suggest BI, Guide through I-D-E-A-S”  “Identify (I step) a problem by asking a sequence of questions”  Dialog moves flexibly realize strategies  Dialog Moves: Suggest, Agree, Ask/Prompt, Re-Ask, Answer, Reassure, Agree/Sympathize, Praise, Offer-Answer, Clarify, Resign, Summarize  Moves organized in state machine

Gina’s moves sensitive to the cognitive and emotional state of Carmen

University of Southern California, Information Sciences Institute Realizing tug-of-war: Carmen’s Role

 Carmen’s emotions & coping drive her dialog  Gina asks about Diana’ uncontrollable tantrums  Threatens Carmen’s ego-ideal of being a good mother  Leads to Anxiety/Anger  Carmen copes by discounting significance of tantrums  “She is just being babyish”  May in turn lead to Guilt

 Realization of Cognitve Appraisal Theory (Lazarus 91) Same model underlies development of BI pedagogy

 User influences Carmen’s emotion and coping choices  Allow pedagogically relevant exploration  Consequences played out dramatically

University of Southern California, Information Sciences Institute CBI Character Agent’s Brain

Dialog Annotations

University of Southern California, Information Sciences Institute