sign in sign up
<<
Home , Affective computing, Cognitive science, Epiphany (feeling)

MIT Media Lab oratory Perceptual Computing Section Technical Rep ort No

Revised Novemb er Submitted for publication

Aective Computing

R W Picard

MIT Media Lab oratory Perceptual Computing Ames St Cambridge MA

picardmediamitedu httpwwwmediamitedupicard

Abstract ground in the scientic study of I will suggest com

putational mo dels for aect recognition and will describ e new

Recent neurological studies indicate that the role of

application s of aective computing to areas such as

in human is essential emotions are

assisted p erceptual retrieval creative arts

not a luxury Instead emotions play a critical role

and entertainment and human health Aective computing is

in rational decisionmaki ng in p erception in human

a critical new research area in need of exploration one which

interaction and in human intelligen ce These facts

can signicantl y contribute to advances in emotion and cog

combined with abilities are acquiring in

nition theory while greatly improving the interaction b etween

expressing and recognizing aect op en new areas for

humans and machines

research This pap er denes key issues in aective

I should state some things that I do not intend aective

computing computing that relates to arises from

computing to address Iamnot prop osing the pursuit of



or delib erately inuences emotions New mo dels are

computerized cingulotomi es or even the business of buildin g

suggested for computer recognition of human emo

emotional computers in the negative of the word emo

tion and b oth theoretical and practical applicatio ns

tional which implies a loss of desirable However I

are describ ed for learning humancomputer interac

will discuss creative and unpredictable computers

tion p erceptual creativearts

This pap er will also not review the massive literature on emo

and entertainment human health and machine in

tion and cognition theory I have only included references where

telligence Signicant p otential advances in emotion

needed to supp ort claims related to aective computing

and cognition theory hinge on the development of af

I will also not prop ose answers to the dicult and intriguing

fective computing esp ecially in the form of wearable

questions what are emotions what causes them and



computers This pap er establishes challenges and fu

whydowehave them It is my hop e and exp ectation that

ture directions for this emerging eld

research in aective computing by using computers to recog

nize and synthesize emotions can assist scientists in getting

closer to the answers of these imp ortant questions

Emotion and

This pap er is organized as follows In the remainder of this

section I briey describ e two recentdiscoveries to supp ort the

Nothing in life is to b e feared It is only to b e under

imp ortance of emotion in cognition rst in p erception and

sto o d Marie Curie

second in rational decision making I also present a scenario

based on the role of emotions in learning as an example of

Emotions have a stigma in science they are b elieved to b e

aective computing Section outlines the key issues to devel

inherently nonscientic Scientic principles are derived from

oping computers that can recognize human emotion and express

rational logical arguments testable hyp otheses and

emotion Section p oses human aect recognition as a pattern

rep eatable exp eriments There is ro om alongside science for

recognition problem and prop oses mo dels for its solution this

noninterfering emotions such as those involved in

section may b e skipp ed by those who are not engaged in the de

and the of discovery In fact muchsci

tails of building these Computers that have emotion

entic research funded by defense budgets has b een essentially

present new moral and ethical dilemmas which are broached in

prompted by fear Nonetheless emotions are generally regarded

Sect Computers which can recognize and express aect lead

as wreaking havo c on reasoning Although emotions p ervade

to a myriad of new applications I have suggested over fty

science their role has b een marginalized

in this pap er with most app earing in Sect This research

Why bring emotion or aect into any of the delib erate to ols

agenda and the contributions of this pap er are briey summa

of science Moreover shouldnt emotion b e completely avoided

rized in Sect

when considering prop erties to asso ciate with computers After

all computers control signicant parts of our lives the phone

Songs vs

the sto ckmarket nuclear p ower plants airplane ights

Let me write the songs of a nation I dont care who and more Who wants a computer to b e able to feel angry at

writes its laws Andrew Fletcher them To feel for any living thing

In this pap er I will set forth key issues in what I call af

Emotion pulls the levers of our lives whether it is that

fective computing computing that relates to arises from or



delib erately inuences emotions I will elab orate further on this

The making of small wounds in the ridge of the limbic sys

denition and its implication s b elow

tem known as the cingulate gyrus a surgical pro cedure to aid

The topic of emotion is a dicult one to treat scientically

severely depressed patients



but that is precisely what needs to b e done In this pap er I

For a list of twelve op en questions in the theory of emotion

will illustrate ways in which aective computing can break new see Lazarus

leads to or curiosity that drives scientic inquiry metab olism An overall increase of metab olism o ccurred

As humans our b ehavior is greatly inuenced by the songs but it was not in the higher cortex where it was exp ected

in our hearts Rehabilitati on counselors pastors parents and

Cytowics studies p oint to a corresp onding increase in ac

p olitician s know that it is not laws that exert the greatest inu

tivity in the limbic system a collection of parts of the brain

ence on p eople but rather the drumb eat to which they march

which lie predominately b etween the brain stem and the two



For example the death p enalty has not lowered the murder rate

hemispheres of the cortex The limbic system has tradition

in the states where it has b een instituted as However mur

ally b een assumed to play a less inuential role in p erception

der rates are signicantl y inuenced by or the cultural

than the cortex which lies ab ove it The limbic system is

tune Consider the following scenario

the seat of emotion and Its activity during

Imagine that your col league keeps you waiting for a highly

synesthesia indicates that the limbic system plays a signicant

important engagement to which you thought you wereboth com

role in p erception

mitted You wait with and with increasing puzzlement

In a recent treatise on emotion theory Izard describ es

by his unusual tardiness You think of promises his delay is

emotion as b oth a motivating and guiding force in p erception

causing you to break except for the promise you made to wait

and attention One do es not need a blo o dow scan or theory

for him Perhaps you swear o futurepromises like these He

of emotion however to recognize that emotion greatly inu

is completely unreachable you think what you wil l say to him

ences p erception We are all familiar with emotions inuence

about his irresponsibility But you stil l wait because you gave

on p erception from observing this inuence in other p eople

him your word You wait with growing impatience and frus

inuences that have received names such as the fearinduced

tration Maybe you waver between wondering is he ok and

tunnel vision or the induced seeing through rosecolored

so irritated that you say under your breath Il l kil l him

glasses

when he gets here

The limbiccortic tangle

When he nally shows after you have nearly given up your

last promise of waiting howdoyou resp ond Whether you

Note that my distinction b etween cortical and limbic func

greet him with or relief do esnt his expression aect your

tions is for emphasis only in practice normal limbic and corti

resp onse Your resp onse changes dramatically if he arrives in

cal brain areas do not op erate in but are functionally

considerately carefree or with wo eful countenance Amere

intertwined Not only have the two areas b een articiall y sepa

expression of aect can p owerfully inuence subsequentbehav

rated in most studies but there is a tendency among scientists

ior

to attribute higher functionality to the cortex whichisphysi

In saying that emotions or songs pull the levers of our

cally higher and much easier to prob e

lives I am not suggesting that laws are unimp ortant or even

However discoveries such as that of the limbic role in the

that wedoaway with the lawbased articial intelligen ce AI

higher function of p erception imply that a high or dominating

systems given to computers even though the latter are widely

function is not necessarily cortical Along with the synesthe

acknowledged to b e brittle and unintelli gent Rather I am say

sia ndings mentioned ab ove the research of LeDoux another

ing that laws and rules are not necessarily the most imp ortant

neuroscientist suggests that the hipp o campus long considered

part in human b ehavior and In fact laws and rules

the key structure of the limbic system is signicantl y involved

do not even play a solo in such cognitive tasks as p erception

in registering and making sense of p erceptual patterns LeDoux

or decisionmakin g according to recent neurological evidence

has also recently mapp ed the paths in the brain which demon

Lets consider briey these two activities b eginning with some

strate how the limbic system resp onds to certain events b efore

evidence regarding p erception as illustrated in the next sce

the cortex is involved notifying it after other emergency mea

nario

sures have b een taken Cytowic p oints out in his b o oks that

there are substantiall y more connections from the limbic system

Limbic p erception

to the cortex than viceversa Some scientists have suggested

Oh dear he said slurping a sp o onful there are

that these discoveries indicate that the limbic inuence may

not enough p oints on the chicken Michael Watson

b e the greater Note that this suggestion do es not imply we

are run by our passions as mightbespoken of someone who

do es not act reasonably rather it implies that even reasonable

Synesthetes may feel shap es on their palms as they

behavior is neurologicall y directed by these socalled passions

or see colors as they hear Synesthetic exp eriences b e

As is often the case with scientic claims philosoph ers and

have as if the are crosswired as if there are no walls

artists haveanticipated them ahead of scientic researchers

between what is seen felt touched smelled and tasted How

These neurology ndings are no exception philosophers have

ever the neurological explanation for this heightened p ercep

argued the predominance of emotion for centuries and artists

tual phenomenon is not crossedwires

have incorp orated it into their foundational b eliefs ab out aes

The neurologist Cytowic has studied the neurophysiol ogy of

thetic goals

synesthetic exp erience Because the cortex is typically re

Although the role of emotions is p owerful we often overlo ok

garded as the home of sensory p erception it is exp ected to show

their inuence However consider that we often hear a p erson

increased activity during synesthetic exp eriences where a p er

say SorryIwasnt thinking but not SorryIwasnt feeling

son exp eriences external and involuntary sensations somewhat

Whatever our p erception of its role the limbic system is a cru

like a crosswiring of the senses for example certain sounds

cial player in our mental activity It is hard to say conclusivel y

may elicit seeing strong colors One would exp ect that during

which system of the brain is directing the show but if the

this heightened sensory exp erience there would b e an increase

limbic system is not directing then it is at least an actor that

in cortical activity p erhaps in the parietal lob es tertiary asso

ciation area where the three senses of vision touch and



There is not complete agreement on what parts of the converge However Cytowic found that scans of cortical blo o d



brain constitute the limbic system but it includes at least

ow during synesthesia episo des indicate a collapse of cortical

the hyp othalamus hipp o campus in the temp oral lob e and the



amygdala Measured by the Ob eristKetty xenon technique

Damasios patients havefrontallob e disorders aecting a has won the heart of the audience

part of the cortex that communicates with the limbic system

Reevaluating decision making

Otherwise the patients app ear to b e intelligent and unusual ly

rational However these same patients suer from an impaired

It is not surprising that emotions play a critical role in p er

abilitytomake decisions Years of studies with frontallob e

ception but what ab out emotion in more rational cognitive

patients indicate that they sp end inordinate amounts of time

functions such as decisionmakin g We all know emotions can

trying to make decisions that those without frontallob e damage

inuence decisions often negatively but do they play a more

can make quite easily For example the mere task of cho os

signicant role Below I will describ e recent neurological evi

ingadatetoschedule an app ointment can lead these patients

dence that indicates a more surprising and signicant role for

through abnormally long chains of decisions p erhaps without

emotions

ever reaching a decision until a date is imp osed up on them by

someone who is tired of waiting for their resp onse

The thinkingfeeling axis

The frontallob e disorder in these patients interferes with

Scientic conclusions must b e decided with the head

their abilitytocombine emotional limbic resp onses with their

whom you cho ose to marry may b e decided with the

otherwise cortical decisionmaki ng Damasios hyp othesis is

heart folk advice

that emotion plays a biasing role in decisionmakin g estab

Most p eople consider that b oth head and heart are useful

lishing the values used in evaluating p otential outcomes and

for decisionmaki ng as long as they are kept in their rightful

essentially warding o an innite logical search

place as in the folk advice ab ove In fact p eople often p olarize

Damasios ndings supp ort indep endent scientic arguments

and sp eaking of them as separate phenomena

for the essential role of emotion JohnsonLaird and Shar have

The p opular MyersBriggs p ersonalitytyp e indicator has

recently reminded the cognition community of the inabili tyof

thinking and feeling as opp osite endp oints of one of its

to determine which of an innite numb er of p ossible con

axes for describing p ersonality In fact the MyersBriggs typ e

clusions are sensible to draw given a set of premises Con

indicator reveals a gender bias along this axis indicating that

sider howdoyou decide which path to takegiven some ev

twothirds of men tend to lie closer to the thinking side and

idence There is not time to consider every p ossible logical

twothirds of women tend to lie closer to the feeling side

constraint and asso ciated path Emotion do es not merely play

This bias sometimes app ears in malefemale stereotyp es and

a tiebreaking role in making certain decisions rather it ap

many b o oks have app eared on its implications for human inter

p ears to b e essential in learning the biases required to construct

action Although I do not wish to pursue the malefemale dis

rational resp onses

tinctions further here it is worth noting that such dierences

Damasios ndings provide neurological supp ort that there

might also extend to humancomputer interaction As such

is no pure reason in the healthyhuman brain emotions are

aective computers might tend to b e considered more fem

vital for healthy rational human thinking and b ehavior His

inine However this is not an appropriate conclusion given

patients are abnormally rational not to o unlike the rulebased

the neurological evidence that b oth male and female rely

programs that comprise to days mo dels of decisionmakin g

on emotion in normal thinking As such aective computers

It must b e emphasized at this p oint that by no means should

should not b e considered more feminine but more human

anyone conclude that logic or reason are irrelevant they are

Neurologically no p olarizatio n or clean dividing line o ccurs

as essential as the laws describ ed earlier Additionall ythe

between thinking and emotions In fact we will nd something

neurological evidence describ es an essential role for emotions

completely unexp ected Recall that the brain do es not cleanly

the songs

separate cortical and limbic activity

Therefore these ndings indicate that scientic study of

emotion should not b e merely an interesting sidearea of study

Authorities in neuroanatomyhave conrmed that the

in but rather the study of emotion is essential

hipp o campus is a p oint where everything converges

if we are to understand human cognition The implication s are

All sensory inputs external and visceral must pass

signicant also for and industry computers

through the emotional limbic brain b efore b eing redis

if they are to b e truly eective at decisionmaking will have

tributed to the cortex for analysis after whichthey

to have emotionlikemechanisms working in concert with their

return to the limbic system for a determination of

rulebased systems Pure reason maycontinue as a Platonic

whether the highlytransformed multisensory input

ideal but in successful cognitive systems it is a logical howler

is salient or not

The limbic brain is the home base of emotion but it is not

Tests of thinking tests of feeling

the only part of the brain engaged in the exp erience of emotion

In normal human cognition thinking and feeling are mutually

Extensive researchby Damasio and his colleagues has identied

present If one wishes to design a device that thinks in the

several nonlimbic regions which aect emotion These ndings

sense of mimicking a human brain then must it b oth think and

have b een recently summarized in the provo cative b o ok But

feel

there is a much bigger in his ndings

Consider briey the classic test of a thinking machine the



Too little emotion impairs decisionmaking

Turing test The examines if in a typical conver

sation b etween two participants who have no sensory contact

We all know that to o much emotion can wreak havo c on rea

with each other the tester cannot tell if the replies are b eing

soning but now there is evidence that too little emotion can

generated byahuman or a machine Although the test can

also wreak havo c This evidence requires a shift from the usual

not prove that a machine do es or do es not think it is a terric

paradigm of how p eople separate emotions and rationalityI

exercise in thinking ab out thinking

refer the reader to the careful arguments and references col

lected by Damasio for the justication such a farreaching



paradigmshift demands and here provide but a brief explana

With slight mo dications from the original prop osed by

tion of the ndings to supp ort the need for aective computers Turing

The Turing test is considered a test of whether or not a ma Because aective communication o ccurs naturally b etween

chine can think in the truest sense of duplicating mental ac p eople it is exp ected by p eople when they interact with com

tivity Since mental activityinvolves a close coupling b etween puters In fact we often see p eople attribute emotion to things

cortical and limbic activity a test of true thinking must involve that clearly do not have emotion a windup dog that wags its

a test of emotion tail for example Although p eople know that windup toys and

computers do not have emotions nonetheless their discourse

Consider that one might converse with the computer pas

and actions often assume them

sionately ab out a song or a p o em or describ e to it the most

Emotion plays an essential role in communication even in

tragic of accidents To pass the test the computer resp onses

its subtlest form where it merely indicates that communica

should b e indistingui sh abl e from human resp onses Because hu

tion has succeeded that we are understo o d If you reprimand

mans almost always resp ond with emotion to highly emotional

someone and their expression do es not change then the natural

events either to empathize or to counter balance the intelli

inclinati on is to continue your communication until you receive

gent computer would need to b e capable of recognizing emotion

a visible or verbal sign that it has succeeded This typ e of hu

and providing aective resp onses

man exp ectation may p erhaps b e at the ro ot of the practice of

Although the Turing test is usually p erformed with textonly

many computer users who keep typing the same wrong thing

communication so that sensory expression viz voice intona

at the computer hoping it will eventually resp ond dierently

tion and do not play a role emotions are

Aect recognition and expression is also necessary for sympa

still communicated through the written word This p ower and

thy and communication of understanding the latter of whichis

imp ortance of inuencing emotion through was a pri

considered one of mans greatest psychological needs Ne

A machine even limited mary tenet of s Rhetoric

reminds us that even a puppycan grop onte in Being Digital

to text communication will b e a more eective communicator

tell when you are angry with it

if given the ability to p erceive and express emotions

Basic aect recognition and expression is exp ected byhu

Of course the crux of the Turing test is what comprises the

mans in communication Computerbased communication to

questions Hofstadter has suggested that humor esp ecially

date has largely removed or ignored aective bits A quantum

emotion would comprise the acid test of intelligen ce for a

leap in communication will o ccur when computers b ecome able

thinking machine But the argument for emotion as nec

to recognize and express aect

essary for intelligenc e go es far b eyond its interplaywithhumor

Goleman has recently argued that emotion is a hallmark of hu

Example The eective and aective

man intelligen ce and that can b e

piano teacher

more imp ortant for predicting success in life than traditional

IQ tests Emotional intelligence involves factors suchasempa

Before moving to the key issues and researchchallenges in af

thy which in a machine would require at least a keen ability

fective computing lets consider an example of its use One of

to recognize understand and express emotions if not also the

the interests in the Media Lab is the building of b etter piano

ability to have emotions

teaching computer systems in particular systems that can

grade some asp ects of a students expressive timing dynamics

phrasing etc This goal contains manychallenges one of

Aective communication

the hardest of whichinvolves expression recognition distillin g

An increasing numb er of p eople sp end more time directly com

the essential pitches of the music from its expression Recogniz

municating with their computer than with other p eople Daily

ing and interpreting aect in musical expression is imp ortant

interaction b etween humans and computers has tremendous

and Ill return to it again later But rst lets consider a sce

psychological impact not to mention billions of dollars of eco

nario where you are receiving piano lessons from a p ersonal

nomic impact It is not my purp ose to review the researchin

computer teacher

this area whichiscovered in numerous conferences however I

Imagine you are seated with your computer piano teacher

would like to describ e one intriguing set of recent results which

and supp ose that it not only reads your gestural input your

supp ort the imp ortance of aective computers

timing and phrasing but that it can also read your emotional

This particular set of studies was recently conducted by Nass

state In other words it not only interprets your musical expres

and his colleagues at Stanford They p erformed a num

sion but also your facial expression and p erhaps other physical

b er of classical studies of human so cial interaction substituting

changes corresp onding to your emotional feelings Assume it

computers into a role usually o ccupied byhumans Hence a

has the ability to distingui sh the three emotions we all app ear



test that would ordinarily study a humanhuman interaction is

to have at birth distress and pleasure

used to study a humancomputer interaction For example one

Given aect recognition the computer teacher mightndyou

exp erimentmight study how what is said byhuman A ab out

are doing well with the music and you are pleased with your

human Bs p erformance changes when A gives the evaluation

progress Am I holding your interest it would consider

facetoface with B vs when A gives the evaluation ab out B

In the armative it mightnudge you with more challengin g

to another presumably neutral p erson In general humans

exercises If it detects your frustration and many errors it

are nicer facetoface In Nass et als variation human B is

mayslow things down and giveyou encouraging suggestions

replaced with computer B and human A gives an evaluation

Detecting user distress without the user making mechanical

b oth to computer B and to another computer Despite the

playing errors might signify a moving requiem a sticky piano

switch Nass and colleagues found that the human results still

key or the need to prompt for more information

held eg the tendency to b e nicer face to face still held Nu

The pianoteacher scenario raises the issue of observing not

merous other exp eriments were done in this vein revealing that

just someones but also their underlying

the classic results of the humanhuman studies were maintained

emotional state Howdowe detect a p ersons emotions Is

in the humancomputer studies After accounting for p otential



biasing factors Nass et al concluded that individua ls inter

This view of is not unchalleng ed facial expression

actions with computers are inherently natural and so cial

in the womb and on newb orns has no broadly accepted

explanation

it via some metaphysical sixth sense Whether or not sucha Duchenne over a century ago

sense might exist or play a role exceeds the scientic scop e of

The muscle that pro duces this on the lower

this pap er consequently this p ossibil ity will not b e further ad

eyelid do es not ob ey the will it is only broughtinto

dressed However there are measurable ways that app ear to

playby a genuine feeling by an agreeable emotion

signicantly inuence howwe discern emotion we can mea

Its inertia in smiling unmasks a false friend

sure digitall y the signals of voice facial expression and other

asp ects of socalled b o dy language Moreover there is evidence

Neurological studies also indicate that emotions travel their

that we can build systems that b egin to identify b oth emotional

own sp ecial path to the motor system If the neurologist asks

expression and its generating state First however it is nec

a patient who is paralyzed on one side to smile then only one

essary to understand more ab out how emotions are generated

side of the patients mouth raises But when the neurologist

and expressed

cracksafunny joke then a natural twosided smile app ears

For facial expression it is widely accepted in the neurological

Physical and Cognitive Asp ects of

literature that the will and the emotions control separate paths

Emotion

If the lesion is in the pyramidal system the patients

cannot smile delib erately but will do so when they feel

What signals assist you in guessing another p ersons emotional

happy Lesions in the nonpyramidal areas pro duce

state Are there universal physiologi cal measurements from

the reverse pattern patients can smile on request

which a computer could discern mo o d Are emotions purely

but will not smile when they feel a p ositive emotion

cognitive purely physical or some kind of mix If an

in

emotion causes an accompanying physiological state can that

state b e induced to cause that emotion

In other words a faked smile travels a dierent path than

The literature on emotion and cognition is still debating most

agenuine one Not only do es this imply that physiologi cal ly

of these questions in fact emotion theorists still do not agree

false and sincere smiles may b e discriminated but it illustrates

even on a denition of emotion The goal of this section is

the existence of multiple paths for emotional expression When

to provide some background from the literature which I nd

I talk later ab out how computers might b e given emotions it

imp ortant to the development of aective computing

is imp ortanttorememb er that emotiongeneratin g mechanisms

Particularly relevant are the two generally opp osing treat

distinct from ordinary rulebased systems might b e required

ments of emotion in the literature emotion as cognition and

Togive computers the ability to recognize emotions we will

emotion as physiologi cal resp onse Below I will emphasize the

need to consider howtheinternal emotional state is communi

role of both physical and cognitive comp onents for aectivecom

cated physiologi cal l y ie in ways that are externally observ

puting I will also clarify some terminology and highlight ways

able Debate p ersists ab out the nature of the coupling b etween

in which emotion is b oth expressed and induced

emotion and physiologica l resp onse although most scientists

It is b eyond the scop e of this pap er to overview the literature

now accept b oth a physiologica l and a cognitivecomponent

with its many theories of emotion I will refer the reader instead

in their treatment of emotion In the following sections I will

to the collections gathered by Plutchik and Kellerman and

discuss key issues related to the physiologi cal comp onents then

to the references at the end of this pap er which themselves



the cognitive comp onents and nally inducementbetween the

contain many excellent surveys The fo cus b elowisonthe

physical and cognitive

background most relevant to giving computers the abilityto

recognize express and mo del aect

Terminology

Poker face p oker b o dy

Before pro ceeding it is helpful to clarify terminology Sentic

emotionaland aective are used interchangeably in this pap er

The level of control involved in p erfecting ones p oker face to

although I will tend to use sentic to refer to the physical man

hide emotions is praised bywestern culture But can weperfect

ifestations of emotion An aective state refers to your internal

apoker b o dy Despite her insistence of condence you hear

dynamic state when you have an emotion This emotional state

fear in her voice although he refuses to cry in your oce you

cannot b e directly observed by another p erson but maybein

see his eyes twitching to hold backapotential o o d You sp ot

ferred

the lilt in mywalk to day and therefore exp ect I am in a go o d

mo o d Although you might successfully conceal nervousness

All you consciously p erceiveofyour own emotional state is

in your voice you may not b e able to suppress it throughout

referred to as your emotional experience Some authors equate

your b o dy its evidence may sneak into a clammy handshakeor

emotional exp erience with emotional feelings but I prefer to

emerge as cold feet

minimize use of the word feelings as it can b e ambiguous with

Although we cannot observe directly what someone feels or sensory feelings eg feeling a pinprick

thinks for that matter and they may try to p ersuade us to

What you reveal to others voluntaril y p erhaps even a false

b elieve they are feeling a certain waywe are not easily de

smile or not what you do not succeed in suppressing is

ceived Beethoven after he b ecame deaf wrote in his conver

your emotional expression Expression via the motor system

sation b o oks that he could judge from the p erformers facial

or sentic mo dulation is usually involuntary and is one clue

expression whether or not the p erformer was interpreting his

which others observe to guess your emotional state

music in the right spirit

Despite the fact that we are not all exp erts at reading faces

but these claims are unproven



and comedians and actors can excel at feigning emotions it is

Of course the cognitive comp onents ie the brain are also

claimed that the attentive observer is always able to recognize

physical Treating the b o dy and separately can lead to er



a false smile This is consistent with the ndings of

rors as captured by the title of The separation will b e used

here primarily to distinguish arguments from the literature



Sentic is from the Latin sentire the ro ot of the words This view is debated eg by who claims that all phe

sentiment and sensation nomena that change with emotion also change for other

Finally mo o d tends to refer to a longerterm emotional these exp eriments the sub ject delib erately expresses an emo

state although duration might b e dicult to quantify given tional state by pressing against a measuring surface while ex

that mo o ds can swing abruptly p eriencing that state The ngerpressure resp onse has b een

measured for thousands of p eople and found to b e not only

Physiological asp ects of emotion sentic

rep eatable for an individ ual but to reveal distinct traces for

states such as no emotion hate love joy sex and

mo dulation

across groups of individu als and to some extent

There is a class of qualities which is inherently linked

Clynes suggests that these traces are indicative of the

to the motor system it is b ecause of this inherent

underlying essentic form Other forms of motor output suchas

link to the motor system that this class of qualities

chin pressure for a patientwhowas paralyzed from the neck

can b e communicated This class of qualities is re

down and fo ot pressure have yielded comparable characteristic

ferred to commonly as emotions

essentic forms

In each mo de the emotional character is expressed

There are manyphysiologica l resp onses whichvary with time

by a sp ecic subtle mo dulation of the motor action

and which mightpotentially b e combined to assist in recogni

involved which corresp onds precisely to the demands

tion of emotional states These include heart rate diastolic and

of the sentic state

systolic blo o d pressure pulse pupillary dilation respiration

Manfred Clynes skin conductance and color and temp erature These forms of

sentic mo dulation will b e revisited near the end of this pap er

The b o dy usually resp onds physically to an emotion al

in the discussion on aectivewearable computers

though Jamess view of this resp onse being the emotion

Given the human is exp eriencing an emotion eg hate then

is not accepted to day Nonetheless the motor system acts as

certain values of motor system observations such as a tense

a carrier for communicating emotional state what I call sen

voice glaring expression or nger pressure strongly awayfrom

tic mo dulation after the foundational principles Clynes estab

the b o dy are most probable Respiration rate and heart rate

lished in this area

may also increase In contrast given feelings of joythevoice

Sentic mo dulation eg voice inection facial expression

mightgoupinpitch the face reveal a smile and the nger

p osture is how an emotional state is typically expressed it is

pressure have a slight b ouncelikecharacter Even the more

the primary means of communicating human emotion When

diculttoanalyze conscious emotions such as and

computers learn to recognize human emotion they will rely pri

exhibit marked p ostural dierences whichmightbe

marily on sentic mo dulation Togive computers aect recogni

observed in howyou stand walk gesture or otherwise b ehave

tion requires understanding the physical manifestations of emo

Aective computers can b egin to recognize these forms of

tion

emotional expression despite the fact that there is still no solid

Anumb er of emotion and cognition theorists have studied

denition of emotions no resolution of basic or discrete

the physiologi cal correlates of emotions Lazarus et al

emotions and no agreed up on evidence for universal patterning

argue that each emotion probably has its own unique somatic

I will discuss these issues further b elow but rst let us consider

resp onse pattern and cite other theorists who argue that each

some of the other complicatin g factors which exist

has its own set of unique facial muscle movement patterns

Complicating conditions

Facial expressions are one of the two most widely acknowl

edged forms of sentic mo dulation Duchenne in his thesis

Studies attempting to asso ciate b o dily resp onse with emo

republished in identied indep endent expressive face mus

tional state are complicated byanumb er of factors For ex

cles such as the muscle of attention muscle of muscle of

ample claims that p eople can exp erience emotions cognitively

disdain or and muscle of joy Most present attempts

suchaslove without a corresp onding physiologi cal resp onse

to automate recognition of facial expression are based on the

such as increased heart rate are complicated by issues such

subsequentFacial Action Co ding System of Paul

as the intensity of the emotion the typ e of love how the state

Ekman whichprovides mappings b etween muscles and an

was supp osedly induced watching a lm imagining a situa

emotion space

tion and how the p erson was or was not encouraged to express

The second widely acknowledged form of sentic mo dulation

or suppress the emotion The latter can b e particularly sub

is via voice intonation you can hear loveinhervoice anxietyin

tle as there are so cial display rules hidden in each culture

his Vo cal emotions can b e understo o d byyoung children b efore

indicating when and where it is appropriate to express certain

they can understand what is b eing said and by dogs which

emotions

we assume can not understand what is b eing said Voice of

Another complicating factor is that there maybephysio

course is why the phone has so much more bandwidth than

logical resp onses similar to those in an emotional state but

email or a written letter Sp oken communication transcends

not corresp onding to an emotion eg heart rate also increases

the message of the words

when exercising However a that is try

Other forms of sentic mo dulation have b een explored by

ing to measure emotions mightalsohave a camera and other

One of his prin Clynes in his pioneering b o ok Sentics

sensors attached eg step rate so that it can recognize you

ciples that of sentic equivalence allows one to select an ar

are exercising These basic activities can b e built into the af

bitrary motor output of sucient degrees of freedom for the

fective recognition mo del as conditioning variables For ex

measurement of essentic form a precise spatiotemp oral dy

ample in our current researchwithawearable wireless aec

namic form pro duced and sensed by the nervous system which

tive headmounted camera for augmenting visual memory we

carries the emotional message The form has a clear b eginning

have found it more relevant to asso ciate framerate not with

and end that can b e expressed byvarious motor outputs By

just heartrate but with a function that combines heartrate

this principl e emotional state could b e determined from out

and steprate

puts other than facial expression or voice

Leidelmeijer overviews several conicting studies in re

The motor output explored most carefully by Clynes is the minding us that a sp ecic situation is not equally emotional for

transient pressure of a nger during emotional expression In all p eople and an individ ual will not b e equally emotional in

all situations Such studies p oint to hardtomeasure bio chem priority it mightquickly alert you to more news on the b omb

ical and cognitive inuences in emotion the latter of which are ing or fetch phone numb ers to help you get through to your

addressed more carefully b elow family memb ers wherever they are

Personal patterning instead of universal Studies out of the lab oratory

patterning

The complications noted ab ovehave particularly plagued

lab oratory studies For example certain sub jects mightfeel

One of the outstanding problems in trying to asso ciate emo

inhibited ab out expressing during a lab oratory study

tions with physiologica l resp onses is that sometimes dierent

Other sub jects might nd the situations in the study contrived

individu als exhibit dierentphysiologica l resp onses to the same

and exhibit a much smaller rep ertoire of emotions than they

emotional state Leidelmeijer discusses the evidence b oth

would exp erience in their natural world Moreover the natu

for and against universal autonomic patterning The diculties

ral world itself will dier among sub jects actors and musi

in nding consistent universal patterning mechanisms app ear

cians tend to exhibit emotions more readily than scientists and

to make the outlo ok grim for constructing computers that can

engineers

recognize aect

As mentioned ab ove emotion theorists have tended to lo ok

However my exp erience with signalpro cessin g recognition

for universal patterns instead of for consistent patterning

systems leads me to suggest that this situation parallels that

within an individu al conditioned on a context With aective

of another problem the problem of constructing sp eaker

p ersonal computers understanding the individu al resp onse is

indep endent sp eech recognition systems and therefore has a

most imp ortant rst after reliable data has b een gathered for

creative solution

individu als then comparisons can b e made across p opulations

The goal of sp eakerindep endent systems is to recognize what

Because most studies on emotion and cognition have b een

was said regardless of who said it Even among p eople who use

conned to articial lab scenarios they have b een severely

the same language this goal is complicated by the fact that

limited Aective computers particularly if wearable would

two p eople saying the same sentence pro duce dierent sound

b e able to measure emotional resp onses via sentic mo dulation

signals The computer has diculty separating the language

wherever and whenever they o ccur for b oth individ ual s and

part of the signal from the part of the signal that identies the

larger groups Aective computing allows the lab oratory to

sp eaker and his or her expression Consequently the computer

visit the sub ject instead of the other way around

has a hard time recognizing what was said unless it was trained

on the individua l sp eaker or on someone that sounds like that

Cognitive asp ects of emotion

sp eaker

Although it would b e a terric accomplishment to solve this

Humans are frequently conscious of their emotions and we

universal recognition problem it is unnecessary in aective

know from exp erience and lab oratory study that cognitiveas

computing Negrop onte p ointed out years ago that an alter

sessment can precede the generation of emotions Consequently

native solution is to solve the problem in a sp eakerdep endent

some prominentscientists have argued that cognitive appraisal

way so that your p ersonal computer can understand you and

is a necessary precondition for emotion Although it is hard to

your language thereafter your computer can translate what

prove that anyhuman exp erience exists indep endent of cog

you said to the rest of the world

nitiveevents there seems to b e ample evidence that emotions

Similarly I b elieve that exp eriments in recognizing emotional

can o ccur without prior cognitive appraisal In par

state from observations of physical expression only need to

ticular the recent neurological evidence seems to supp ort that

demonstrate consistent patterning for an individua l in a given

emotions can hijack the cognitivecenters of the brain

p erceivable context It is only necessary that your p ersonal

Additionall y noncognitivebiochemical events can strongly in

computer b e able to recognize your aect it can then translate

uence mo o d Only recently have scientists b egun to unlo ck

this information if you p ermit it to b e communicated to others

the secrets of hormonal chemistry the role of neurotransmitters

For example the aect recognizer for one context mayndit

in depression and other signicant noncognitive contributors to

most reliable to use a blend of blo o dow and facial expression

human emotion

for one p erson and a measure of skintemp erature galvanic

It seems safe to conclude that b oth cognitiveandphysio

skin resp onse and respiration for another p erson This adapt

logical events can contribute to emotion and viceversa Note

ability greatly simplies the problem as a computer learning

there is no hard distinction b etween cognitive and noncognitive

can b e used to determine which features are the b est

just as there is no hard line b etween the brain and the b o dy

predictors for each individual Moreover typically in adap

after all thinking is b oth a cognitive and a physiologica l event

tive scenarios a relatively small numb er of categories emerges

the mindb o dy separation is one of convenience not physical

ie certain individu al s tend to exhibit similar physiologica l re

reality

sp onses simplifying the amount of adapting the system has to

A helpful distinction for sorting the noncognitivel y

do

generated and cognitively generated emotions is made by

The individ ual s p ersonal computer will resp ond b est if it

Damasio who distinguish es b etween primary and sec



is also able to p erceivecontext eg sense if youre climbing

ondary emotions Damasios idea which is also supp orted in

stairs if the ro om temp erature changed if you just read a news

much of the emotion theory literature is that there are certain

story ab out a tragic b ombing etc The computer can there

features of stimuli in the world that we resp ond to emotionally

fore identify autonomic resp onses conditioned on p erceivable

rst and which activate a corresp onding set of emotions and

factors For b est p erformance p erceivable context should ulti

cognitive state secondarily Such emotions such as startle

mately include not only the public milieu such as the comfort

up on hearing a loud bang or the fear that causes an infantto

index of the weather but also the private milieu for exam

retreat when a large ob ject lo oms rapidly near are primary

ple the information that you have family in the town where

and reside in the limbic system most likely the amygdala

the giantbomb just explo ded The priorities of your p ersonal



aective software agent need to shift with your aective state

Damasios distinction compares to that of preattentive

and p ostattentive pro cessing in the vision literature For example predicting your concern for your family as a top

These are the innate emotions Jamesian in their accompany ceivable environment and physiologi cal resp onse are imp ortant

ing physical resp onse But these are not all Damasio denes in b eginning to recognize and understand human emotions in

secondary emotions as those that arise later in an individ natural situations

uals development when systematic connections are identied As data is collected from a variety of situations patterns may

between primary emotions and categories of ob jects and situ b e found whichwould improve a computers ability to predict

ations An example is grief where physical resp onses o ccur in cognitive emotional resp onses to situations

conjunction with cognitive understanding of an event suchas The successful predictivemechanisms as in the physical case

death of a loved one For secondary emotions the limbic struc is likely to dep end on an individu al For example someone who

tures are not sucient prefrontal and somatosensory cortices has never seen a neighb orho o d cat run over by a car mightbe

are also involved deeply disturb ed the rst time they are close to suchanevent

however the man who routinely removes animal carcasses from

Damasios patients were unusual in that they had primary

the roads mayhave no emotional resp onse The p ersons his

emotions but not secondary emotions The hardwired emo

toryvalues and general emotional maturity combine to inu

tions were there but the ability to generate appropriate cog

ence their cognitive resp onses

nitive emotions was not even though the patients knew the

Therefore universal mo dels of emotionpro ducin g mecha

appropriate emotional resp onse to b e generated

nisms are likely to b e oversimplied and in need of customiza

The cortical involvement in the secondary emotions presum

tion when it comes to predicting individ ual resp onses An af

ably helps construct imp ortant links connecting cognitively

fective computer could collect and analyze events and resp onses

recognized events to primary emotional resp onses These links

for an individua l and compare these pairings to a set of predic

allow us to generalize primary events such as ight from big

tive mo dels determining which mo dels give the most accurate

lo oming ob jects into cognitive constructs suchasstayoutof

predictions under dierent circumstances Such adaptive data

the paths of big ob jects such as cars trucks and trains The

collection abiliti es will b e necessary to develop comprehensive

corticallimbi c links set up during construction of secondary

theories of the cognitive asp ects of emotion

emotions can also b e used to cognitively call forth a corresp ond

ing primary emotion at will

Achallenge in understanding

The complex cortical activities available to humans probably

An area for fruitful investigation is understanding how

also account for their ability to construct selfconscious cog

cognitiveaective links inuence memory retrieval and in con

nitive emotions such as shame and guilt which are not present

junction creativity Improvement in memory retrieval has b een

in infants but develop later in life Babies demonstrate a

found to o ccur when a p erson is in the same emotional state

less complicated rep ertoire of emotions than cogitating adults



asso ciated with the memory b eing retrieved Memory

despite the fact that babies have not learned the so cial rules of

retrieval is a key activity in making free asso ciations which are

suppressing emotions

imp ortant for analogical thinking and creativity Hence it is

Complicating conditions

natural to exp ect to nd relations b etween emotional states and

creativity

Anumb er of factors confound purely cognitive attempts

Damasios ndings linked cortical constructs to primary emo

to understand emotion Several of the factors mentioned in the

tions consequentlywe might exp ect cortical constructs for cre

previous section are relevant to lab studies of cognitiveemo

ative thinking and memory retrieval to also develop emotion

tions esp ecially so cial display rules and bio chemical inuences

links The mechanism Damasio describ es may therefore ac

In cognitive studies of emotion sub jects are usually asked to

count for a separate idea recently prop osed by the computer

verbalize their emotional state as opp osed to or in addition

scientist Gelernter Gelernter has suggested a phenomenon

to its b eing physically measured However the problem of

he calls aect linking which mightplay an imp ortant role in

attaching adjectives to emotions is immense

creativityHowever Gelernter suggested that this phenomenon

Wierzbicka has made one of the most comprehensive

arises primarily during what he termed low fo cus thinking

attempts to dene emotion in terms of universal se

and not during what he termed high fo cus reasoning How

mantic primitives such as go o d bad and want resulting

ever as I have indicated the neurological evidence indicates

in a distinct script for each emotion in terms of a set of

that emotion do es play an imp ortant role in higherlevel deci

primitives These scripts involve goals and their achievement

sion making it is therefore not restricted to lowfo cus think

or lackofachievement

ing as Gelernter muses

Aective computers could simulate many of the comp et

ing goalmotivation theories of emotion by enco ding them into

A note on inducement of emotion

scripts Computers could then b e put into so cial situations

Certain physical acts are p eculiarly eective esp e

likewas done in the Stanford studies mentioned earlier Aec

cially the facial expressions involved in so cial com

tive computing could therefore provide a test b ed for imp or

munication they aect the sender as muchasthe

tant cognitivegenerati ve theories of emotion However aec

recipient

tive computers cannot currently exp ect to measure cognitive

inuences these dep end on selfrep orts whicharelikely to b e

There is emotional inducementever at work around us a

highly variable and no one can read your mind yet

go o d marketing professional playwright actor or p oliticia n

Under controlled environments or with the assistance of

knows the imp ortance of app ealing to your emotions Aristotle

some wearable acoustic and visual scene analysis wemayhow

devoted much of his teachings on rhetoric to instructing sp eak

ever b e able to measure the p erceivable cognitive milieu eg

ers how to arouse the right emotions in their audience

the ro om is tiny and hot a stranger enters walking towards

Although inducement of emotions may b e delib erate it seems

the sub ject rapidly etc We can also measure physiologi cal

we the receiver often enjoy its eects Certainlywe enjoy

resp onses such as facial expression breathing etc with the

selecting a stimulus suchasmusic that will aect our mo o d in

more mo des initiall y the b etter We should at least b e able



to measure physiologi cal ly those emotions which are already

A discussion of numerous aectmemory exp eriments as

well as some controversy surrounding them app ears in manifest to others Both the cognitive milieu includin g p er

a particular wayWe tend to b elievethatwe are also free to However computer recognition of aective states app ears

cho ose our resp onse to the stimulus An op en and somewhat doable in many cases via the measurementofsentic mo dula

ominous question is are wealways free to do so In other tion Note that I am not prop osing one could measure aective



words can some part of our nervous system b e externally state directly but rather measure observable functions of such

activated to force exp erience of an emotion states These measurements are most likely to lead to success

ful recognition when sub jects do not delib eratel y suppress emo

Anumb er of theorists have p ostulated that sensory feedback

tional expression If one can observe reliable functions of hidden

from muscle movements such as facial is sucient to induce

states then these observations may b e used to infer the states

a corresp onding emotion For example Laird divides p eo

themselves Thus I may sp eak of recognizing emotions but

ple into cueing categories based on whether or not p osturing

this should b e interpreted as measuring observations of motor

themselves in a particular expression induces the corresp onding

system b ehavior that corresp ond with high to one

emotional exp erience Izard overviews some of the evidence for

or more underlying emotions

and against various sensorimotor claims

Despite its immense diculty recognition of expressed emo

Whether or not such sensorimotor inputs can induce emo

tional states app ears to b e much easier than recognition of

tion they app ear to at least b e eectiveinmaintaining and

thoughts In pattern recognition the diculty of the problem

expressing emotion Posture is correlated with expressions of

usually increases with the numb er of p ossibili ties The number

selfesteem A successful scho ol of acting after Michael

of p ossible thoughts you could haverightnow is limitless nor

Chekhov student of Stanislavsky is based on imagining emo

are thoughts easily categorized into a smaller set of p ossibil i

tive scenarios and adjusting ones b o dy p osition in accord with

ties Thought recognition even with increasingl y sophisticated

that emotion

brain imaging techniques mightwell b e the largest inverse

Actors who excel at this metho d strengthen their asso ciation

problem in our world In contrast for a

with their characters emotional state When the b o dys emo

relatively small numb er of simplifying categories for emotions

tional expression eg an angry face and p osture agrees with

have b een commonly prop osed

the cognitive emotion mycharacter is now angry the com

bined emotional exp erience is enhanced I feel angry Con

Basic or prototyp e emotions key issues

sequently its communication to the audience is more p ower

ful Of course these actors adjust their p osture in accord with

Categories or continuum

an initiall y cognitive goal Hence this is an example where

Diverse writers have prop osed that there are from twoto

emotions are initiall y cognitivel ygenerated and the b o dymind

twenty basic or prototyp e emotions See for example p

reinforcementintensies and regulates the exp erience

p The most common four app earing on these lists

The b o dymind reinforcementmay also provide a subliminal

are fear anger and joy Plutchik distinguish ed

way to induce emotion p erhaps by inducing involuntary eye

among eight basic emotions fear anger joy disgust

saccades Although the p otential of such metho ds to induce

and surprise Ortony et al provide a

emotion is unknown the answers to questions like this may

helpful summary of lists of basic emotions in their b o ok



hinge on only a slight willingness to b e op en to inducement

Sometimes these basic emotions are dened to b e essentially

The p ossibili ty of subliminal inducementmayevoke disturb

innate like Damasios primary emotions but there is no con

ing thoughts of p otentially harmful mind and mo o d control

sensus on their denition

or p otentially b enecial mental enhancement and increased af

The actual existence of basic emotional states is disputed by

fective freedom It is not an area to b e entered into without

some authors Leidelmeijer and Stein and Oatley bring

considering b oth negative and p ositive asp ects of howsuchnew

together evidence for and against the existence of basic emo

understanding could b e used As computers develop the ability

tions esp ecially universall y although I distinguish universality

to recognize aect they p otentially may b e used for monitoring

as a separate issue addressed b elow

it for b oth helpful and harmful purp oses

Some authors have b een less concerned with eight orsoproto

To what extent do neural sensorimotor motivational and

typ e emotions and instead refer primarily to continuous dimen

cognitive systems interact in emotion inducement and suppres

sions of emotion such as negative or p ositive emotions Three

sion These are op en research areas and imp ortant for under

dimensions show up most commonly Although the precise

standing b oth how natural helpful emotions are induced and

names vary the two most common dimensions are

how harmful inducement can b e minimized

calmexcited and valence negativep ositi ve The third

dimension tends to b e called control or attention address

ing the internal or external source of the emotion eg contempt

AectivePattern Recognition

or surprise

Although with the aid of new measuring devices we can dis

The problem of not b eing able to precisely dene categories

tinguish many new activitylevels and regions in the brain

is not restricted to emotions It o ccurs all the time in cogni

we cannot at present directly access anothers thoughts or

tion pattern recognition and socalled fuzzy classication

emotions Instead thoughts and emotions are communicated

Nonetheless I think the use of categories greatly simplies

through words gesture music and other forms of expression

recognition and communication of emotional state for b oth p eo

The couplings b etween emotions and these forms are not well

ple and computers A simplifyi ng set of categories chosen for

understo o d What signals to measure how to pro cess these

their relevance to a particular practical domain assists com

signals and howtointerpret them are op en questions These

puters in b eginning the dicult task of aect recognition It is

and many other hurdles must b e overcome to give computers

tting that babies app ear to have a smaller rep ertoire of emo

the ability to recognize aective states

tions than adults

The development of aective computing do es not require res



olution of these theoretical issues although it should aid in their

External in contrast with direct of the brain

investigation In a typical application or context suchasthe

which is known to elicit various emotions



Perhaps this willingness may also b e induced ad innitum piano tutor ab ove recognition of only a small set of emotions

can b e of great b enet Recognition of elements in the set do es An interesting set of tests could involve actors Imagine a

not imply that they are discrete or basic The lack of consensus highlyskil l ed actor trying to play the role of a hop eful character

ab out the existence of precise categories of basic emotions do es when he is feeling anger right b efore the show In order to

not interfere with the ideas I present b elow delib erately express hop e he suppresses his anger or overrides

it with hop e If he is successful onstage in communicating hop e

For aective computing the recognition and mo deling prob

has he merely forgotten his anger so that it will return after

lems are simplied by either the assumption of a small set of

hop e has nished its reign on stage or is there a therap eutic

discrete emotions or the assumption of a small number of di

eect that takes place during this p erformance Measurements

mensions Those who prefer to think of emotions as continuous

of his emotion b efore during and after the p erformance could

can consider the discrete categories to b e regions in a contin

b e studied b oth for understanding the purity of emotions as

uous space or can adopt one of the dimensional frameworks

well as for understanding their therap eutic eects

In a sense the choice of discrete or continuous states is likethe

Although forgetting sounds like a cognitiveactithasto

choice of particles or waves in describing light Ultimatelythe

o ccur b o dily for emotions or the audience will sense conict in

one which is b est dep ends on what you are trying to explain

the actor instead of hop e and think him to b e a bad actor

Either representation continuous or discrete is commonly used

The intensity of the aective communication is a function of

in pattern recognition and is accompanied by helpful analysis

the actors sentic mo dulation voice face p osture and more

to ols

As the actor delib erately brings all these mo des into a consis

Universality

tent expression not only is his communication more eective

but the theory is that he moves himself closer to a pure state of

Much of emotion theory has b een stymied on the issue of

emotion If Clyness theory holds then the purer the emotional

universality regarding the existence of emotion categories In

state the more eective and aective the actors communica

other words if a set of emotions is truly basic then shouldnt

tion will b e

they exist in all humans

Theories such as this examining the purity of emotions and

Like many questions in emotion theory the study of this

of their inuences could b e tested empiricall y with an aective

question is complicated by factors suchashow the emotion

computer that recognizes emotional states as describ ed next

was provoked expressed communicated and lab eled after

all dierent mayormay not use the same words

Mo deling aect

for describing emotive phenomena One of the p otential b ene

How do es the computer represent emotions Obviouslycurrent

ts of aective computing lies in its abilitytomake measure

computers do not have the equivalent of a limbic brain and a

ments conditioned on individu al s and on circumstances aect

cortical brain or the bio chemical washes that connect these

ing them Analytic to ols can subsequently b e used to search for

regions or the pyramidal and nonpyramidal systems and so

universal patterns in the data

forth Computers have b o dies but they are currently not

aective

Pure or mixed

In this section I will suggest b oth computational and rule

The debate in the literature ab out the purity of emotional

based mo dels for representation of emotions This section con

states is another debate where exp eriments could b e conducted

tains technicalitie s whichcanbeskippedby those who are not

with aective computers

engaged in designing systems that p erform emotion analysis

For example Clyness exclusivity principle of emotional

recognition or emotion synthesis prediction and generation

states suggests that wecannotexpress one emotion when

The mo dels will roughly b e divided into three typ es com

we are feeling another eg we cannot express anger when we

putational mo dels for discrete states computational mo dels for

are feeling hop e Clynes emphasized the purity of the basic

emotion spaces and rulebased mo dels

emotional states and suggested that all other emotional states

are derived from this small set of pure states eg melancholy

Discrete aective state mo dels

is a mixture of love and sadness

This section considers p ossible mo dels for the discrete hid

Plutchik said that one can account for any emotion bya

den paradigm of emotional states Figure shows an example

mixture of the principal emotions and that emotions are

of one p ossible mo del the Hidden Markov Mo del HMM This

rarely p erceived in a pure state This idea was captured by

example shows only three states for ease of illustration but

carto onist Johnny Hart and reprinted in in his BC

it is straightforward to include more states such as a state of

carto on illustratin g an example of a mixed emotion seeing

no emotion The basic idea is that you will b e in one state

your longlost dog come b ounding up your freshly p oured front

at any instant and can transition b etween states with certain

sidewalk

probabiliti es For example one might exp ect the probabili ty

The distinctions b etween the views of Plutchik and Clynes

of moving from an interest state to a joy state to b e higher

app ear to b e a matter of intensity and delib erate expression If

than the probabili tyofmoving from a distress state to a joy

you are deeply involved in playing a mournful piece of music

state The actor describ ed in the previous section might circu

you may b e in a pure state of sadness However if you are

late among a hop e state an angry state and a mixed state

thinking ab out lunch while playing the same piece of music

The HMM is trained on observations which could b e any

the measured state will likely not b e as pure

measurements of sentic mo dulation varying with the underly

How do es sentic mo dulation change as a p erson suppresses ing states suchaschanges in voice inection or heartrate The

one stronglyfelt state and tries to feel another Could sentic input at anytimeistheseobservations the output can b e either

measurements help p eople in identifying an emotion they are the state that the p erson is most likely in or it can b e identi

masking such as in the expression of anger to hide fear These cation of an entire HMM conguration thereby recognizing a

are questions that aective computing could address by con pattern of emotional b ehavior

structing mo dels of aective states and giving the computer the In the latter case there would b e a family of congurations

ability to recognize and record observations that correlate with one corresp onding to each useful emotional b ehavior or each

these states individu als characteristics for a given b ehavior In either case

ould b e a set of observations the output a set of

O(V|I) O(V|J) The input w

probabiliti es for each p ossible emotional state Such a proba

bilistic formulation can also provide fuzzy classications of

emotional state where someones state might b e describ ed as y and distress as p erhaps in the BC carto on men

Pr(J|I) jo

tioned ab ove e describ ed for aect recognition can b e used

INTEREST JOY The mo dels Iv

t emotional states and b ehaviors but also Pr(I|I) Pr(J|J) not only to represen

Pr(I|J)

to predict and synthesize them The prediction pro cess is

one of partial recognition rst determine which mo del and

our currentobservations and then see

Pr(I|D) Pr(J|D) parameters b est ts y

what state that system would most likely synthesize next Such

a mo delbased prediction would givealikely outcome but

could not predict with certainty what would actually

Pr(D|I) Pr(D|J) happ en

Notice that emotional state synthesis by the computer in

volves no sentic mo dulation but only the generation in name

of a state or b ehavior We can synthesize samples from a prob

ypical b ehaviors according

Pr(D|D) DISTRESS abilistic mixture mo del to obtain t

to that mo del eg when these nine variables lie in this range

then the prediction is that grief will arise but we cannot make et although they are getting b etter

O(V|D) a computer cry or laugh y

at voice inection and it is a small step to use a synthesized

mo del state to driveavo cal output Synthesis of emotion is a

topic I will revisit in Sect

Figure The state eg Interest Distress or Joy of a p erson

Numerous other to ols from pattern recognition are also likely

cannot b e observed directly but observations whichdependon

to b e useful in aect recognition Articial neural nets can

a state can b e made The Hidden Markov Mo del shown here

p erform a variety of recognition tasks and can function like

characterizes probabiliti es of transitions among three hidden

mixture mo dels hence they should b e useful for emotional

states IDJ as well as probabili tie s of observations measur

state mo deling Neural nets and related mo dels suchasthe

able essentic forms such as features of voice inection V given

MLattice can also mo del certain nonlinear dynamical sys

a state Given a series of observations over time an algorithm

tems Camras has prop osed that dynamical systems the

such as Viterbis can b e used to decide whichsequenceof

ory b e considered for explaini ng some of the variable physio

states b est explains the observations

logical resp onses observed during basic emotions but has not

suggested any mo dels Freeman has mo deled olfaction with dy

namical systems and prop oses the imp ortance of this approach

for mo deling limbic inuences on intention and motivation in

the HMM states do not have to corresp ond to pure emotional

but he has not prop osed any his b o ok So cieties of Brains

states as illustrated in Fig but may corresp ond to even more

computational mo dels for the latter

fundamental buildin g blo cks p erhaps identied by the com

puter as it works to t the data

Continuous aect mo dels and eigenmo o ds

In either case dierent HMMs can b e trained as a function

Instead of assuming discrete states sometimes it is more ap

of environmental or so cial context eg driving a car in the

propriate to start with the data and p erform factor analysis

country vs in the city at hour going out with an old

or an eigenvector decomp osition to discover its underlying di

friend vs on a blind date There could also b e dierent HMMs

mensions In this case one tries not to uncover discrete states

based on timing relative to a hormone cycle or to exam season

but rather continuous axes which describ e the most imp ortant

Hence the probabiliti es states and their connections mayvary

variations in the data

dep ending on a variety of factors ultimately determined bythe

For example in the case of an eigenvector decomp osition

intended use of the aective state recognizer The mo del is

wewould start with a variety of observations of sentic mo dula

exible and can accommo date manyvariables of interest in

tion measurements use these to form a covariance matrix and

emotion theory

then nd the eigenvectors which diagonali ze this covariance

HMMs can also b e adapted to represent complex mixture

The eigenvectors corresp onding to the top eigenvalues could b e

emotions One could design exp eriments to see which mix

kept resulting in a space of an arbitrarily small numb er of di

ture combinations made the b est predictors of individua l emo

mensions Discriminants could b e computed to determine how

tional resp onses For example an HMM of interest in stress

well distances in the resulting eigenspace corresp onded to p er

ful learning situations might represent a causal sequence of

ceived dierences in the corresp onding emotional states

attentionhigharou sal followed by distress and then by relief

The most useful dimensions in a p ersons emotion eigenspace

states The choice of states could b e made by clustering physi

might corresp ond to eigenmo o ds providing buildin g blo cks

ological variables and assigning each cluster its own state

for emotional expressions As new expressions are observed and

pro jected into this space they are describ ed in terms of these

Static mixtures may also b e mo deled and tailored to an indi

underlying eigenmo o ds The eigenmo o ds might corresp ond to

vidual and to their context by explicit mixture mo dels such

either either pure or mixture emotions the eigenspace mo del

as the clusterbased probability mo del of Popat and Picard

works with either interpretation

In such a case highdimensio nal probability distribution s are

Note that in mo deling any signal can b e decomp osed into ba learned for emotional states or their mixtures based on the val

sis comp onents therefore one can always nd subcomp onents ues of or functions of the values of the physiological variables

Arousal

verned by emotions and p eople attach cathexis emotional

high go

energy to things Freud called ob jects Quoting from Werb os

Mutilated face

Ski jump

According to his Freuds theory p eople rst of all

learn causeandeect asso ciations for example they

may learn that ob ject A is asso ciated with ob ject

B at a later time And his theory was that there is

ow of emotional energy If A causes abackwards

B and B has emotional energy then some of this

energy ows back to A If A causes B to an extent

kwards ow of emotional energy from Valence W then the bac

neg pos

B back to A will b e prop ortional to the forwards rate kpropagationIf A causes B then

Cute baby That really is bac

you havetondaway to credit A for B directly

If you wanttobuildapowerful system you need a

kwards ow

Cemetary bac

The use of some form of feedback is a signicant part of most

computer learning metho ds to day Usually the feedback is via

a set of p ositive or negative examples provided by a user A

Flowers

more p owerful learning system might augment the p ositive and

negative examples with aective resp onses from the user

ords now the computer is trying to learn from the

low In other w

user Supp ose it tries to learn your preferences for art to hang

in your livingro om As you browse a of images lo oking

Figure Two dominant dimensions of emotion here used to

for suitable pictures it detects your resp onse to dierent im

lo cate the aective resp onse of certain photo contents as p er

ages After you have indicated your favorites to it it could try

the studies of Lang

to infer which art features eg artist color texture content

and whichsentic features eg facial expression heartrate skin

resp onse were the b est predictors of what you liked It might

even b e able to asso ciate certain categories of images with cer

a socalled mixture even if the signal is purely of one kind

tain categories of your aective resp onses In the future it

Consequently from a mo deling p ersp ective the theoretical is

mightwatch for o ccurrences of those same features to help

sue of pure vs impure emotional states is not problematic

you save time lo cating things you like or lo cating things that

Eigenspaces constructed from measurements of sentic mo du

suit your present mood

lation over the course of a p ersons daytoday activities should

Such an aective learning agentmight beavaluable assistant

b e compared to the spaces found by factor analysis in emotion

to your interior decorator or to the p ersonalized environments

theory The spaces could b e estimated under a varietyofcondi

of the future that mightfavorably customize your hotel con

tions to b etter characterize features of emotion expression and

ference or retreat surroundings to win your business

their dep endencies on external eg environmental and cogni

Avariation on the HMM ab ove could also b e used to in

tive eg p ersonal signicance factors Tra jectories can also

corp orate aective feedback The recentPartially Observable

be characterized in these spaces to study the dynamic asp ects

Markov Decision Pro cesses are set up to give a reward asso ci

of emotion

ated with executing a particular action in a given state

An example of the most common dimensioned space for emo

These mo dels p ermit observations at each state which are

tions is shown in Fig Such a space might b e more useful than

actions hence they could incorp orate not only autonomic

an HMM if the aective computer is trying to gauge how p os

measures but also observations of your b ehavior

itive or negative users feel ab out a new software pro duct for

Rulebased emotion mo dels and their

example

limitations

Given one of these dimensionspace mo dels trained on motor

outputs corresp onding to known emotional states then features

The fo cus in the previous section was aect recognition us

of unknown motor outputs can b e collected and used with signal

ing mathematical mo dels which in some cases can also b e

pro cessing to ols such as maximum a p osterior decisionmakin g

used for emotion synthesis and prediction Alternatively non

to recognize or classify a new emotion Because the recognition

mathematical rulebased mo dels maybeused

of emotional state can b e set up as a pattern recognition prob

Some work on rulebased mo dels of emotion has b een done

lem a variety of techniques are available for its solution

in the AI community where the emphasis has b een on writing

scripts to pro duce states that are lab eled with various emotions

See Pfeifer for a nice overview The AI emphasis has

Cathexis in computing

b een on rulebased synthesis of cognitivestatesin the computer

Although most computer mo dels for imitating mental activ

which receive emotional names for example if the computer

ity do not explicitly consider the limbic resp onse a surprisingl y

has the goal of getting you to ll out a form and you do not ll

large numb er implicitly consider it Werb os writes that his

it out after it rep eatedly asks you to do so then it mightenter

original inspiration for the backpropagation algorithm exten

a state called frustrated

sively used in training articial neural networks came from

Although I have not yet seen work with rulebased mo dels

trying to mathematically translate an idea of Freud

for emotion recognition and prediction this would b e p ossible

given a go o d synthesis mo del coupled with signal pro cessing Freuds mo del b egan with the idea that human b ehavior is

and pattern recognition to convert the human signal into the What do I app ear to b e feeling What

language of the script or vice versa The sentic mo dulation

am I feeling Where am I

measurements would rst undergo translation from the numer

Lets pause and consider another scenario involving human af

ical to symb olic form or b etter the AI mo dels would adapt to

fect analysis and synthesis

learn the sentic mo dulation language In either case it will still

Supp ose you hear the winning lottery numb er and recall that

b e imp ortant to use some of the techniques describ ed ab ove

it is your friends favorite numb er and she has played it every

Therefore the mo dels of Pfeifer and other cognitive

week for the last ten years You cognitively predict rulebased

motivational generators of emotion such as the prop osed

synthesis that when she walks in she will b e elated Your

scripts of Wierzbicka in which could b e programmed p o

own physiological system in anticipation mightsynthesize a

tentially not only could run a script to generate a cognitive

corresp ondingl y excited emotional state

emotional state but also could identify whichcomponents of

If she walks in lo oking terribly distraught and saying she

the script are satised by a set of observations of a human In

heard the winning numb er then you feel surprise recognizing

this way the script could recognize someone elses emotional

that your prediction is wrong Then you cognitively learn

state or predict a forthcoming emotion

that she cannot nd her ticket After you talk to her and learn

Recognition analysis mo dels in general do not havetobe

that her ticket blew away in the wind then you maysynthesize

capable of synthesis This holds true for mo deling emotion

a new emotion such as

allowing emotion to b e recognized by a computer without b eing

The conict which o ccurred b etween the synthesis mo dels

synthesized Howeveritisinteresting to p oint out that even

prediction and the analysis mo dels recognition is not only ac

with emotion analysis and synthesis a rulebased system is still

ceptable but also ags an interesting event one whichislikely

likely to b e defective

to generate a state of higher arousal p erhaps priming the phys

Consider again the frontallob e patients of Damasio These

iological system for a faster resp onse in case of a lifethreatening

patients can recognize emotions and can predict synthesize

emergency or lifeenhancing opp ortunity

that they should feel certain emotions such as horror given

A computer could also p otentially ag these events although

certain events such as seeing a picture of a head b eing chopp ed

a computer has apparently never b een given a fastresp onding

o They have cognitive analysis and synthesis capabili ties

physiologi cal system for the purp ose of avoiding lifethreatening

but still cannot b ehave normally What they are missing is the

emergencies or to pursue lifeenhancing opp ortunities The

physiologi cal resp onse that must work in concert with cognitive

physical correlates in a computer would likely take the form

evaluation

of a separate parallel emotion synthesis mo del capable of gen

Based on the ndings of Damasio it seems to b e true that

erating interrupts of priority sucienttooverride or hijack

not only are emotion analysis and synthesis an essential part

the cognitive system if needed

of a future intelligent computer but two kinds of synthesis are

Coupling b etween synthesis and analysis mo dels has some

needed the counterparts of b oth the cognitiveandthephysio

imminent practical application s For example in an existing

logical systems The cognitive comp onent mightbetwell bya

environmentsuch as ALIVE a synthesis mo del could adjust

set of laws or rulebased mo dels However these laws would not

each software agents p osture facial expression and gestures

b e enough relying only on them to simulate emotion would b e

to reect its synthesized emotional state The state itself

a mo dern form of Descartes error in separating b o dy and mind

could also b e resynthesized via the cognitive scripts mentioned

Rather the computer must also have a second comp onenta

earlier as the agent recognizes the states of humans interacting

b o dily comp onent This comp onentwould not follow the log

with it in this virtual world

ically neuronalinspi red mechanisms of present computers but

In suchaninteractive system a human might roleplay

would aim to function as the bio chemical ow that o o ds the

dierent metho ds of emotional interaction Hence the system

brain activating and fo cusing attention determining salience

b ecomes a testb ed for new strategies or games involving aec

biasing decisions and driving thoughts This b o dily comp onent

tive communication More imp ortantly p erhaps such a test

in the computer could b oth interact with and if needed over

bed provides a safe and controllable environment for exploring

ride the laws and rules of the reasoning comp onent marchto

the nature and developmentofemotionalintelligence which

its own tune so to sp eak acting as the equivalent of our human

according to Goleman can b e learned

songs

Another imp ortant example of coupling synthesis and anal

If these two comp onents are not present then not only can

ysis is in the case of a sp eakingimpa ired human relying on

we not exp ect to see creative computers but we can exp ect

aspeechsynthesizer Such p eople are usually limited to one

them to remain decisionimp ai red The I have made is

inectionless digital voice Control over aect in synthetic

of rulebased machines to patients like those of Damasio who

sp eech is particularly imp ortant for these p eople With

are highly rational but unable to function normally in so ciety

an aectivewearable computer that senses sentic mo dulation

However Damasios patients at least have primary emotions it

and couples it to a synthesis mo del parameters for voice in

is the secondary ones they are missing Therefore this b o d

ection could b e synthesized and fed directly into their sp eech

ily comp onent I am prop osing must do more than synthesize

synthesizer The result would allow a sp eakingimp aired indi

a primary emotional state for the computer it must interact

vidual to for example express anger with her voice by merely

as do secondary emotions in a human with the rulebased rea

feeling angry The coupling could allow nonverbal expression

soning of the machine The rulebased decisions and emotional

to drive the synthetic verbal expression

biases must op erate in concert

If sentic mo dulation synthesis is coupled to the inverse prob

To summarize the emotion comp onent in the computer lem of sentic mo dulation analysis recognition then in the

should not b e a mere script for generating states or mere ran ultimate mo deling scenario the emotion synthesis mo del could

dom p erturbations to bias an otherwise rulebased system In synthesize the aective resp onses of a b o dy separated from its

stead the emotion comp onentmust b e closely coupled into the brain as in Dennetts fantastic story Where am I Like

machines intelligen ce able to learn and able to run the show wise to the extent that sensorimotor stimuli aect and reinforce

from time to time the cognitive state of emotion a b o dys sentic mo dulation could

after analysis drive the brain into a corresp onding cognitive is the brain and central nervous system of the spaceship Dis

state Although this scenario is unthinkabl e with real humans covery The computer who prefers to b e called Hal has

and imp ossible with foreseeable technology it could p otentially p erceptual abilitie s whichemulate those of a human Hal is a

b e simulated with the aid of aective computers Suchadmit true thinking machine in the sense of mimicking b oth cog

tedly farout scenarios are nonetheless intriguing for the inves nitive and emotional functions Humans who interact with Hal

tigation of b etter theories ab out cognition and recognize his emotional abilities as evinced in this exchange

mindb o dy interactions between a rep orter and crewman of the Discovery

Rep orter One gets the sense that he Hal is capable

Things Better Left Unexplored

of emotional resp onses When I asked him ab out his

abilities I sensed a sort of

Im wondering if you mightbehaving some second

thoughts ab out the mission Hal in the movie

Crewman Well he acts like he has genuine emotions

ASpace Odysseyby Stanley Kubrick and Arthur C

Of course hes programmed that way to make it easier

Clarke

for us to talk with him But whether or not he has

real feelings is something I do not think anyone can

A curious student p osed to me the imp ortant infrequently

truly answer

asked question if aective computing is not a topic b etter left

unexplored byhumankind At the time the worst p otential

As the movie unfolds it b ecomes clear that the articulate

danger in my mind was that computers mightbeusedtomali

Hal is capable of b oth expressing and p erceiving emotion

ciously induce emotions My inclinati on was to resp ond Emo

tion manipulati on for b oth go o d and bad purp oses is already

I feel much b etter now

commonplace lo ok at music cinema marketing and p olitics

Lo ok Dave I can see youre really upset ab out this

Wouldnt aective computers at least help educate p eople so

that they can b e more in control

But Hal go es b eyond expressing and p erceiving emotion In

However questions like this do not havesuch easy answers it

the movie Hal app ears to have fear of b eing disconnected as

is oversimplifying to say that improving p eoples understand

indicated not just by his sp oken expression but also byhis

ing is the b est solution Although the Marie Curie quote which

reactivebehavior The novel indicates that Hal exp eriences

leads o this pap er most succinctly summarizes where I lean

internal conict b etween truth and concealment of truth This

it would b e irresp onsibl e to not consider the p otential harm as

conict results in Hal killing all but one of the crewmen

well as go o d that can come from this new area of research Af

Hal is more than a thinking and feeling machine not only

fective computing has p otential for b oth go o d use and for mis

can he pass the Turing test but also he can kill the p erson

use Aective computers convert p ersonal emotional expression

administering it

into bits making this information sub ject to all the problems

The ctional message has b een rep eated in many forms and

asso ciated with digital media includi ng issues of broadcast

is serious a computer that can express itself emotionally will

privacyandeven copyright

some day act emotionally and the consequences may b e tragic

Ob jection to development of emotional computers based

A dilemma

on fear of the consequences parallels the Heads in the Sand

There are many b enecial reasons to pursue the development ob jection one of nine ob jections playfully prop osed and refuted

of aective computers Nonetheless their development raises byTuring in to the question Can machines think But

the following dilemma fear of the consequences is to b e balanced against the practical

Can we create computers that wil l recognize and express af b enets that should app ear given the imp ortance of the lim

fect exhibit creativity intel ligentproblem solving and bic emotional brain role in thinking Cytowic talking ab out

and never bring about harm by emotional actions how the limbic system eciently shares comp onents suchas

To elab orate I will briey presenttwo scenarios from artists attention memory and emotion notes

viz ction writers and movie pro ducers who have presaged sci

Its ability to determine valence and salience yields

entists in considering what may happ en when computers receive

a more exible and intelligent creature one whose

emotions Subsequently I will discuss more imminent concerns

behavior is unpredictable and even creative

The rst scenario comes from Asimovs The Bicentennial

Today with the everincreasing information available to ma

Man Asimov sub jects his aective rob ots to three laws

chines and computer software agents it is more imp ortant

of b ehavior to prevent them from bringing harm to p eople His

than ever for a computer to b e given the ability to determine

laws put human life ab ove the selfpreservation of the rob ot

valence and salience It is also commonly agreed that creativ

However his laws are not infallib le one can prop ose logical

ity and exibili ty are necessary comp onents of intelligence

conicts where the rob ot will not b e able to reach a rational

However howtoconstructsuch qualities based on AIstyle rules

decision based on the laws Indeed his rob ots could b e ren

or without gratuitous randomness has so far eluded scientists I

dered completely ineectual by getting into states where they

think that the construction of such qualities will require mecha

could not reach a decision due to rulebased conicts With

nisms that duplicate b oth limbic abilities and cortical abilities

out a separate emotion system that can determine saliency and

Ihave argued a variety of reasons whyintelligent machines

ultimately override rules a lawbased rob ot is severely handi

will need emotions The issue is no longer is emotion neces

capp ed in its decisionmaki ng ability not to o unlike Damasios

sary for intelligen ce nor even howmight machines analyze

frontallob e patients

and synthesize emotions The evidence supp orts an answer of

A somewhat more sinister scenario of an emotional machine



yes to the rst question and I have explained mechanisms for

o ccurs in the science ction classic A Space Odyssey

pro ceeding with the second Instead the issue at hand is do we

A HAL computer b orn January in the novel

wanttogive computers the abilitytohave emotions the kind



that can hijack their reasoning systems and ultimately pro duce

The lm was based on the screenplayby Kubrick and

Clarke Clarkes novel came out afterward in the unpredictable b ehavior that is the hallmark of creativity

Can we build a machine and give up control over it In other

Computer Cannot Can

words are we willing to give it to makevaluebased

express aect express aect

emotional decisions Such a machine would b e guided bybut

ultimately not constrained by the ethics or mores whichwe

give it This machine would b e free for example to denyour

Cannot p erceive aect I II

role in its creation

Can p erceive aect III IV

Unemotional but aective computers

Mans greatest p erfection is to act reasonably no less

Table Four categories of aective computing fo cusing on

than to act freely or rather the two are one and the

expression and recognition

same since he is the more free the less the use of

his reason is troubled by the inuence of

Gottfried Wilhelm Von Leibniz

in aective computing further discussion of these is outside the

scop e of this pap er

Although expressing and recognizing aect are imp ortant for

Clearly computers could b enet from development of ethics

computerhuman interaction building emotion into the motiva



morals and p erhaps also of religion These developments are

tional b ehavior of the computer is a dierent issue In fact the

imp ortanteven without the amplier of aect But computer

word emotional when it refers to p eople or to computers usu

scientists are not equipp ed to deal with these issues on their

ally connotes an undesirable reduction in rationalityDoyou

own much less to create machines with these abiliti es Sci

wanttowait for your computer to feel interested b efore it will

entists understanding of these great human issues has barely

listen to you Or p erhaps it would b e b enecial if p eople who

moved past that of Mary Shelleys Frankenstein Aective com

bombard you with email had to make the content suciently

puting needs input from outside the cognitive and computer

interesting b efore your computer would accept it

Interestingly in the p opular series Star Trek The Next Gen

eration the aable android Data was not originall y given

Four cases for the present

emotions although he was given the ability to recognize them

in others Datas evil android brother Lore had an emotion

The imaginary scenarios ab ovetake us far from presentreality

chip and his daughter develop ed emotions but was to o imma

as provocative as they may b e they remain science ction For

ture to handle them Although b oth Data and his brother had

the rest of this pap er I would like to fo cus on more immediately

the ability to kill Data evidently could not kill out of malice A

available cases of aective computing

later episo de fo cused on the maturity pro cess needed for Data

Four such cases are summarized in Table Of course these

to deal with the replaced emotion chip The pro cess parallels

are not all p ossible cases for example I omitted the rows Com

what we exp ect with the development of emotional intellige nce

puter cancan not induce the users emotions as it is clear

in humans

that computers and all media already inuence our emotions

One might argue that computers should not b e given the abil

the op en questions are how delib erately directly and for what

ity to kill But it is to o late for this as anyone who has own

purp ose I also omitted the columns Computer cancan not

in a commercial airplane acknowledges Alternatively p erhaps

act based on emotions for the reasons describ ed ab ove The



computers with the p ower to kill should not have emotions

ethical and philosophi cal problems to b e addressed b efore the

or they should at least b e sub ject to the equivalent of the psy

developmentofsuch emotionally based computers go b eyond

chological and physical tests which pilots and others in life

the scop e of this pap er these p ossibili ties are not included in

threatening jobs are sub ject to

Table or addressed in the application s b elow

The fact remains giving computers the ability to recognize

This leaves the four cases describ ed b elow

express and have emotions is only the b eginning of greater

issues of how should they use these emotions I will touchon

I Most computers fall in this categoryhaving less aect

this again b elow in Sect

recognition and expression than a dog Such computers

Anumb er of so cial and philosophi cal questions are raised

are neither p ersonal nor friendly

by aective computers that have emotions Should we follow

I I This category aims to develop computer voices with nat

the human paradigm and allowmachines to express an emo

ural intonation and computer faces p erhaps on agentin

tion that is dierent from what they are exp eriencing inter

terfaces with natural expressions When a disk is put in

nally Should the aective computer have separate channels of

the Macintosh and its diskface smiles users may share

expression likewehave in our pyramidal and nonpyramidal

its momentary pleasure Of the three categories employ

systems Alternatively should we give computers abilities hu

ing aect this one is the most advanced technological ly

mans do not have such as the abilitytohavemultiple pure

although it is still in its infancy

emotions or is this imp ossibl e without it having parallel self

awareness systems multiple consciousnesses multiple p ossible

I I I This category enables a computer to p erceiveyour aective

parallel p ersonalities What should the nature of the com

state whichIhave argued is imp ortant ifitistoadjust

puters selfawareness b e Should we the maker of the emo

its resp onse in ways that might for example makeita

tional machine b e the only one allowed to p eer inside and see

b etter teacher and more useful assistant This category

its true internal state These are but a few of the issues raised

allays the of those who are uneasy with the thought

of emotional computers in particular if they do not see



Although I refer to a computer as having emotions I in

the dierence b etween a computer expressing aect and

tend this only in a descriptive sense eg lab eling its state

b eing driven by emotion

of having received to o much conicting information as frus

tration I doubt electronic computers will have feelings as



humans do but I recognize the parallels in this statementto The latter raises particularly intriguing questions eg

debates ab out machines having consciousness Should they fear only their makers maker

What is the relevant set of emotions for this applicatio n IV This category maximizes the sentic communication b e

tween human and computer p otentially providing truly

How can these b est b e recognizedexpressed mo del ed

p ersonal and userfriendly computing It do es not im

What is an intelligent strategy for resp onding to or using

ply that the computer would b e driven by its emotions

them

Aective symmetry

Aective computing primarily equips the computer to handle

In crude videophone exp eriments we wired up at Bell Labs over

the rst two issues The third issue requires domainsp ecic

a decade ago my colleagues and I learned that p eople preferred

b eyond aect analysis and synthesis and into issues

seeing not just the p erson wewere talking to but also the image

not just of emotional intelligence but of combining emotional

they were seeing of us Indeed this symmetry in b eing able

intelligen ce with the general goals of the application

to see at least a small image of what the other side is seeing is

As one example consider again the aective piano tutor sce

now standard in video teleconferencing

nario An appropriate set of states for the computer tutor to

It is my opinion that a similar form of symmetry should b e

recognize might include distress interest and pleasure The

considered for computers in categories I I I and IV ie infer

computer might recognize these states from watching your face

hidden emotional state based on physiologi cal and b ehavioral

gestures p osture or measuring other resp onses of your auto

observations Your computer should let you see what it is

nomic nervous system Aective computing in this applicatio n

reading as your state

primarily enables pattern recognition of human emotional ex

pression

More precisely aectiveinteraction with a computer can give

a p erson direct feedback that is usually absentinhuman inter However how should the computer resp ond when you make

action The hidden state mo dels prop osed ab ove can reveal an error the rst time The Nth time When you do something

their state to us indicating what emotional state the computer well How should it adapt its resp onses to optimize your learn

has recognized Of course this information can b e ignored or ing exp erience Merely adapting to always please the user is

turned o but my guess is p eople will leaveiton naive and conjures up the somadep endent so ciety of Huxleys

BraveNewWorld Indeed the answers to these questions

This feedback not only helps debug the development of these

go b eyond aective computing into questions of learning epis

systems but is also useful for someone who nds that p eople

temology and more

misunderstand his expression Such an individual may never

get enough precise feedback from p eople to knowhowtoim In some cases this third issue op ens up so cial and ethical

questions For example supp ose that an automobiles aective

prove his communication skills in contrast his computer can

provide ongoing p ersonal feedback steering wheel senses that you are unusually stressed or angry

Many automobile accidents are caused by p eople who are an

However as computers receive the ability to synthesize inter

gry or upset Recognition of your dangerous state is the job of

nal emotional states should these b e observable by other com

aective computing howtheintelligent vehicle should resp ond

puters Or p ossibly observable only through some imp erfect

is p otentially a legal issue

expressions as is the case for humans What are the implica

I will oer ideas b elowastohowaective computing might

tions for communication if emotion is unambiguous It seems

b e used to help develop new theories to deal with this third

that imitating the human paradigm where computers could

issue However success with the third issue will require more

mask their emotions from each other would lead to interesting

than emotional intelligence

developments for example a software agentthatwas esp ecially

talented at negotiation

Applications of Aective Computing

Consider the issue of whether there should b e an asymmetry

giving humans unequivo cal access to read internal synthesized

Lets move past the philosophi cal issues ab ove and into exam

computer states Asymmetry since I dont think computers

ples of imminent practical applications The rest of this pap er

will achieve recognition of human emotional states In

considers p otentially b enecial scenarios for applying aective

if Hals emotional state were observable at all times by

computing All of them assume the involvement of one or more

his crewmates then they would haveseenthathewas afraid

human participants who willingly participate in aective com

as so on as he learned of their plot to turn him o If they had

munication

observed this fear and used their heads then the tragic

The examples b elow come mainly from cases I I I I I and IV

storyline would not work

in Table where computers can p erceive andor express aect

Instead Hal illustrates the case of a computer that could

The scenarios assume mo dest success in relating observations

hide its emotional state b etter than most p eople Sp eaking in

of an individu al s sentic mo dulation with at least a few appro

the spirit of its ctional character it could have its feelings

priate aective states which is the most dicult new hurdle for

hurt and not let the humans know I exp ect computers will

aective computing research However the hurdle has already

indeed somedayhave the ability to feign emotions b etter than

b een leap ed in some of the cases describ ed b elow

p eople If so a p ossible preventive step for disasters and mis

communication would b e to prohibit the machine from hiding

Entertainment

its emotions Such a constraint however conicts with the goal

Why I do so well is I induce emotion Sean D

of giving it the full freedom required in true creativity

Tucker American aviation artist

A distinction aect recognition vs

One of the worlds most p opular forms of entertainmentis

intelligent resp onse

large sp orting events whether it is an outdo or air show the

Olympics the World Series Sup er Bowl or anynumber of

Before pro ceeding with applications in the next section there

other large gatherings there is an excitement in the air when

is an imp ortant distinction to b e made regarding the scop e of

fans come together to watch athletes p erform

aective computing Just ab out any application involvin g an

aective computer will require attention to the following three One of the that p eople receive from these events

issues whether or not their side wins is the opp ortunity to freely ex

press intense emotions I would not b e surprised if the stereo b enets studies on other forms of attentiongetting entertain

typ ed unemotional American male might need these events mentmightreveal other b enets

psychological l y a stadium is one of the few places where such Ihaveobserved that similar b enecial eects o ccur for p eo

a p erson can yell and jump up and down releasing emotion ple also in roleplaying scenarios whether during group ther

without app earing childish Emotional expression is acceptable apy where a p erson acts out an emotional situation or during

for sp orts athletes and sp ectators Indo ors in frontofaTV roleplaying games such as the p opular computer MUDs and

sp orts game a mature adult can yell and jump with emotional interactive communities where one is free to try out new p erson

expression that otherwise would b e disdained Emotions and alities A friend who is a Catholic priest once acknowledged how

their expression are a healthy asp ect of entertainment muchheenjoyed getting to playanevilcharacter in one of these

roleplaying games Suchentertainmentprovides a healthy and

Do you feel like I do Do you feel likeIdo

safe way to expand ones emotional dynamic range

Peter Frampton

Successful entertainment need not draw forth a roar of emo

tional resp onse a lot of waggling of reectors or a lot of pushing

Although Im not a fan of Peter Framptons music I am

of buttons as in the interactive theaters coming so on from Sony

still moved by the tremendous resp onse of the crowd in his live

Nonetheless even in the quietest symphony hall a successful

recorded p erformance where he asks them this question rep eat

p erformer can sense how the audience is resp onding and is in

edly with increasingl y mo died voice Each time he p oses the

turn aected by their resp onse

question the crowds excitement grows What causes sucha

Audience resp onse could b e captured byavariety of aec

tremendous emotional resp onse from a crowd Are they mind

tive things by cameras that lo oked at the audience by active

less fans who would resp ond the same to a mechanical rep eating

programs they hold in their hands bychair arms and byoors

of the question or to a rewording do you think like I do

that sense Such aective sensors would add a new avor of in

Or is there something more fundamental in this crowdarousal

put to entertainment providing dynamic forms that comp osers

pro cess

mightweaveinto op eras that interact with their audience

I recently participated in a sequence of interactive games

For example the o ors in the intermission gathering spaces

with a large audience SIGGRAPH Orlando where we

mightbelive comp ositions waiting to sense the mo o d of the

without anycentralized co ordination started playing Pong on

audience and amplify it with music The environment itself

a big screen by ipping in front of a camera p ointed at us

might b ecome a new musical instrument p erhaps likeoneof

from b ehind a p opsicle stick that had a little green reector

Machovers hyp erinstruments but equipp ed to sense aect

on one side and a red reector on the other One color moved

directly augmenting the mo des of expression available to the

the Pong paddle up the other down and so on the audience

p erformer

was gleefully waggling their sticks to keep the ball going from

In general audience appraisal would lo ok not only at the

side to side Strangers grinned at each other and p eople had

p eople but also at the media content to distingui sh for exam

fun

ple the viewers sadness due to story content eg the death of

Pong is p erhaps the simplest there is and yet

Bambis mom and the viewers unhappiness due to other fac

it was signicantl y more pleasurable than the more challengin g

tors p ossibly a degraded color channel or garbled soundtrack

submarine steering adventure that followed on the interac

If the aective sensors were wearable and p erhaps seeing every

tive agenda Was it the rhythmic pace of Pong vs the tedious

thing you see See for examples of these sensors then they

driving of the sub that aected our engagement After all

might correlate visual exp eriences with heart rate respiration

rhythmic iambs lift the hearts of Shakesp eare readers Was it

and other forms of sentic mo dulation

the fastpaced unpredictabil ity of the Pong ball or Pong cat or

Aective computers provide a new set of to ols b oth for con

other character it changed into vs the predictable errors the

structing new forms of entertainment and for constructing new

submarine would makewhenwe did not steer correctly What

theories of what makes it succeed

makes one exp erience pleasurably more engaging than another

Expressivecommunication

Clyness selfgenerating principle indicates that the inten

sity of an emotional state is increased within limits bythe

The power of essentic form in communicating and

rep eated arrhythmic generation of essentic form Clynes

generating a sentic state is greater the more closely

has carried this principle forward and develop ed a pro cess of

the form approaches the pure or ideal essentic form

sentic cycles whereby p eople in a controlled and voluntary

for that state Seventh Principle of Sentic Commu

manner may exp erience a sp ectrum of emotions arising from

nication

within The essence of the cycles is supp osedly the same as

Clynes argues that music can b e used to express emotion

that which allows music to aect our emotions except that in

more nely than any language But how can one master this

music the comp oser dictates the emotions to you Clynes cites

nest form of expression The master cellist Pablo Casals ad

evidence with extensivenumb ers of sub jects indicatin g that the

vised his pupils rep eatedly to play naturally Clynes says he

exp erience of sentic cycles pro duces a varietyoftherapeutic

came to understand that this meant to listen inwardly with

eects

utmost precision to the inner form of every musical sound and

Go o d entertainmentmayormay not b e therap eutic but

then to pro duce that form precisely Clynes illustrates with

it holds your attention Attention mayhave a strong cog

the story of a young master cellist at Casalss house playing

nitive comp onent but it nds its home in the limbic system

the third movementoftheHaydn cello concerto All the atten

as mentioned earlier Full attention that immerses and pulls

dees admired the grace with whichheplayed except Casals

you in b ecomes apparentinyour face and p osture Aective

computers might measure these resp onses to dierent forms of Casals listened intently No he said and waved his

entertainment providing researchers with signals that can b e hand with his familiar denite gesture That must

correlated with other measures of mental and physical health b e graceful And then he played the same few bars

b enets Just as studies involving test groups on meditation a and it was graceful as though one had never heard

form of fo cused attention haveshown certain improved health grace b efore a hundred times more graceful so

that the melted in the hearts of the p eople such as skin conductivity temp erature and pulse all observa

who sat there and listened tions whichmaybecombined to identify emotional state An

aective mouse could collect nger pressure information to

Clynes attributes the p ower of Casalss p erformance to the

determine the values of your resp onses while you cruise the

purity and preciseness of the essentic form The purer the emo

picture libraries or virtual museums Al

tional state the purer its expression and communication In

though none of this information should b e a forced broadcast

expression teaches Clynes faithfulness to the purest inner form

it could b e an option available for the user like the abilityto

pro duces the b est results

use a videophone with or without the camera capp ed

With aective recognition the computer music teacher might

Enco ding aective information along with a text message

not only try to hold your interest longer to help you learn more

could tell an audio receiver what aect to use in reading the

but it might also give feedbackasyou develop preciseness of ex

message to its recipient Aective bits could also b e used to

pression This emotional through measuring es

set the expression on a talking head that delivers news to

sentic form p erhaps via nger pressure fo ot pressure or func

your lowbitrate video receiver Moreover aective state in

tions of inspiration and expiration as you breathe could help

formation can b e transmitted to the widest variety of media

you compare asp ects of your p erformance that havenever b een

visual auditory text and deco ded according to the abilities

measured or understo o d b efore

of that receiver As technologists try to enable the fullest band

Recently Clynes has made signicant progress in this

width human communication in limited bandwidth systems it

area giving a user control over such expressive asp ects as pulse

is essential to extract the key bits of information In human

note shaping vibrato and timbre Clynes recently conducted

communication the aective bits are often key



a Musical Turing test to demonstrate the ability of his new

A helpful analogy is to consider the prociency with which

sup erconductor to ols In this test hundreds of p eople lis

someone who knows you well can read your emotions We are

tened to seven p erformances of Mozarts sonata K Six of

often wrong ab out rst impressions but as you get to know

the p erformances were by famous pianists and one was bya

someb o dy you b ecome increasingl y adept at guessing what lies

computer Most p eople could not discern which of the seven was

b eneath their expression A p erson you knowwell maywalk in

the computer and p eople who ranked the p erformances ranked

and say go o d morning with a particular tone that tells you

the computers as second or third on average Clyness com

they are in a hurry but happytoseeyou Someone you dont

puters p erformances whichhaveplayed to the ears and hearts

knowwell maysay go o d morning with the exact same tone

of many master musicians demonstrate that we can identify

and you mayinterpret it quite dierently

and control meaningful expressive asp ects of music often called

What is really communicated when you say go o d morning

the nest language of emotion

or share other socalled small talk with someone you see reg

ularly The words have virtually no information content given

Expressive mail and small talk

their rep eated o ccurrences Might it b e that the primary role

Although emotional states may b e subtle in their mo dulation

of small talk is that of communicating aect

of expression they are not subtle in their p ower to communi

Intelligent coworkers adapt their resp onses to you dep ending

cate and corresp ondingly to p ersuade When sentic mo dula

on the aect you express via small talk or other communication

tion is missing misunderstandi ngs o ccur Consider the tremen

Intelligent computers should b e given the same opp ortunity

dous reliance of many p eople on email that is currently limited

p erhaps via an aectively sp oken go o d morning exchanged

to text Most p eople who use email have found themselves mis

during a login session

understo o d at some p oint their comments received with the

Vo calFacial Communication

wrong tone

Voice inection is a subtle but p owerful to ol for communicat

By necessity email has had to develop its own set of sym

ing not only the mornings mo o d but also imp ortant emotions

b ols for enco ding tone emoticons such as and turn

such as anger grace and empathy Inection can also signal

your head to the left to recognize the smileys However these

interest or disinterest Without doubt inection adds avor to

icons are limited consequentlymuch aectless email has re

our sp eech and content to its message Even in telling a joke

sulted in a loss of pro ductivity as p eople exp end energy trying

everyone knows its how you tell it that determines its success

to undo misunderstandi ngs or as p eople exp end time trying to

Avariety of features of sp eech are mo dulated by emotion

word their email more carefully When aect communication is

these may b e divided into the three categories of voice quality

most imp ortant then p ersontop erson contact carries the most

utterance timing and utterance pitchcontour Murray and

information email presently carries the least

Arnott provide a recent review of these features Although

Although it is often desirable to delib erately limit emotional

virtually no work seems to have b een done on computer analysis

expression say during card games or business negotiations it

of aect in voices several features have b een demonstrated for

is almost never desirable to b e forced to do so by the available

synthesizing intonation in computergenerated sp eech

medium To free email from this restriction to ols that recog

With a suitable aectivevoice computers can communicate

nize and express aect could augment text with other mo des of

in a more natural and so cial waywithhumans Monotonous

expression suchasvoice face or p otentially touch In addition

voice mail recordings and voicereminder systems could vary

to intonation and facial expression recognition currentlowtech

their voices from daytodaylikeahuman voice varies Such

contact with keyb oards could b e augmented with simple atten

variation would render them more pleasant and could b e used

tion to typing rhythm and pressure as another key to aect

to more quickly ag imp ortant and urgent information

The new ring mouse could p otentially pick up other features

Another form of aective computing that has already met



with some success involves facial expression recognition Faces

Although Turing eliminated sensory auditory visual tac

app ear to b e the most imp ortant means for visual communi

tile olfactory taste expressions from his test one can imag

cation of emotion Emotionmo deled faces can b e used to give

ine variations where each of these factors is included eg

computers graphical faces which mimic the emotive expressions

music faces force feedback electronic noses and comestible

comp ositions identied by Ekman making the computer faces seem more

human Several categories of human facial expression can b e derstanding causes of arousal one of the key dimensions of af

recognized by computers b oth from still images and from fect Arousal excitedcalm has b een found to b e a b etter pre

motion images the latter which is more reliable dictor of memory retention than valence pleasuredisp leas ure

Image descriptions given in Fig indicate asso ciations of

The enco ding of facial expression parameters may

arousal with image content

also provide a simultaneousl y ecient and meaningful descrip

tion for video compression two attributes that satisfy imp or

In fact nding digital photographs having a particular

tant criteria for future co ding systems Instead of sending

mo o d was the most frequent request of advertising customers

over a new picture each time the p ersons face changes you

in a study of image retrieval made with the Ko dak Picture Ex

need only send their basic emotion faces once and up date

change Sub ject and action content whichwere most fre

with descriptions of their emotional state and any slightvari

quently requested for editorial purp oses can also b e a p owerful

ations

contributor to mo o d in a photo

Wehave recently built some of the rst to ols

Filmvideo

that enable computers to assist humans in annotating video at

taching descriptions to images that the p erson and computer

A lm is simply a series of emotions strung together

b oth see Instead of the user tediously entering all the de

with a plot though ippant this thought is not far

scriptions by hand our learn which usergenerated

from the truth It is the lmmakers job to create

descriptions corresp ond to which image features and then try

moods in such a realistic manner that the audience



to identify and lab el other similar content

will exp erience those same emotions enacted on the

screen and thus feel part of the exp erience Ian Aective computing can b e coupled with learning systems

suchasthatofMinka and Picard to b egin to identify not

Maitland EmmyAward winning director and editor

only whichcontent is most salientorinteresting but also which

It is the job of the director to create onstage or onscreen a

emotions tend to b e evoked by the content Successful learning

mo o d that provokes a desired aect in the audience A director

algorithms for contentbased similaritymay also b e able to learn

or writer adjusts the characters in the movie or script until they

examples of aect or mo o d similarity

feel rightuntil they communicate the intended message

In fact unlike trying to search for a shot that has a par

and its emotions to the user An aective computer assistant

ticular sub jectaction content aective annotations esp ecially

might help the novice director discern what is not rightwhen

in terms of a few basic emotions or a few dimensions of emo

it do esnt feel right for example is the conguration of the

tion could provide a relatively compact and salient index for

set or the lighting in conict with what typically evokes the

retrieval of data For example p eople may tend to gasp at the

intended mo o d

same shots that guy is going to fall o the cli Shots

Sometimes expressions of mo o d in lm can b e easily quali

could b e lab eled initiall y byahuman with descriptions such

ed lighting from b elow to create an eerie eect for example

as thrilli ng The computer can later learn from the human

However determination of precisely what constitutes an essen

which visual features b est predict the most useful annotations

tic form in dierent media is p o orly understo o d in general The

For example instead of annotating this is a sunnydaytime

forms by which emotions are communicated are an op en area

shot of a student getting his diploma and jumping o the stage

for research

the human might annotate this shot of a student getting his

Despite a lack of understanding of how emotion is commu

diploma and jumping makes p eople grin The latter is an

nicated there is undoubtably a p ower humans have to transfer

aective annotation Of course although the latter example

genuine emotion we sometimes say emotions are contagious

indicates a joyful shot for most viewers it will not provokea

Clynes suggests that the purer the underlying essentic form

grin for everyone an example is the mother whose son would

the more p owerful its communication This p ower to purely

have b een at that graduation if he were not killed the week

transfer emotion exists not just from p erson to p erson but also

b efore In other words the cognitive and emotional state of

through external forms such as lm

the viewer interacts with what is p erceived to pro duce the nal

aect

Skip ahead to the interesting part

These sorts of complicating factors will not b e easy to ad

My primary research for the last decade has fo cused on

dress Although aective annotations like sub jectaction an

helping computers see as p eople see with all the unknown

notations will not b e universal digitizin g them will still help

and complicated asp ects human p erception entails One of the

reduce time humans have to sp end searching for the right scene

newest applications of this research is the construction of to ols

Both kinds of annotation are p otentially p owerful we should b e

that aid consumers and lmmakers in retrieving and editing

learning how b oth are p erceived and including them in digital

video Example goals are asking the computer to nd more

audio and visual libraries

shots like this or to fast forward to the dinosaur scene

Amuch harder but related goal is to teach a computer to

Emotions in learning

make a long story short How do es one summarize hours of

video into a form pleasurable to browse Howdoweteachthe

Fascinating Sp o ck Star Trek

computer which parts lo ok b est to extract Finding a set of

A learning episo de might b egin with curiosity and fascina

rules that describ e content for retrieving more shots like this

tion As the learning task increases in diculty one mayexpe

is one diculty but nding the content that is the most in

rience frustration or Learning maybeaban

teresting ie involving aect and attention is a much greater

doned b ecause of these negative feelings If the learner man

challenge These new challenges are ones which computer sci

ages to avoid or pro ceed b eyond these emotions then progress

entists are not equipp ed to address but where crossdiscipli ne

eorts b etween cognitive science emotion theory and computer



science are sorely needed

Computers have a hard time learning similarity so this

The problem of lo cating a rememb ered scene or an image

system tries to adapt to a users ideas of similarity whether

p erceptual semantic sub jective or otherwise with particularly interesting content is also the problem of un

may b e rewarded with an Aha and accompanying neurop ep aective expression The online collection of this information

tide rush Even the unemotional Sp o ck frequently exclaimed could not only lead to more relevantfeedback to the user but

up on learning something new that it was fascinating could also b e of great use in consumer satisfaction studies not

to mention in the development of more pleasurable and eective

Dr Barry Kort a mentor of children exploring and construct



learning exp eriences

ing scientic worlds on the MUSE and a volunteer for nearly

a decade in the Discovery Ro om of the Boston Museum of Sci

Learning is almost always bidirectional for example the

ence says that learning is the quintessential emotional exp e

teacher usually learns things ab out the students attention and

rience Kort says his goal is to maximize intrigue the

preferences during their interaction Aective computer pro

fascination stage and to minimize anxiety

grams particularly software agents whichwatch their users

Whatever her strategy the go o d teacher detects imp ortant

carefully could b egin to learn the humans preferences much

like a trusted assistant However in the short term likethe aective cues from the student and resp onds dierently b e

cause of them For example the teacher mightleave subtle

dog walking the p erson and the new b o ots breaking in your

feet we will likely nd ourselves doing as much adapting to the hints or clues for the studenttodiscover thereby preserving

the learners sense of selfprop elled learning

agents as they do to us During this mutual learning pro cess it

mightbefavorable if the agent at least paid attention to your

Whether the sub ject matter involves delib erate emotional ex

frustration or satisfaction

pression suchasmusic or a nonemotional topic suchassci

ence the teacher that attends to a students interest pleasure

For example the agent might notice our resp onse

and distress is p erceived as more eective than the teacher

to to o much information as a function of valence plea

that pro ceeds callously The b est human teachers know that

suredispleasu re with the content To o many news stories tai

frustration usually precedes quitting and knowhowtoskill

lored to our interests might b e annoying and an o ccasional

fully redirect or motivate the pupil at such times They get to

insertion of humor stories might lead to greater tolerance for

know their student and howmuch distress that studentcan

the necessary but less pleasurable reading Our tolerance may

withstand b efore learning breaks down

b e describ ed not only as a function of the dayofweek or time

Computers that cannot recognize human aect are severely

of day but also as a function of baseline mo o d measured that

handicapp ed In contrast with observations of your emotions

morning The agent learning to distingui sh which features of

the computer could learn to resp ond to you more like the b est

information b est please the user while meeting his or her needs

human teachers giving you oneonone p ersonalized guidance

could adjust itself appropriately User friendly and p ersonal

as you explore Educational toys could havenumerous learning

computing mightmove closer to their stated meanings

strategies built in changing their resp onse as the child shows

dierent levels of interest

Aectiveenvironments

Aect has b een largely ignored in theories of learning p er

Sometimes p eople likeachange of environment sometimes it

haps b ecause it is hard to measure Like in all activities that

drives them crazy These resp onses apply to all environments

demand mental p erformance weknow emotion is a determin

not just your building home or oce but also your com

ing factor Hebb showed the classic invertedU curve in

puter software environment with its lo ok and feel the inte

relating p erformance to arousal Performance is lowest when

rior of your automobile and all the appliances with whichyou

the sub ject is awaking or when the sub ject is aroused to the

surround and augmentyourself What makes you prefer one

point of emotional disturbance p erformance is optimized at an

environment to another

intermediate state of arousal

Ho op er identied three kinds of resp onses to architec

Now supp ose we could assess emotional communication dur

ture which I think hold true for all environments cogni

ing a learning episo de as a set of parameters much like health

tive and p erceptual hearsee symb olic and inferential

researchers measure parameters such as heartrate to develop

thinkknow and aectiveandevaluative feellike

b etter theories of athletic training Perhaps there is some ana

Perceptual computing primarily computer vision and audition

log to b e found in learning theory which corresp onds to the

and articial intellige nce have b een largely concerned with mea

ruleofthumbinphysical conditionin g theory sustain of

suring information in the rst and second categories Aective

maximum heartrate for twentyminutes for optimal condition

computing addresses the third

ing Although certainly a successful learning theory for mind

In trying to understand what designs bring longterm sat

or for sp orts would b e more complicated than this my p ointis

Stewart isfaction in his recent b o ok Building s that Learn

that emotion variables could playakey role in developing new

Brand emphasizes not the role of buildings as space but their

theories of learning Such theories might b egin to address issues

role in time Brand applauds the architect who listens to and

such as not merely trying to optimize the students

learns from p osto ccupancy surveys But he further cautions

but orchestrating a balance b etween diculty frustration and

that b ecause these are written or verbal rep orts and the lan

accomplishment satisfaction

guage of emotions is so inexact these surveys are limited in

Analyzing emotion b oth in the sender and receiver might

their ability to capture what is really felt Brand notes that

also lead to progress in understanding the b enecial contagious

surveys o ccur at a much later time than the actual exp erience

eects of emotion such as the lifechanging impact of those

and hence may not recall what the visitors or inhabitants liked

sp ecial teachers who stir our interest in a topic

most

When using a computer p eople often nd themselves try

In contrast measuring sentic resp onses of p eople in the build

ing to learn Whether it is learning a new piece of software

ing could tell you how the customers feel when they walk in your

or directly using an educational program the exp eriences are

bank vs into the comp etitors bank capturing those imp ortant

not generally considered pleasant What if computer programs

rst impressions Surveys of willing newcomers who express

could like the piano tutor ab ove pay attention to the users

their feelings when they enter your building for the rst time



might b e recorded by an aective computer

Email o cmusenetorg or p ointyour Gopher or Web

After b eing in a building awhile your emotions in that space

browser at cyb erionmusenetorg for information howto

connect are no longer likely to b e inuenced by its structure as that

has b ecome predictable Environmental factors suchastem emotion What is the space of transformations that an essentic

p erature lighting sound and decor to the extent that they form could b e put through without it losing its identity and

change are more likely to aect you Alive ro oms or alive purity

furniture and appliances that sense aective states could ad

If such forms could b e identied and measured wecould

just factors suchaslighting natural or a variety of articial

search the visual of the worldwide web for instances

choices sound background music selection active noise can

of them and then test p eoples resp onses to the images to see if

cellation and temp erature to either matchorstimulate an ap

the images communicate the same emotion It seems it might

propriate mo o d Your digital disc jo ckey might help suggest

b e easy to nd conicting examples esp ecially across cultural

music of a particular mo o d Computer please adjust the en

educational and so cial strata Howevertomy knowledge no

vironment for a p eaceful ambience at our party tonight

suchinvestigation has b een attempted yet

Despite the lackofcertainty ab out a visual form for emotion

Aesthetic pleasure

in pictures there is something in a picture or piece of music that

aects p eople emotionally the question is what is the nature of

Art do es not think logically or formulate a logic of

this something Note that even if suchaninvestigation results

b ehavior it expresses its own p ostulate of If in

in an ambiguous resp onse it would still not imply that Clyness

science it is p ossible to substantiate the truth of ones

hyp othesized internal essentic forms do not exist as seen in the

case and prove it logically to ones opp onents in art

following scenario

it is imp ossibl e to convince anyone that you are right

One of Cytowics synesthetic patients saw lines pro jected in

if the created images have left him cold if they have

front of her when she heard music Her favorite music makes

failed to win him with a newly discovered truth ab out

the lines travel upward If a computer could make the synes

the world and ab out man if in fact face to face with

thetic asso ciations she makes then presumably it could help her

the work he was simply b ored Andrey Tarkovsky

nd new music she would like by lo oking for pieces where the

lines wentupward However although synesthetes have b een

As creation is related to the creator so is the work of

found to make the same crossmo dal asso ciations throughout

art related to the lawinherent in it The work grows

life dierent synesthetes mayhave dierent asso ciations For

in its own way on the basis of common universal

an individu al synesthete rules might b e discovered to predict

rules but it is not the rule not universal a priori

these aesthetic feelings Across synesthetes the rules for the

The work is not law it is ab ove the law Paul Klee

shap e of the essentic form may b e dierent Nonetheless there

are consistentinternal forms that an individual synesthete can

use What ab out internal forms for the rest of us

Psychology so ciology ethnology and other sciences

Cytowics idea is that p erception in all of us passes through

have attempted to describ e and explain artistic phenomena

the limbic system Unlike most p eople synesthetes can p erceive

Manyhave attempted to understand what constitutes b eauty

the form while it is passing through this intermediate stage

and what leads to an aesthetic judgement The elusiveness and

Perhaps the limbic system is where Clyness hyp othesized es

complexity of aesthetics is due in part to the fact that aect

sentic form resides Measurements of internal essentic forms

plays a primary role

may somedaycontribute to ob jective recognition of the aes

Computers have b een not only aectblind but aesthetically

thete Just as lines going up and at an angle coo ccurred with

blind Consider a scenario where a computer is assembling a

music the synesthetic woman liked so certain essentic forms

presentation for you In the not to o distant future the com

and their puritymay b e asso ciated with preferences of other

puter will b e able to search digital libraries all over the world

art forms

lo oking for images and video clips with the contentyou request

With the rapid development of image and database query

eg colorful scenes of Bill Gates with a cereal b owl Supp ose

to ols we are entering a time where one could browse for exam

it nds hundreds of shots that meet the requirements you gave

ples of such forms hence this area is now more testable than

it for content What you would really like at that p ointisfor

ever b efore But lets again set aside the notion of trying to

it to narrowdown the set to just the go o d ones to showyou

nd a universal form to consider a more p ersonal scenario

Howdoyou teach it what is go o d Can something b e mea

sured in a picture sculpture building piece of music or ower

Personal taste

arrangement that will indicate its b eauty and app eal

You are strolling past a store window and a garment catches

Hidden forms

your eye My friend would love that pattern you think

Later you lo ok at a bunch of ties and mo ck How could any

Clynes has suggested that essentic forms capture emotion

body like these

in art He has identied visual essentic forms in a number of

Peoples preferences dier wildly in clothing They may rea

great works of art for example the collapsed form of grief

son ab out their taste along dierentlevels quality of the gar in the PietaofMichelangelo and the curved essentic

ment its stitching and materials its practicality or feel its

form of reverence in Giottos The Clynes

p osition in the fashion sp ectrum style and price and p ossibly suggests that these visual forms which match his measured

even the reputation and expressive statement of its designer A

ngerpressure forms are indicative ofatrueinternal essentic

buyer knows however that these features cannot simply b e op form Moreover shap e is not the only parameter that could

timized into a garment that all will nd maximally app ealing

communicate this essentic form color texture and other fea

it is absurd to assume the existence of a universal predictor of

tures maywork collectivel y

what is b est

The viewp oint that we could nd some combination of prim

Although you mayormay not equate garments with art itive elements in a picture that corresp onds to an emotion is

an analogy exists b etween ones aesthetic judgmentinthetwo debated On one hand it seems that if you could nd a basic

cases Artwork is evaluated for its quality and materials how essentic form in one image that you could yank it out of that

well it will t where you want to display it its feel its p osition image into some imagemanipul atio n software and construct a

in the world of art style and price its artist and her or his new image around it one that do es not communicate the same

expressiveintent Just as for clothing the nding of a universal function that inuenced her one way or the other Suchlaws

aesthetic predictor may not b e p ossible play an imp ortant role However none of them are inviolab le

However selecting something for someone you knowwell What seems to o ccur is a nearly ineable recognition a p er

something you think they would like is commonly done We ceptual aha when all is right

not only recognize our own preferences but we are often able

Although we can measure qualities of ob jects of the space

to learn anothers

between them and of many comp onents of design we cannot

Moreover clearly there is something in the app earance of the

predict how these alone will inuence the exp erience of the ob

garment or artwork that inuences our judgement But what

server Design is not solely a rulebased pro cess and computer

functions of app earance might enable the computer to reach

to ols to assist with design only help explore a space of p os

the same judgement Perhaps if the lines in that printwere

sibiliti es Todays to ols eg in graphic design incorp orate

straighter it would b e to o b oring for you On the other hand

principles of physics and computer vision to b oth judge and

you might treasure the b old lines on that table

mo dify qualities such as balance symmetry and disorder

There are many problems with trying to nd something to

But the key missing ob jective of these systems is the goal of

measure b e it in a sculpted style painting print fabric or

arousing an exp erience in the user arousing to provokeat

ro om decor Ordinary pixels and lines do not induce aesthetic

tention interest memory and new exp eriences For this the

feelings on their own unless p erhaps it is a line of Klee used

system must b e able to recognize the users aect dynamically

to create an entire gure Philosophers such as Langer have

as the design is changed This assumes the user is a willing

taken a hard stance in seeking to understand pro jectivefeeling

participant not suppressing their feelings ab out the computers

in art

design suggestions

Aesthetic success may b e said to b e communicated via emo

There is however no basic vo cabulary of lines and

tion you like something b ecause it makes you feel go o d or

colors or elementary tonal structures or p o etic

because you like to lo ok at it or it inspires you or makes you

phrases with conventional emotive meanings from

think of something new this brings you joy A design solves a

which complex expressive forms ie works of art

problem you have and you feel relief Eventually it brings you

can b e comp osed by rules of manipulatio n

to a new state that is more satisfying than the one you were in

Despite Langers claim neither do we exp erience aesthetic

Although the computer do es not presently knowhowtolead

pleasure without the pixels lines notes and rhythms Moreover

a designer to this satised state there is no reason it could

Clynes do es seem to have found a set of mechanisms from which

not b egin to store sentic resp onses and gradually try to learn

complex expressive forms can b e pro duced as evidenced in his

asso ciations b etween these resp onses and the underlying design

musical Turing test this is further collab orated with his recent

comp onents Sentic resp onses have the advantage of not having

studies indicating the signicant role of comp osers pulses in

to b e translated to language which is an imp erfect medium for

appreciation of music

reliably communicating feedback concerning design In fact

Just thinking of a magnicentpainting or piece of music do es

frequently it is precisely the sentic resp onse that is targeted

not usually arouse the same emotions as when one is actually

during design

exp eriencing the work but it may arouse similar fainter emo

Aective computing will playakey role in gathering infor

tions Beethoven still comp osed some of the greatest music in

mation for improving our aesthetic understanding esp ecially in

the world after he could no longer hear Aesthetic feelings ap

areas suchasentertainment and design

p ear to emerge from some combination of physical p erceptual

and cognitive forms

Aectivewearable computers

Eorts to give computers recognition of what we think is

The most dicult thing is that aective states are

b eautiful should aid our understanding of this p erplexing and

not only the function of incoming sensory signals ie

universally imp ortant problem Like all computer recognition

visual auditory etc but they are also the function of

problems this one will probably require huge sets of examples

the knowledgeexp eri ences of individ ual s as well as

of things that we do and do not like Additional ly I exp ect

of time What you eat in the morning can inuence

it will b e improved as the computer learns to directly incor

the wayyou see a p oster in the afterno on What

p orate aective feedback from us The computer will need to

you read in tomorrows newspap er maychange the

infer which features are common to those examples that welike

wayyou will feel ab out a magazine page youre just

and distinguish these from features common to the negative

lo oking at now Suguru Ishizaki

examples

With knowledge of aective preferences the computer can

The ab ovepeekinto the unpredictable world of aesthetics

cruise the networks at night helping shop for clothes furni

emphasizes the need for computers which p erceive what you

ture wallpap er music gifts artwork and more Online mu

p erceive and which recognize p ersonal resp onses as you change

seum tours that are b ecoming available on the worldwide web

them In the most p ersonal form these are computers that

could suggest to you additional collections by guessing what

could accompanyyou at all times

you might like after observing your reaction to what you have

The idea of wearing something that measures and commu

already seen Aective computers p otentially provide more p er

nicates our mo o d is not new the mo o d rings of the s

sonal service tailored to your everchanging interests

are probably due for a fad rerun and mo o d shirts are supp os

edly available now lo cally Of course these armpit heattocolor

Design

transformers do not really measure mo o d Nor do they compare

You can not invent a design You recognise it in the

to the clothing jewelry and accessories we could b e wearing

fourth dimension That is with your blo o d and your

lap el communicators a watch that talks to a global network a

b ones as well as with your eyes David Herb ert

network interfacethatiswoven comfortably into your jacket or

Lawrence

vest lo cal memory and a micropro cessor in your b elt a minia

Haveyou asked a designer how she arrived at the nal de ture video camera and holographic displayonyour eyeglasses

sign Of course there are design principles and constraints on and more

Wearables may fulll some of the dreams esp oused by Clynes correct a bad habit after back surgery p erhaps byasubtle

when he coined the word cyb org Wearable computers nudge in a helpful place Wearables that measure other phys

can augmentyour memory any computer accessible informa iological resp onses can help you identify causes of stress and



tion available as you need it or your reality zo oms in when anxiety and howwell yourbodyisrespondingtothese Such

you need to see from the back of the ro om Your wearable cam devices might b e connected to medical alert services a commu

era could recognize the face of the p erson walking up to you nity of friends and family or p erhaps just a private slowdown

and remind you of his or her name and where you last met and attend to what youre doing service providing p ersonal

Signals can b e passed from one wearable to the other through feedbackforyour private reection I sense more joyinyou

your conductive Bo dyNet A handshake could instantly tonight

pass to my online memory the information on your business

With willing participants and successful aectivecomput



card Note that these examples are not science ction all of

ing the p ossibil iti es are limited only by our imagination Af

these functions or their basic technologies have b een realized in

fectivewearables would b e communication b o osters clarifying

present research in the MIT Media Lab oratory

feelings amplifying them when appropriate and leading to

An Orson Scott Card science ction novel features a sen

imaginativenewinteractions and games Wearables that de

tient b eing named Jane that sp eaks from a jewel in the ear of

tect your lackofinterest during an imp ortant lecture might

Ender the hero of the storyTo Jane Ender is her brother as

switchinto a recording mo de for you taking notes while as

well as dearest friend lover husband father and child They

suming that your mind is wandering Games where players

keep no secrets from each other she is fully aware of his mental

don aective computers might add p oints for Your

world and consequently of his emotional world Jane cruises

wearable might encourage you during a workout I sense anger

the universes networks scouting out information of imp ortance

reduction Wearables with networked agents might help p eo

for Ender She reasons with him plays with him handles all

ple reach out to contact those who want to b e contacted not

his business and ultimately p ersuades him to tackle a tremen

just based on common interests as expressed through internet

dous challenge Jane is the ultimate aective and eectivecom

news groups but also through common mo o d For example

puter agent living on the networks and interacting with Ender

it might recognize your emotional state could b e improved by

through his wearable interface

striking up a conversation with someone with common interests

Although Jane is science ction agents that roam the net rightnow and it mightletyou know whos available that would

works and wireless wearables that communicate with the net

enjoy this opp ortunity

works are current technology Computers come standard with

Of course you could remap your aective pro cessor to change

cameras and microphones ready to see our facial expression

your aective app earance or to keep certain states private In

and listen to our intonation People who work with computers

oces one might wishtoreveal only the states of no emotion

generally have more physical contact with computers than they

disgust pleasure and interest Aective computing do es not

have with p eople computers are in a unique p osition to sense

enforce emotion recognition or expression on anyone Instead

our aective state

it provides an opp ortunity for additional communication one

The bandwidth humans have for communicating thoughts

which can b e used for b oth go o d and bad purp oses with hop eful

and emotions to each other can also b e available for communi

emphasis on the former

cating with computer agents My wearable agentmight b e able

New data needed

to see your facial expression hear your intonation and recog

nize your sp eech and gestures Your wearable mightfeelthe

Despite a numb er of signicant eorts emotion theory is in

changes in your skin conductivity and temp erature sense the

its infancyPeoples emotional patterns dep end on the context

pattern of your breathing measure the change in your pulse

in which they are elicited and so far these have b een limited to

feel the lilt in your step and more all in an eort to b etter un

lab settings Problems with studies of emotion in a lab setting

derstand you You could cho ose to whom your wearable would

esp ecially with interference from cognitive so cial rules are well

communicate these p ersonal clues of your emotional state

do cumented The ideal study to aid the developmentofthe

theory of emotions is reallife observation recently b elieved to

Iwant a mo o d ring that tells me my wifes mo o d

b e imp ossible

b efore I get home Walter Bender

However as in the examples ab ove a wearable aective com

If wewere willing to wear a pulse respiration or moisture

puter that attends to you during your waking hours could p o

monitor the computer would have more access to our motor ex

tentially notice what emotions you express as well as a variety

pression than most humans This op ens numerous new commu

of conditionin g factors suchaswhatyou eat what you do what

nication p ossibili tie s such as the message p erhaps encrypted

you see hear etc Computers excel at amassing information

to your sp ouse of howyou are feeling as you head home from

and their ability to analyze and identify patterns is b eing im

the oce The mo o d recognition might trigger an oer of infor

proved rapidly Given a willingness on the part of the wearer

mation such as the news via the network that the lo cal orist

to share this information with researchers a wealth of imp or

just received a delivery of your sp ouses favorite protea A mo o d

tant data could b e gathered for furthering theories of learning

detector might make suggestions ab out what fo o ds to eat so

intelligen ce p erception diet exercise communication mental

called mo o d fo o ds and collect information continuously

health and more

through the diet contributing to our ongoing understanding of

bio chemical inuences on mo o d

Summary

Aectivewearables oer p ossibili tie s of new health and med

Emotion was identied by Donald Norman in as one of

ical research opp ortunities and applications Medical studies

the twelve ma jor challenges for cognitive science In this

could move from measuring controlled situations in labs to

pap er I have argued that emotions can no longer b e considered

measuring more realistic situations in life A jacket you cho ose

a luxury when studying essential rational cognitive pro cesses

to wear that senses your p osture mightgently remind you to

 

It could also pass along a virus See for a discussion of emotions and stress

instead recent neurological evidence indicates they are neces the human qualities that so imp ortantly should b e asked of re

sary not only in human creativityandintelligen ce but also in searchlike this It is with I acknowledge Bil Burling

rational human thinking and decisionmaki ng I have suggested Manfred Clynes Allan Collins Richard E Cytowic Dave De

that if computers will ever interact naturally and intelligentl y Maris Peter Hart Jennifer HealeyMitch Kap or Fanya Mon

with humans then they need the ability to at least recognize talvo Alex Pentland Len Picard Josh Wachman and Alan

and express aect Wexelblat for helpful and encouraging comments on early drafts

of this manuscript

Aective computing is a new area of research with recent

results primarily in the recognition and synthesis of facial ex

pression and the synthesis of voice inection However these

References

results are just the tip of the iceb erg a varietyofphysiologi

R S Lazarus Emotion Adaptation New York NY

cal measurements are available whichwould yield clues to ones

Oxford University Press

hidden aective state Moreover these states do not need to b e

universal in their expression for a computer to recognize them

RECytowic The Man Who Tasted Shapes New York

Ihave prop osed some p ossible mo dels for the state identica

NY G P Putnams Sons

tion treating aect recognition as a dynamic pattern recog

R E Cytowic Synesthesia a union of the sensesNew

nition problem More research is needed to discover whichof

York SpringerVerlag

these to ols coupled with which measurements b oth of the p er

son and their environment give reliable indicators of aect for

C E Izard Four systems for emotion activation Cog

an individ ual

nitive and noncognitive pro cesses Psychological Review

Given mo dest success recognizing aect signicant leaps in

vol no pp

b oth theory and practice are p ossible Aect plays a key role

O Kro eger and J M Thuesen TypeTalk at WorkNew

in understanding phenomena such as attention memory and

York Delacorte Press Bantam Doubleday Dell Publishin g

aesthetics Ihave describ ed over fty p ossible applicatio ns

Group Inc

in learning information retrieval communications entertain

ment design health and human interaction where aective

R E Cytowic The Neurological Side of

computing would b e b enecial In particular with wearable

Cambridge MA MIT Press To App ear

computers that p erceive context and environmentaswell as

A R Damasio Descartes Error Emotion Reason and

physiologi cal information there is the p otential of gathering

the Human BrainNewYork NY GossetPutnam Press

data for advances in cognitive and emotion theoryaswell as

for improving our understanding of factors that contribute to

human health and wellb eing

P N JohnsonLaird and E Shar The interaction b e

Although I have fo cused on computers that recognize and

tween reasoning and decision making an intro duction

p ortray aect I havealsomentioned evidence for the imp or

Cognitionvol pp

tance of computers that would have emotion Emotion is not

A M Turing Computing machinery and intellig ence

only necessary for creative b ehavior in humans but neurologi

Mindvol LIX pp Octob er

cal studies indicate that decisionmaki ng without emotion can

b e just as impaired as decisionmakin g with to o much emotion

Aristotle The Rhetoric of Aristotle New York NY

Ihave used this evidence to suggest that buildin g computers

AppletonCenturyCrofts An expanded translation

that makeintelligent decisions may require building computers

with supplementary examples for students of comp osition

that have emotions

and public sp eaking byLCooper

Ihave also prop osed a dilemma that arises if wecho ose to

D R Hofstadter and D C Dennett The Minds I Ban

give computers emotions Without emotion computers are not

tam Bo oks

likely to attain creative and intellig entbehavior but with to o

D Goleman Emotional Intel ligence Bantam Bo oks

much emotion we their maker may b e eliminated by our cre

ation Although this scenario is far aeld and included mostly

C I Nass J S Steuer and E Taub er Computers are

as a worstcase p ossibility it is imp ortant that researchers dis

so cial actors in Proceeding of the CHI Proceedings

cuss p otential hazards of aective computing together with its

Boston MA pp April

p otential b enets

C Nass and S S Sundar Is humancomputer interac

Ihave suggested a wide range of b enets if we build comput

tion so cial or paraso cial Submitted for Publication

ers that recognize and express aect The challenge in buildin g

Stanford SRCT Pap er

computers that not only recognize and express aect but which

have emotion and use it in learning and making decisions is a

S R Covey The Seven Habits of Highly Eective People

challenge not merely of balance but of wisdom and spirit It is

New York Fireside Simon and Schuster

a direction into whichwe should pro ceed only with the utmost

N Negrop onte Being Digital New York Alfred A Knopf

resp ect for humans their thoughts emotions and freedom

Acknowledgments

E D Scheirer Personal Communication

It is not often that one reads b o oks that redirect their course of

M Lewis Selfconscious emotions American Scientist

research I am indebted to Manfred Clynes Richard E Cy

vol pp JanFeb

towic and Antonio Damasio whose pioneering studies into

R Plutchik and H Kellerman eds Emotion Theory Re

emotion argued p ersuasively for more attention on this topic

search and Experiencevol Academic Press

ultimately capturing myown Thanks to Beth Callahan Len

Series of selected pap ers

Picard and Ken Haase for enjoyable science ction p ointers

M Clynes Personal Communication and to Dave DeMaris and Manfred Clynes for reminding me of

K Popat and R W Picard Novel clusterbased proba G Duchenne The Mechanism of Human Facial Expres

bility mo dels for texture synthesis classication and com sion New York NY Cambridge University Press

pression in Proc SPIE Visual Communication and Im Reprinting of original dissertation

age Procvol Boston pp Nov

R E Cytowic Personal Communication

A Sherstinsky and R W Picard Orientationsensiti ve

D M Clynes Sentics The Touch of the EmotionsAnchor

image pro cessing with MLattice A novel nonlinear dy

PressDoubleday

namical system in IEEE First Int Conf on Image Proc

R S Lazarus A D Kanner and S Folkman Emotions

vol I I I pp Nov

A cognitivephenomeno log ica l analysis in Emotion The

L A Camras Expressive development and basic emo

ory Research and Experience R Plutchik and H Keller

tions Cognition and Emotionvol no and

man eds vol Theories of Emotion Academic Press

W J Freeman Societies of Brains A Study in the Neuro

scienceofLove and Hate Hillsdale NJ Lawrence Erlbaum P Ekman and W Friesen Facial Action Coding System

Consulting Press

Asso ciates Publishers

I R Murray and J L Arnott Toward the

P J Lang The emotion prob e Studies of motivation and

of emotion in synthetic sp eech A review of the literature

attention American Psychologistvol no pp

on human vo cal emotion J Acoust Soc Amvol

pp Feb

R O Duda and P E Hart Pattern Classication and

K Leidelmeijer Emotions Anexperimental approach

Scene Analysis WileyInterscien ce

Tilburg University Press

C W Therrien Decision Estimation and Classication

R B Za jonc On the primacy of aect American Psy

New York John Wiley and Sons Inc

chologistvol pp Feb

PWerb os The brain as a neuro controller New hy

M Lewis Ch The of human emotions in

p otheses and new exp erimental p ossibili ties in Origins

Handbook of Emotions M Lewis and J Haviland eds

Brain and SelfOrganization K H Pribram ed Erl

New York NY pp Guilford Press

baum

H G Wallb ott and K R Scherer Assessing emotion by

E J Sondik The optimal control of partially observable

questionnai re in Emotion Theory Research and Expe

Markov pro cesses over the innite horizon Discounted

rience R Plutchik and H Kellerman eds vol The

costs vol pp March

Measurement of Emotions Academic Press

April

A Wierzbicka Dening emotion concepts Cognitive

W S LovejoyAsurvey of algorithmic metho ds for par

Sciencevol pp OctDec

tially observed Markov decision pro cesses Annals of Op

J D Laird J J Wagener M Halal and M Szegda Re

erations Researchvol pp

memb ering what you feel Eects of emotion on mem

C C White I I I A survey of solution techniques for

ory Journal of Personality and vol

the partially observed Markov decision pro cess Annals

no pp

of Operations Researchvol pp

D Gelernter The Muse in the MachineOntario The

T Darrell Interactive vision using hidden state decision

Free Press Macmillan Inc

pro cesses PhD Thesis Prop osal

M Minsky The Society of Mind New York NY Simon

Schuster

R Pfeifer Articial intelligence mo dels of emotion

in Cognitive Perspectives on Emotion and Motivation

W R Hess The functional organization of the dien

V Hamilton G H Bower and N H Frijda eds vol

cephalon New York Grune Stratton

of Series D Behavioural and Social Sciences Nether

K Levitt Personal Communication

lands pp Kluwer Academic Publishers

M Chekhov On the Technique of Acting New York

P Maes T Darrell B Blumburg and A Pentland The

Harp erCollins Pub Chekhov Preface

ALIVE system Fullb o dy interaction with autonomous

and Afterword byMPowers

agents Proceedings of Apr

R Plutchik A general psycho evolution ary theory of

N Alm I R Murray J L Arnott and A F Newell

emotion in Emotion Theory Research and Experience

and aect in a communication system for

R Plutchik and H Kellerman eds vol Theories of

nonsp eakers Journal of the American Voicevol

Emotion Academic Press

pp March IO So ciety Sp ecial Issue People

A Ortony G L Clore and A Collins The Cognitive

with Disabili ties

Structure of Emotions Cambridge MA Cambridge Uni

D Dennett Where am I in Brainstorms Philo

versity Press

sophic Essays on Mind and PsychologyMontgomery

N L Stein and K OatleyedsBasic EmotionsHove

VT Bradford Bo oks

UK Lawrence Erlbaum Asso ciates Bo ok is a sp ecial

I Asimov The Bicentennial Man and Other StoriesGar

double issue of the journal Cognition and EmotionVol

den City NY Doubleday Science Fiction

No

A C Clarke A Space Odyssey New York NY New L R Rabiner and B H Juang An intro duction to hidden

American Inc based on the screenplay Markov mo dels IEEE ASSP Magazine pp Jan

by S Kubrick and A C Clarke

S Brand How buildings learn what happens after theyre D R Hofstadter Godel Escher Bach an Eternal Golden

builtNewYork NY Viking Press Braid New York NY Basic Bo oks Inc Publishers

A Tarkovsky Sculpting in Time Reections on the Cin G W V Leibniz Monadology and Other Philosophical

Essays Indianap oli s The BobbsMerrill Company Inc

ema London Fab er and Fab er ed by K Hunter

Blair Essay Critical Remarks Concerning the General

Part of Descartes Principles Translated by P

PKlee The Thinking Eye New York NY George Wit

Schrecker and A M Schrecker

tenb orn Edited by Jurg Spiller Translated byRalph

Manheim from the German Das bildnerische Denken A H Brave new world Brave new world

revisited New York NY Harp er Row

S K Langer Mind An Essay on Human Feelingvol

S Tucker Media Lab Spring Collo quium

Baltimore The Johns Hopkins Press

T Machover Hyp erinstruments A comp osers approach

M Clynes Microstructural musical com

to the evolution of intelligentmusical instruments Cy

p osers pulses are liked most bythebestmusicians Cog

berArts pp Jan

nitionvol pp

S Mann See the world through myeyesawearable

X Yang Visual balance The tightrop e of computer gen

wireless camera

erated layout Masters thesis MIT Cambridge MA

httpwwwwhitemediamitedusteve netcamhtml

June

J E Cahn The generation of aect in synthesized

M Clynes and N S Kline Cyb orgs and space Astro

sp eech Journal of the American Voice IO Societyvol

nauticsvol pp and Sept

July

T E Starner Wearable computing Perceptual Com

I A Essa Analysis Interpretation and Synthesis of Facial

puting Group Media Lab MIT Cambridge MA

Expressions PhD thesis MIT Media Lab Cambridge

MA Feb

N Gershenfeld and M Hawley Personal Communi

G W Cottrell and J Metcalfe EMPATH Face emotion

cation

and gender recognition using holons in Neural Informa

O S Card Speaker for the DeadNewYork NY Tom

tion Processing Systems R P Lippmann J E Mo o dy

Doherty Asso ciates Inc

D S Touretzky and B M Spatz eds vol San Ma

J J Wurtman Managing your mind and moodthrough

teo CA pp Morgan Kaufmann Publishers Inc

foodNewYork Rawson Asso ciates

G Mandler Mind and Body Psychology of Emotion and

Y Yaco ob and L Davis Computing spatiotemp oral rep

StressNewYork NY W W Norton Company

resentations of human faces in Computer Vision and Pat

tern Recognition Conference pp IEEE Computer

D A Norman Twelve issues for cognitive science in

So ciety

Perspectives on Cognitive Science D A Norman ed

Hillsdal e NJ pp Erlbaum

S Morishima Emotion mo del A criterion for recogni

tion synthesis and compression of face and emotion

Int Workshop on Face and to Ap

p ear

R W Picard Content access for imagevideo co ding

The Fourth Criterion Tech Rep MIT Media Lab

Perceptual Computing Cambridge MA MPEG

Do c Lausanne

B Reeves Personal communication MIT Media Lab Col

lo quium

D Romer The Ko dak picture exchange April

seminar at MIT Media Lab

R W Picard and T P Minka Vision texture for anno

tation Journal of Multimedia Systemsvol pp

T P Minka and R W Picard Interactive learning us

ing a so ciety of mo dels Submitted for Publication

Also app ears as MIT Media Lab Perceptual Computing

TR

B W Kort Personal Communication

D O Hebb ATextbook of Psychology Philadelphi a W

B Saunders Co

K Ho op er Perceptual asp ects of architecture in Hand

book of Perceptual E C Carterette

and M PFriedman eds vol X New York NY Aca

demic Press