Representing People in Virtual Environments Will Steptoe 17th November 2009 INTRODUCTION What’s in this lecture? First Hour Overview and Applications - State-of-Art, Social Agency, Human Behaviour, Realism, Applications, Agency (Agents and Avatars), CVEs, Avatar Control. Second Hour Technical Aspects and Demonstration - Graphics, Animation, Behaviour - Application in 3DSMax INTRODUCTION State-of-Art Real-Time Pre-Rendered Heavy Rain The Curious Case of Quantic Dream, 2009 Benjamin Button David Fincher, 2008 INTRODUCTION Virtual Humans • Complex problem of technical and human factors. • Generating subtleties of human behaviour is a problem beyond raw computing power. The more real they look, the more real we expect them to behave. • To generate completely realistic characters we have to completely understand human perception in reality! • ... but why are we so sensitive to minor defects in virtual humans? SOCIAL AGENCY Social Agency and the ELIZA effect • People generally require minimal encouragement to view computer systems and applications as social agents, reading far more understanding than is warranted from symbols and graphical displays. “Individuals mindlessly apply social rules and expectations to computers” – Nass and Moon, 2000. • This was unexpectedly observed, and first documented, by Weizenbaum (1966) when performing user studies with ELIZA - a computer program for the study of natural language communication between man and machine. SOCIAL AGENCY Social Agency and the ELIZA effect • During the purely text-based interactions between participants and the system, ELIZA simulated a Rogerian psychotherapist by rephrasing input statements from the user, and returning them as questions. (i.e. “I’m feeling depressed” -> Why do you think you are feeling depressed?) • Weizenbaum observed many examples of people becoming emotionally engaged when ‘communicating’ with ELIZA, and some even asked to be left alone with the system. SOCIAL AGENCY Social Agency and the ELIZA effect • This phenomenon has become known as the ‘ELIZA effect’, and may be considered a precursor to many observations found in the VE literature concerning presence (place illusion) and copresence. • People are particularly responsive to depictions of humans. SOCIAL AGENCY The Fear of public speaking • David • Not very comfortable with public speaking • Asked to speak about his favourite subject: cables • Behaviours triggered at appropriate intervals Pertaub, D.-P., Slater, M., and Barker, C. (2002). An experiment on public speaking anxiety in response to three different types of virtual audience. Presence: Teleoperators and Virtual Environments, 11(1): 68-78 SOCIAL AGENCY The Fear of public speaking • The user was asked to give a presentation three times – Positive, Negative and Mixed • Positive - agents smiled, leaned forward, faced the user, maintained gaze, clapped hands, etc. • Negative - agents yawned, slumped forward, put feet on the table, avoided eye contact, and finally walked out • Mixed - agents started off with largely negative responses and gradually turned positive SOCIAL AGENCY Realistic responses in VE ? • Individuals' self-rated performance was positively correlated with the perceived good mood of the agents • Evidence of a negative response especially strong with the negatively inclined audience – Sweating and stammering – Vocal protests at the agent behaviours • Virtual humans with minimal behavioural-visual fidelity can elicit significant user responses • End Goal: Virtual humans with high visual fidelity that mimic real-life context-appropriate behaviours HUMAN BEHAVIOUR Categories of behavioural cues Argyle, M. (1998). Bodily Communication. Methuen & Co Ltd, second edition. • Vocal properties – Tone, Pitch, Loudness… • Facial expressions – The most studied behavioural cue due to it’s role in communication • Gaze behaviour – Probably the most intense social signallers • Kinesics: Posture and Motion – Numerous gestures depending on culture for instance • Proxemics – Culture and gender dependent HUMAN BEHAVIOUR Facial expression • In reality, 20000 facial expressions exist • Normally animated by blending “Morph Targets” • Different granularities of facial expression – Facial action parameters (most basic units) • Basic emotions – Phonemes (mouth shapes for lip-sync) – Principal component analysis HUMAN BEHAVIOUR Gesture • Normally animated by choosing from a library of gestures • Very closely associated with speech – Also back channel gestures by listeners (e.g. head nod) • Different types of gesture – E.g. beat, iconic • Again see Cassell’s work referenced earlier HUMAN BEHAVIOUR Posture Coulson, M. (2004). Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28(2):117–139. • Over 1000 stable postures have been observed • Normally animated by choosing from (or blending between) a library of gestures • Associated with attitude and emotion • Associated also with interpersonal attitude HUMAN BEHAVIOUR Measuring Success • So the careful design of behaviour is important but there are caveats • Success of a VE is measured in terms of the extent to which sensory data projected within a virtual environment replaces the sensory data from the physical world – quantified by rating the individuals’ sense of presence during the experience • For Virtual Humans: Success is taken as the extent to which participants act and respond to the agents as if they were real – Subjective: Questionnaires, Interviews – Objective: Physiological, Behavioural HUMAN BEHAVIOUR Subjective means • Traditional methods: Questionnaires and interviews – Various questionnaires exist – http://www.presence-research.org • Criticised due to its various dependencies – the individual’s accurate post-hoc recall, – processing and rationalisations of their experience in the VE and – Varying interpretations of the word ‘presence’ HUMAN BEHAVIOUR Objective: Responses to stimuli • Numerous possible objective measures – Subconscious responses • Threat-related facial cues provokes individuals to use different viewing strategies – Neural responses • Different areas of the brain are activated during +ve, -ve and neutral situations – Psychological responses • Stress and Anxiety in response to threat – Physiological responses • Galvanic Skin Responses, Heart Rate Variability, electrocardiograms, electromyography, Respiratory activity – Behavioural responses • Flight or Fight (based on cognitive appraisal) • Vary based on cognitive factors, personality, emotional state, gender etc. – How do we interpret the data and results? REALISM Uncanny Valley • As the behaviour and representation of robots (and other facsimiles) of humans approaches that of actual humans, it causes a response of revulsion among human observers. • Theory from 70s by roboticist Masahiro Mori – Controversial, its not very rigorous or scientific, many people don’t believe it – There are problems but maybe it captures something REALISM The Uncanny Valley REALISM The Uncanny Valley Dreamworks reduced realism of Princes Fiona (Shrek): “…she was beginning to look too real, and the effect was getting distinctly unpleasant.” Final Fantasy movie: “…it begins to get grotesque. You start to feel like you're puppeteering a corpse” REALISM REALISM Uncanny Valley • At low levels of realism, the more realistic a character the more people like it. • But when you get almost real then characters start to get disturbing - corpses are used a lot as metaphors • Interestingly, there are two graphs: movement and appearance, movement is more important. REALISM Different Types of Realism • Visual Realism – What it looks like (pictures, film, games, VE) • Animation Realism – How it moves, animation (film, games, VE) • Behavioural Realism – How it responds and interacts (games, VE) REALISM Mismatch in Realism • Maybe the problem is that levels of movement and behavioural realism do not match graphical realism. • This mismatch disturbs us, something that looks human but does not act like a human. • Consistency is important. REALISM Appearance vs. Behaviour Vinayagamoorthy, V., Garau, M., Steed, A., and Slater, M. (2004b). An eye gaze model for dyadic interaction in an immersive virtual environment: Practice and experience. Computer Graphics Forum, 23(1):1–11. REALISM Appearance vs. Behaviour App. Cartoon Higher – App. Cartoon Higher – – Form Fidelity – Form Fidelity Beh. Beh. Random 3 ♂ pairs 3 ♂ pairs Random High Low gaze 3 ♀ pairs 3 ♀ pairs gaze Inferred* 3 ♂ pairs 3 ♂ pairs Inferred* Low High gaze 3 ♀ pairs 3 ♀ pairs gaze Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., and Sasse, A. M. (2003). The impact of avatar realism and eye gaze control on the perceived quality of communication in a shared immersive virtual environment. In Proceedings of SIGCHI, pages 529–536. REALISM Appearance vs. Behaviour • Realistic gaze behaviour had a positive impact on the perception of more visually-realistic avatars. • In the case of a lower visually realistic avatar, the more complex gaze model had a negative effect on participant response. • Important to note that the differences between both the gaze models were very subtle – saccadic velocity and fixation durations. • Analysis demonstrated a very strong interaction effect between the type of avatar and the fidelity of the gaze model. REALISM Realism vs Believability • The lesson is that we need to be careful with realism for virtual humans • Often we prefer to use the term “believability” – Not how much a