MIT | Media Lab | Exploratory Tool for Autism Spectrum Conditions Alea Teeters Rosalind Picard http://www.media.mit.edu/affect BSN 2006 Workshop Autism Spectrum Conditions

autism mind-reading machines demo challenges

Center for Disease Control and Prevention (2005) – 1 child in 166 has ASC

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing New Initiative: Autism Wearables

autism mind-reading machines demo challenges

Repetitive, obsessive behavior Related work • Monitoring • Assessment • Natural environment

Communication Social interaction

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Examples of Autism Wearables

autism mind-reading machines demo challenges

Automated capture to support therapists Recognition of stimming behavior during intervention sessions (e.g. flapping, rocking) (Digital Pen, Voice Input and Video) Bluetooth accelerator and HMMs

Kientz, Broing, Abowd, Hayes Westeyn, Vadas, Bian, Starner, (Ubicomp 2005) and Abowd (ISWC 2005)

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Autism Spectrum Conditions

autism mind-reading machines demo challenges

Repetitive, obsessive behavior

Communication Social interaction

Our research • Intervention • Assistive • Natural environment

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Mind-Read > Act > Persuade

autism mind-reading machines demo challenges

hmm … Roz looks busy. Its probably not a good time to bring this up

Analysis of nonverbal cues Inference and reasoning Modify one’s actions about mental states Persuade others

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Real time Mental State Inference

autism mind-reading machines demo challenges

El Kaliouby and Robinson (2005)

Facial feature Head & facial Head & facial Mental state extraction action unit display inference recognition recognition

Head pose estimation

Feature point hmm … Let tracking* me think about this

* Nevenvision face-tracker {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Assertive Committed Affective-Cognitive MentalAgreeing StatesPersuaded Sure autism mind-reading machines demo challenges Baron-Cohen et al. AUTISM RESEARCH CENTRE, Assertive CAMBRIDGE Committed Agreeing Persuaded Sure Absorbed Concentrating Concentrating Complex Vigilant Mental Disapproving States Disagreeing Discouraging (subset) Disinclined Asking Curious Interested Impressed Interested Brooding Choosing Thinking Thinking Thoughtful Baffled Confused Unsure Undecided {kaliouby, teeters, picard}@media.mit.edu Unsure MIT | Media Lab | Affective Computing Accuracy > Posed > Actors

autism mind-reading machines demo challenges

Accuracy of system when trained and tested with posed actrors

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Posed > Non-actors

autism mind-reading machines demo challenges

Agreeing Disagreeing Confused Concentrating Thinking Interested

IEEE CVPR Conference, 2004 {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Posed > Non-actors

autism mind-reading machines demo challenges

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Accuracy > Posed > Non-actors

autism mind-reading machines demo challenges

Accuracy of panel of 18 people Accuracy of system classifying the videos (as good as the top 6%)

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Real-time Performance

autism mind-reading machines demo challenges

Level Time on Frequency Load 3.4GHz p4 FaceTracker 3.00 ms 30 Hz 9.0%

Action Units 0.09 ms 30 Hz 0.3%

9 Display HMMs 0.14 ms 6 Hz 0.1%

6 Mental State DBNs 41.10 ms 6 Hz 24.7%

Total 34.1%

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Sense > Explore > Assist

autism mind-reading machines demo challenges

Comfortable sensing in Explore socio-emotional Assist in communication natural Environment cues in self and others (how to respond to disinterest)

Partner with behavioral programs already in place (Groden Center)

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Self-Cam

autism mind-reading machines demo challenges

Videos recorded by myDejaView camera

Monitoring self-expressions (along with other body sensors) Opportunity to learn about expression in self Networked > exchange of social-emotional cues {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing 2-person interaction > Monologue

autism mind-reading machines demo challenges

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Challanges

autism mind-reading machines demo challenge to BSN

Infrastructure for exchanging this information

Novel apps

On-body processing Form-factor Data sensor fusion Analysis Wearable Inference High-res Prediction Wide-angle Privacy High frame rate Power consumption

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Social Sensor Networks

autism mind-reading machines demo challenge to BSN

Peer-to-Peer PANs

PAN > Roz PAN > Seth

Sensor sampling Sensor data analysis Mental state inference Share state

PAN > Alea {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Social Sensor Networks

autism mind-reading machines demo challenge to BSN

• Networks to exchange social-emotional cues – Real-time – Not limited to facial expressions

• Examples of cues: – Facial expressions – Affect in speech – Physiology – Affective-Cognitive States – Activity – E.g.: is it a good time to interrupt

• On-Body Processing – Alleviates privacy concerns – You choose what and who to share your affective-cognitive states with

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing autism and mind-reading sense > explore > assist

{kaliouby, alea, picard}@media.mit.edu wearable social sensor networks

Acknowledgements: www.myDejaview.com for after-the-fact cameras www.nevenvision.com for face tracking technology Matthew Goodwin, Groden Center NSF and TTT consortium for funding this research Expression Capture

autism mind-reading machines demo challenges

Camera by myDejaView

Opportunity to replay/reflect on expressions of people you interact with Fun using a camera Improved ability to look at, recognize, and respond to expressions

{kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing