Exploratory Tool for Autism Spectrum Conditions
Total Page:16
File Type:pdf, Size:1020Kb
MIT | Media Lab | Affective Computing Exploratory Tool for Autism Spectrum Conditions Rana el Kaliouby Alea Teeters Rosalind Picard http://www.media.mit.edu/affect BSN 2006 Workshop Autism Spectrum Conditions autism mind-reading machines demo challenges Center for Disease Control and Prevention (2005) – 1 child in 166 has ASC {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing New Initiative: Autism Wearables autism mind-reading machines demo challenges Repetitive, obsessive behavior Related work • Monitoring • Assessment • Natural environment Communication Social interaction {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Examples of Autism Wearables autism mind-reading machines demo challenges Automated capture to support therapists Recognition of stimming behavior during intervention sessions (e.g. flapping, rocking) (Digital Pen, Voice Input and Video) Bluetooth accelerator and HMMs Kientz, Broing, Abowd, Hayes Westeyn, Vadas, Bian, Starner, (Ubicomp 2005) and Abowd (ISWC 2005) {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Autism Spectrum Conditions autism mind-reading machines demo challenges Repetitive, obsessive behavior Communication Social interaction Our research • Intervention • Assistive • Natural environment {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Mind-Read > Act > Persuade autism mind-reading machines demo challenges hmm … Roz looks busy. Its probably not a good time to bring this up Analysis of nonverbal cues Inference and reasoning Modify one’s actions about mental states Persuade others {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Real time Mental State Inference autism mind-reading machines demo challenges El Kaliouby and Robinson (2005) Facial feature Head & facial Head & facial Mental state extraction action unit display inference recognition recognition Head pose estimation Feature point hmm … Let tracking* me think about this * Nevenvision face-tracker {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Assertive Committed Affective-Cognitive MentalAgreeing StatesPersuaded Sure autism mind-reading machines demo challenges Baron-Cohen et al. AUTISM RESEARCH CENTRE, Assertive CAMBRIDGE Committed Agreeing Persuaded Sure Absorbed Concentrating Concentrating Complex Vigilant Mental Disapproving States Disagreeing Discouraging (subset) Disinclined Asking Curious Interested Impressed Interested Brooding Choosing Thinking Thinking Thoughtful Baffled Confused Unsure Undecided {kaliouby, teeters, picard}@media.mit.edu Unsure MIT | Media Lab | Affective Computing Accuracy > Posed > Actors autism mind-reading machines demo challenges Accuracy of system when trained and tested with posed actrors {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Posed > Non-actors autism mind-reading machines demo challenges Agreeing Disagreeing Confused Concentrating Thinking Interested IEEE CVPR Conference, 2004 {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Posed > Non-actors autism mind-reading machines demo challenges {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Accuracy > Posed > Non-actors autism mind-reading machines demo challenges Accuracy of panel of 18 people Accuracy of system classifying the videos (as good as the top 6%) {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Real-time Performance autism mind-reading machines demo challenges Level Time on Frequency Load 3.4GHz p4 FaceTracker 3.00 ms 30 Hz 9.0% Action Units 0.09 ms 30 Hz 0.3% 9 Display HMMs 0.14 ms 6 Hz 0.1% 6 Mental State DBNs 41.10 ms 6 Hz 24.7% Total 34.1% {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Sense > Explore > Assist autism mind-reading machines demo challenges Comfortable sensing in Explore socio-emotional Assist in communication natural Environment cues in self and others (how to respond to disinterest) Partner with behavioral programs already in place (Groden Center) {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Self-Cam autism mind-reading machines demo challenges Videos recorded by myDejaView camera Monitoring self-expressions (along with other body sensors) Opportunity to learn about emotion expression in self Networked > exchange of social-emotional cues {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing 2-person interaction > Monologue autism mind-reading machines demo challenges {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Challanges autism mind-reading machines demo challenge to BSN Infrastructure for exchanging this information Novel apps On-body processing Form-factor Data sensor fusion Analysis Wearable Inference High-res Prediction Wide-angle Privacy High frame rate Power consumption {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Social Sensor Networks autism mind-reading machines demo challenge to BSN Peer-to-Peer PANs PAN > Roz PAN > Seth Sensor sampling Sensor data analysis Mental state inference Share state PAN > Alea {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing Social Sensor Networks autism mind-reading machines demo challenge to BSN • Networks to exchange social-emotional cues – Real-time – Not limited to facial expressions • Examples of cues: – Facial expressions – Affect in speech – Physiology – Affective-Cognitive States – Activity – E.g.: is it a good time to interrupt • On-Body Processing – Alleviates privacy concerns – You choose what and who to share your affective-cognitive states with {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing autism and mind-reading sense > explore > assist {kaliouby, alea, picard}@media.mit.edu wearable social sensor networks Acknowledgements: www.myDejaview.com for after-the-fact cameras www.nevenvision.com for face tracking technology Matthew Goodwin, Groden Center NSF and TTT consortium for funding this research Expression Capture autism mind-reading machines demo challenges Camera by myDejaView Opportunity to replay/reflect on expressions of people you interact with Fun using a camera Improved ability to look at, recognize, and respond to expressions {kaliouby, teeters, picard}@media.mit.edu MIT | Media Lab | Affective Computing .