Experience Engineering & Conscious AI:

Olaf Blanke of . Consciousess model is based on multisensory and sensorimotor brain signals and also includes the subject of conscious experience.

Experience Engineering is a laboratory-based research and engineering field that applies engineering principles to consciousness.

I will argue that AI should fully embrace - next to the many cognitive aspects of the mind - the topic of consciousness. The Mind in and AI

- Reasoning & problem solving - Knowledge representation - Planning - Learning and Memory - Language - Perception - Sensorimotor processing - Autonomous cars/drones

giphy.com; rcs.el.tum.de Consciousness ? Consciousness Consciousness: from a problem in philosophy to an empirical scientific challenge Perceptual Consciousness: The case of Blindsight Visual behavior and visual consciousness

(Weiskrantz et al., 1974; Heeger) Perceptual Consciousness: Visual Illusions

Ed Adelson Perceptual Consciousness

Visual consciousness

Auditory consciousness Consciousness is multisensory and is an Tactile consciousness integrated and unitary experience. Gustatory consciousness Motor consciousness Conscious states are mental states of … someone, they are experienced by a subject of conscious experience. Multisensory consciousness Consciousness is multisensory and is an integrated and unitary experience.

Conscious states are mental states of someone, they are experienced by a subject of conscious experience.

How can we study this entity? This « someone », this self, and « I » of conscious experience ? Multisensory own body illusions allow to study and manipulate self-consciousness

Rubber hand illusion

Subjects are exposed to prolonged multisensory (visuo-tactile- proprioceptive) stimulations, exploiting visual dominance over somatosensory cues

[see also video on youtube: « Virtual rubber hand illusion »; « Virtual Out-of-body experience »] Full-Body Illusion

Studying bodily self-consciousness using immersive and multisensory stimulation Subject with Head Mounted Display

Video camera with 3- D converter

Virtual body

2m

Actual body

[Lenggenhager et al., Video Ergo Sum, Science 2007] Full-Body Illusion

Studying bodily self-consciousness using immersive virtual reality and multisensory stimulation Inducing changes in bodily self-consciousness A virtual or filmed body feels like my body (self-identification) and “I” am localized at the virtual body’s position (self-location)

Self-identification Multisensory stimulation Self-location

Position 1 minute Questionnaire Recalibration

[Lenggenhager et al., Science 2007] [youtube: « Virtual out-of-body experience »] Altered states of self-consciousness are associated with changes in tactile perception, body temperature regulation, analgesia, and heartbeat control

Decrease in Cardiac stimuli Body-self- Analgesia body specificity temperature

[Hänsel et al., Eur J Pain, 2011; Aspell et al. Psychological Science [Palluel et al., J Neurophysiol 2011; [Salomon et al., Frontiers Romano et al., Beh Brain Res, 2013; Aspell et al., PlosOne 2009] Neuroscience 2013] 2014, J Pain, 2015] Park et al. J Neurosci 2016; Salomon et al. J Neurosci 2016 Experience Engineering Integrating virtual reality and robotics with brain imaging (fMRI)

Subject MRI-compatible … in the scanner Avatar position robot

Activation of temporo-parietal junction (SMG, AG, pSTG,) reflects robotically- induced changes in bodily self-consciousness (self-identification and self- location)

[Ionta et al., Neuron 2011] [Givraz et al., Neuroimage 2017] What happens if we disrupt the brain mechanisms of bodily self-consciousness not by VR and multisensory conflicts, but by brain damage ?  Out-of-body experiences

[Blanke et al., Nature 2002] [Blanke et al.,Brain 2004, Ionta et al., Neuron 2011] Out-of-body experiences are common (5%); they are not paranormal phenomena, but specific brain phenomena (comparable to phantom limbs): OBEs are phantom bodies

« I was asleep on a settee with my wife. I was aware at one stage that I was out of my own body poised six feet or so above and peering down at the two of us laying on the settee. » [Celia Green, Out-of-the-body experiences, 1968]

“One day, at age 17, I was walking alone at night during a snowstorm in a singularly quiet place. I noticed that the ground looked further away than usual, and then it seems that I was looking down from a height of perhaps 10 meters, watching myself crossing the field.” Marvin Minsky Neurophysiological basis of bodily self-consciousness: Trimodal neurons in primate posterior parietal cortex Arm-centered

Monkey Multisensory before after neurons (visual- tactile, proprioceptive) in posterior parietal Monkey and premotor cortex Conscious before after Enlarged AI Visual receptive fields visual receptive fields (peripersonal space) (peripersonal space)

Human

Trunk- centered before after [Iriki et al., 1996, 2001; [Serino et al., Sci Reports 2015; Graziano et al., Science 2000; Blanke Nature Rev Neurosci 2012; Maravita and Iriki, 2004] Blanke et al., Neuron 2015] Neuroscience of consciousness: Consciousess is based on specific multisensory and sensorimotor bodily brain signals and includes the subject of conscious experience.

Experience Engineering is a laboratory-based research and engineering field that applies engineering principles to consciousness.

AI should embrace - next to the many cognitive aspects of the mind - the topic of consciousness, including self- consciousness, because it is the most important and fascinating aspect of consciousness. Ghost in the machine ! Experience Engineering

Neuro-inspired embodiment and multisensory teleportation using Virtual and technology (Robotics/Haptics) Why are only certain parts of the brain associated with consciousness, but not others?

Are there other species that are conscious?

Are babies conscious?

Could certain systems behave like us and yet have no consciousness (i.e. zombie)?

Can machines be conscious?

What is the function of consciousness? Or is consciousness just an epiphenomenon ? Clinical applications

Consciousness deficits in patients (, Sleepwalking, Coma, Minimal conscious states)

Too much consciousness (in patients with schizophrenia, chronic pain, hallucinations in Parkinson’s disease)

Artificial limbs and artificial bodies Thank you !

Supported by: Swiss National Science Foundation Bertarelli Foundation European Science Foundation Spoelberch foundation IRP foundation

[email protected]

Laboratory of Cognitive Neuroscience Center for Neuroprosthetics

Swiss Federal Institute of Technology