Intersubjectivity, Empathy and Nonverbal Interaction Nicola Jane Plant [email protected] School of Electronic Engineering and Computer Science Queen Mary University of London Submitted in fulfilment of the requirements of the Degree of Doctor of Philosophy May 2017 Declaration I, Nicola Jane Plant, confirm that the research included within this thesis is my own work or that where it has been carried out in collaboration with, or supported by others, that this is duly acknowledged below and my contribution indicated. Previously published material is also acknowledged below. I attest that I have exercised reasonable care to ensure that the work is original, and does not to the best of my knowledge break any UK law, infringe any third party's copyright or other Intellectual Property Right, or contain any confidential material. I accept that the College has the right to use plagiarism detection software to check the electronic version of the thesis. I confirm that this thesis has not been previously submitted for the award of a degree by this or any other university. The copyright of this thesis rests with the author and no quotation from it or information derived from it may be published without the prior written consent of the author. Signature: Date: 6th May 2017 1 Abstract Empathy is thought to involve cognitive processes that depend on the simulation of another's experiences. Embodiment has a key role for empathy as vehicle for recreating the experience of another. This thesis explores the validity of this claim by investigating what people do when communicating about their experiences. In particular, what is the contribution of our embodied resources such as gestures, postures and expressions to empathy and intersubjectivity? These questions are explored against two corpora of dyadic interactions. One features conversa- tions of people describing recalled embodied experiences to each other, such as painful or pleas- ant bodily experiences like a headache or laughing. The other features a series of interactions designed to emulate informal conversations. The analysis uses hand coded gestures, feedback and clarification questions, body movement data and a new approach to quantifying posture congruence. The analysis shows the embodied responses observed within these interactions are intentionally placed and formulated to facilitate the incremental process of a conversation as a joint activity. This is inconsistent with accounts that propose there is an automatic and non-conscious propensity for people to mimic each other in social interactions. Quantitative analysis show that patterns of gesture type and use, feedback form and posture differ systematically between interlocutors. Additionally, results show that resources provided by embodiment are allocated strategically. Nonverbal contributions increase in frequency and adjust their form responding to problems in conversation such as during clarification questions and repair. Detailed qualitative analysis shows the instances that appear to display mimicry within the interaction function rather as embodied adaptations or paraphrases. In their con- trast with the original contribution they demonstrate a specific understanding of the type of experience being conveyed. This work shows that embodiment is an important resource for in- tersubjectivity and embodied communication is specifically constructed to aid the collaborative, sequential and intersubjective progression of dialogue. Acknowledgements This work is supported by the Media and Arts Technology programme, EPSRC Doctoral Train- ing Centre EP/G03723X/1 Firstly, I would like to thank my supervisor Professor Patrick Healey not only for his guidance and intuition, but also for being extraordinary patient and never telling me off for being slow, giving support that I would get there in the end. I would next like to thanks my internal com- mittee, firstly my secondary supervisor Dr Martin Welton, who gave me resources that inspired and expanded my vision early on. Secondly, Dr Graham White who was always gently encour- aging and placed everything together for me while I had my head in a cloud of phenomenology. I would like to thank the thinking writing team at QMUL for organising the most pleasant and productive writing retreats and bootcamps that helped me get back on track when I'd lost confidence when writing up. I would also like to extend this to the rest of the academic staff in the Cognitive Science Research Group, for taking the time to suggest methods or ideas about my research as well always doing interesting research. I would like to thank Richard Kelly, the programme organiser at the time, for making things so easy and fun for me throughout the pro- cess. I would like to give a huge thanks to the rest of the Media and Arts Technology Pioneers, the later cohorts and PhD students of the Cognitive Science Research Group for always being around to hang out, inter-annotate, proof read, have a laugh and share things with. Lastly, I turn to non-academic support. My friends, for always being excited for me and putting up with me going AWOL for two and a half years. My family, always being so proud that I would finally make it, their kindness, generosity and positivity over the whole process. 1 Contents 1 Introduction 10 1.1 Embodiment and the Other . 10 1.2 Intersubjective experiences . 11 1.3 Empathy . 11 1.4 Empathy between moving bodies . 12 1.5 Mimicry and intersubjective understanding . 13 1.6 Empathy in actions . 14 1.7 Exploiting our embodied resources . 15 1.8 Document structure . 15 I Literature review 17 2 Concepts of intersubjectivity 18 2.1 Cognitive theories of mind . 18 2.1.1 Theory theory . 18 2.1.2 Simulation theory . 19 2.2 Phenomenological accounts of intersubjectivity . 20 2.2.1 Husserl's notion of intersubjectivity . 20 2.2.2 Expressive unity and intercorporeality . 22 2.3 Concepts of empathy and emotional contagion . 24 2.3.1 Emotional contagion: Embodying expressions . 25 2.3.2 Empathy: Two-step models . 29 2.4 Schutz and practical intersubjectivity . 30 2.5 Summary . 32 2.6 Conclusion . 35 3 Empirical studies on intersubjectivity 36 3.1 The development of intersubjectivity in infant-carer nonverbal interactions . 37 3.1.1 Evidence of innate behavioural mimicry . 37 3.1.2 Discussion of the evidence behind infant-carer behavioural mimicry . 40 2 3.1.3 Summary . 47 3.2 Empirical studies on intersubjectivity in adult interactions . 49 3.2.1 Behavioural mimicry . 49 3.2.2 The perception-behaviour link . 49 3.2.3 Mimesis and cognitive neuroscience . 52 3.2.3.1 The mirror neuron system . 52 3.2.3.2 Associative learning and mirror responses . 56 3.2.3.3 Empathy, embodied simulation and the mirror system . 59 3.2.3.4 Empathy for pain and neural correlates . 61 3.2.4 The mimicry prosocial effect . 62 3.2.4.1 Facilitators/Inhibitors . 65 3.2.5 Mimicry distinctions . 67 3.2.6 Posture . 68 3.2.6.1 Posture as indicative of social cues and affective states . 68 3.2.6.2 Posture congruence . 69 3.2.7 Gesture . 70 3.2.7.1 Gestural mimicry and entrainment . 72 3.2.7.2 Motor mimicry as a strategic communicative mechanism . 75 3.2.8 Feedback behaviours . 76 3.3 Conclusion . 81 II Methods 83 4 General methods: Analysing multimodal interactions 84 4.1 Aims of analysis . 84 4.2 Corpus 1: Conversations about bodily experience . 85 4.2.1 Pilot studies . 85 4.2.2 Participants: Corpus 1 . 86 4.2.3 Materials: Corpus 1 . 86 4.2.4 Procedure: Corpus 1 . 88 4.3 Corpus 2: Conversations about current affairs . 90 4.3.1 Participants: Corpus 2 . 90 4.3.2 Materials: Corpus 2 . 91 4.3.3 Procedure: Corpus 2 . 91 4.4 Motion capture as a tool for analysing embodied expressions . 92 4.4.0.1 Calibrating the system . 92 4.4.0.2 Marker placement and VSK files . 97 4.4.0.3 Post-processing: Reconstructing and labelling movement data . 97 4.4.0.4 Processing and cleaning the motion capture data . 97 3 4.4.0.5 Exporting 3-dimensional data . 98 4.5 Post-processing motion capture data . 99 4.5.1 Calculating a nodding index . 99 4.5.2 Calculating hand movements . 99 4.5.3 Providing a similarity measure for posture congruence . 100 4.5.3.1 Factors constraining posture . 100 4.5.3.2 Cross recurrence plots . 102 4.5.3.3 Measures of posture similarity . 104 4.6 Video-based analysis . 104 4.6.1 Aligning movement data to video . 105 4.6.2 Human coding and explorative observations . 105 4.6.3 Preparing video for analysis . 107 4.6.4 Transcription . ..