UNIVERSITY OF CALIFORNIA Los Angeles Causal Action: A Framework to Connect Action Perception and Understanding A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Psychology by Yujia Peng 2019 ©Copyright by Yujia Peng 2019 ABSTRACT OF THE DISSERTATION Causal Action: A Framework to Connect Action Perception and Understanding by Yujia Peng Doctor of Philosophy in Psychology University of California, Los Angeles, 2019 Professor Hongjing Lu, Chair Human actions are more than mere body movements. In contrast to dynamic events involving inanimate objects, human actions have a special status in that they control interactions with the world and afford privileged access to the experience of agency and to control interactions with the world. Several causal constraints on human actions support the generation and the understanding of actions. For example, human actions inherently involve a causal structure: limb movements generally cause changes in body position along a path through the environment to achieve intentional goals. However, it remains unclear how the system that supports action perception communicates with high-level reasoning system to recognize actions, and more importantly, to achieve a deeper understanding of observed actions. My dissertation aims to ii determine whether causality imposes critical motion constraints on action perception and understanding, and how causal relations involved in actions impact behavioral judgments. The project also investigates the developmental trajectory and neural substrate of action processing, and whether a feedforward deep learning model is able to learn causal relations solely from visual observations of human actions. Through behavioral experiments, an infant eye movement study, a neural study using magnetoencephalography, and model simulations, my dissertation yields a number of insights. 1) Humans implicitly and automatically rely on causal expectations to explain motion information when perceiving body movements and meaningful social interactions; 2) Sensitivity to causal constraints on actions develops early in infants even before 18 months of age, and shows a clear relation with the development of gross motor functions. 3) Congruency to causal relations involved in human actions can be decoded from neural MEG signals, with the processing of causal actions eliciting a distributed neural network. The brain network involves temporal, parietal, and frontal regions that are important locus for spatial relation reasoning, decision making, and intention understanding. 4) Recent stimulus-driven deep learning neural networks are unable to learn the causal relations involved in actions, failing to create high-level representations for causal actions. Overall, my dissertation reveals the importance of causality in bridging action perception and understanding, highlights the key missing computational components in deep learning models for complex visual stimuli such as human actions, and provides a potential framework to connect seeing and thinking. iii The dissertation of Yujia Peng is approved. Chris Hak Wan Lau Scott Pratt Johnson Yingnian Wu Hongjing Lu, Committee Chair University of California, Los Angeles 2019 iv Dedication This dissertation is dedicated to my inspiring parents, for their endless love, support, and wisdom; My wonderful husband, Jiming, for his love and knowledge that encourage me to pursue my dreams. v TABLE OF CONTENTS ACKNOWLEDGEMENTS .......................................................................................................... xii VITA ............................................................................................................................................ xiv CHAPTER 1: Introduction ............................................................................................................. 1 1.1 The Phenomenal Causality in Moving Objects ................................................................ 5 1.2 The developmental trajectory of causal perception .......................................................... 8 1.3 Neural Bases for Perceptual Causality and Action Recognition .................................... 11 1.4 Computational Models and Deep Learning Algorithms of Action Recognition ........... 13 CHAPTER 2: Causal Action: A Fundamental Constraint on Perception and Inference about Body Movements .......................................................................................................................... 17 2.1 Abstract .......................................................................................................................... 17 2.2 Introduction .................................................................................................................... 17 2.3 Experiment 1 .................................................................................................................. 20 2.3.1 Method .................................................................................................................... 20 2.3.2 Results .................................................................................................................... 24 2.4 Experiment 2a ................................................................................................................ 26 2.4.1 Method .................................................................................................................... 27 2.4.2 Results ..................................................................................................................... 29 2.5 Experiment 2b ................................................................................................................ 32 2.5.1 Method .................................................................................................................... 33 2.5.2 Results ..................................................................................................................... 33 2.6 An Ideal Observer Model Based on Visual Statistics .................................................... 37 2.7 General Discussion ......................................................................................................... 39 CHAPTER 3: Perception of Continuous Movements from Causal Actions ................................ 43 3.1 Abstract .......................................................................................................................... 43 3.2 Introduction .................................................................................................................... 44 3.2 Experiment 1 .................................................................................................................. 48 3.2.1 Methods................................................................................................................... 48 3.2.2 Results ..................................................................................................................... 53 3.3 Experiment 2 .................................................................................................................. 55 vi 3.3.1 Methods................................................................................................................... 56 3.3.2 Results ..................................................................................................................... 57 3.4 Experiment 3 .................................................................................................................. 58 3.4.1 Methods................................................................................................................... 59 3.4.2 Results ..................................................................................................................... 60 3.5 Experiment 4 .................................................................................................................. 62 3.5.1 Methods................................................................................................................... 62 3.5.2 Results ..................................................................................................................... 63 3.6 General Discussion ......................................................................................................... 65 CHAPTER 4: A motion consistency constraint on infant perception of body movements.......... 69 4.1 Abstract .......................................................................................................................... 69 4.2 Introduction .................................................................................................................... 69 4.3 Method ........................................................................................................................... 72 4.3 Results ............................................................................................................................ 75 4.4 Discussion ...................................................................................................................... 81 CHAPTER 5: Revealing the Neural Mechanism of Causal Action Perception Using MEG ....... 85 5.1 Abstract .......................................................................................................................... 85 5.2 Introduction .................................................................................................................... 86 5.2 Methods .........................................................................................................................