ENVIRONMENTAL PHYSICAL–VIRTUAL INTERACTION TO IMPROVE SOCIAL PRESENCE WITH A VIRTUAL HUMAN IN MIXED REALITY by KANGSOO KIM B.S. Hanyang University, 2009 M.S. Hanyang University, 2011 M.S. University of Central Florida, 2015 A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the College of Engineering and Computer Science at the University of Central Florida Orlando, Florida Fall Term 2018 Major Professor: Gregory F. Welch c 2018 Kangsoo Kim ii ABSTRACT Interactive Virtual Humans (VHs) are increasingly used to replace or assist real humans in various applications, e.g., military and medical training, education, or entertainment. In most VH research, the perceived social presence with a VH, which denotes the user’s sense of being socially connected or co-located with the VH, is the decisive factor in evaluating the social influence of the VH—a phenomenon where human users’ emotions, opinions, or behaviors are affected by the VH. The purpose of this dissertation is to develop new knowledge about how characteristics and behaviors of a VH in a Mixed Reality (MR) environment can affect the perception of and resulting behavior with the VH, and to find effective and efficient ways to improve the quality and performance of social interactions with VHs. Important issues and challenges in real–virtual human interactions in MR, e.g., lack of physical–virtual interaction, are identified and discussed through several user studies incorporating interactions with VH systems. In the studies, different features of VHs are prototyped and evaluated, such as a VH’s ability to be aware of and influence the surrounding physical environment, while measuring objective behavioral data as well as collecting subjective responses from the participants. The results from the studies support the idea that the VH’s aware- ness and influence of the physical environment can improve not only the perceived social presence with the VH, but also the trustworthiness of the VH within a social context. The findings will contribute towards designing more influential VHs that can benefit a wide range of simulation and training applications for which a high level of social realism is important, and that can be more easily incorporated into our daily lives as social companions, providing reliable relationships and convenience in assisting with daily tasks. iii To my parents, family, and friends. iv ACKNOWLEDGMENTS First, I would like to thank my wonderful advisor, Prof. Greg Welch, for guiding and supporting me with full of inspiration, ideas, and patience on the journey of this dissertation research. Greg was not just my academic advisor, but a great mentor who opened my eyes to various aspects of research and cared for me and my family’s well-being. Without his thoughtfulness, compassion, and respect, it would have been difficult for me to complete this work. It was a great pleasure being and working with him as one of his PhD students in an important part of my life. I would also like to acknowledge all other committee members: Prof. Jeremy Bailenson, Prof. Gerd Bruder, Prof. Stephen Fiore, Prof. Avelino Gonzalez, and Prof. Gita Sukthankar. Of course all the inputs from the committees are invaluable, but especially I want to thank Jeremy and Gerd for sharing their insights and thoughts, and actually spending a lot of time together to discuss to make things right for my research. Special thanks to all the current and former members of the SREAL lab: Prof. Charles Hughes, Dr. Leila Barmaki, Dr. Kate Ingraham, Dr. Sungchul Jung, Dr. Arjun Nagendran, Dr. Andrew Raij, Luke Bölling, Salam Daher, Manuel Gonzalez, Steffen Haesler, Jason Hochreiter, Eric Imperiale, Brian Kelly, Barbara Lee, Myungho Lee, Divine Maloney, Nahal Norouzi, Ryan Patrick, Remo Pillat, Ryan Schubert, and Greg Wiatroski. I acknowledge Prof. Eleazar Vasquez and Prof. Xin Yan for sharing their expertise in study design and statistical analysis with me. I also want to thank Dr. Peter Squire (Office of Naval Research) for the continuous funds for our lab and my work during the entire period of my PhD program. I would like to thank Prof. Jong-Il Park (my BS and MS advisor at Hanyang University) and Prof. Woontack Woo (my undergraduate intern advisor at GIST) for their introduction and guidance to the research field of AR/MR. In addition, I want to thank all the research advisors, colleagues, and v friends in Korea including Prof. Sang-Hwa Lee, Dr. Byung-Kuk Seo, Dr. Junyeong Choi, Jungsik Park, Jae-Hyek Han, Youngju son, Prof. Ahyoung Choi, Prof. Changgu Kang, Dr. Youngkyoon Jang, Dr. Sejin Oh, Dr. Mingyu Park, and Dr. Choonsung Shin. From my college years to present, interacting with them always encouraged me to move forward. Special thanks to Byung-Kuk, Youngkyoon, and Choonsung for sharing their thoughts and discussing my research together. I had three wonderful internship opportunities at Universal Creative, Facebook, and US Army Research Lab-West in the middle of my PhD program. Here I would like to thank all the people who helped and worked with me including my intern managers: Justin Schwartz, Dr. Sophie Kim, and Dr. Celso de Melo. During the journey of my PhD, I met a lot of researchers and collaborators from different fields who have influenced me and my research directly or indirectly. I would thank all those people including Prof. Mark Billinghurst, Prof. Jonathan Gratch, Prof. Dongsik Jo, Prof. Sin-Hwa Kang, Prof. Si- Jung Kim, Prof. Gale Lucas, and Dr. Jongwook Kim. I learned a lot from them while reading about their research, discussing research ideas and working together with them. Particularly I want to thank Sangwoo Cho for sharing all the happiness and difficulty together as PhD students. Finally, I would like to thank my family, the most important people in my life. My lovely wife and little kids, Jihyun Kim, Juli Kim, and Daniel Kim. Their endless support and cheer for me made this work possible. Thanks to my sister, Seoyoun Kim, who has always walked one step ahead and guided me since our childhood. Also, I thank my parents-in-law, Jinbae Kim and Kwangock Seo, for their considerate and warm heart supporting me and my family. The most special and sincere thanks go to my beloved parents, Jongo Kim and Jongsook Lee, who are the origin and the role model of my life. They have always shown faith in me and given me liberty to choose what I desired. I could have the strength and patience to work through all these years for their selfless dedication. Thank you so much from the bottom of my heart. vi TABLE OF CONTENTS LIST OF FIGURES . xv LIST OF TABLES . .xviii CHAPTER 1: INTRODUCTION . 1 1.1 Research Motivation and Objectives . 3 1.2 Exploration Areas and Scope of the Dissertation Research . 5 1.3 Thesis Statements and Contributions . 7 1.4 Outline of the Dissertation . 9 CHAPTER 2: BACKGROUND . 12 2.1 Pervasive Mixed Reality . 12 2.1.1 Mixed Reality in Reality–Virtuality Continuum . 13 2.1.2 Ubiquitous Computing and Artificial Intelligence . 17 2.2 Presence and Social Influence . 19 2.3 Virtual Humans in Mixed Reality . 22 2.3.1 Robotic Humans . 23 vii 2.3.2 Virtual Humans . 24 2.3.2.1 Virtual Humans in Monitors and Immersive Projection . 25 2.3.2.2 Virtual Humans in Immersive VR HMD . 26 2.3.2.3 Virtual Humans in AR HMD . 27 2.3.2.4 Virtual Humans without Visual Embodiment . 29 2.4 User Studies with Virtual Humans . 30 2.4.1 Social Presence and Co-Presence with Virtual Humans . 30 2.4.2 Behavioral Influences of Virtual Humans . 32 2.4.3 Trust and Confidence in Virtual Humans . 34 CHAPTER 3: PROPOSED ENVIRONMENTAL PHYSICAL–VIRTUAL INTERACTION INMR..................................... 36 3.1 Visual Embodiment of Virtual Humans . 36 3.2 Interaction: Awareness and Influence . 37 3.3 Plausibility and Coherence . 40 CHAPTER 4: INFLUENCE OF PHYSICAL BODY IN HUMAN BEHAVIOR . 43 4.1 Material . 44 4.1.1 Technological Human Surrogate . 44 viii 4.1.1.1 RoboThespian (Robotic Human Surrogate) . 44 4.1.1.2 Human in the Loop . 45 4.1.2 Experimental Setup . 47 4.1.2.1 Surrogate and Environment Settings . 47 4.1.2.2 Surrogate Control and Communication Connections . 48 4.2 Methods . 51 4.3 Participants and Procedure . 53 4.4 Measures and Hypotheses . 55 4.4.1 Data Collection and Human Observer . 55 4.4.2 Dependent Variables . 57 4.5 Results . 58 4.5.1 Proactivity and Reactivity . 64 4.5.2 Commitment . 68 4.5.3 Compliance . 69 4.5.4 Photo Proximity . 71 4.6 Discussion . 73 4.7 Summary . 76 ix CHAPTER 5: SPATIAL AND BEHAVIORAL COHERENCE IN MR . 78 5.1 Virtual Human’s Physical–Virtual Conflict . 78 5.1.1 Material . 79 5.1.1.1 Physical Environment and Recordings . 79 5.1.1.2 Virtual Humans and Human Controller . 79 5.1.2 Methods . 81 5.1.3 Participants and Procedure . 82 5.1.4 Measures and Hypotheses . 84 5.1.4.1 Physicality and Interaction in Physical Space . 84 5.1.4.2 Copresence . 85 5.1.4.3 Sense . 85 5.1.4.4 Realism . 85 5.1.4.5 Godspeed Questionnaire . 85 5.1.5 Results . 86 5.1.6 Subjective Responses . 86 5.1.6.1 Physicality and Interaction in Physical Space . 86 5.1.6.2 Copresence . 88 x 5.1.6.3 Sense & Realism . 88 5.1.6.4 Godspeed Questionnaire . 88 5.1.7 Avoidance Behavior . 90 5.1.8 Discussion . 91 5.1.9 Summary .