Clemson University TigerPrints All Dissertations Dissertations 5-2017 A Voice and Pointing Gesture Interaction System for Supporting Human Spontaneous Decisions in Autonomous Cars Pablo Sauras-Perez Clemson University, [email protected] Follow this and additional works at: https://tigerprints.clemson.edu/all_dissertations Recommended Citation Sauras-Perez, Pablo, "A Voice and Pointing Gesture Interaction System for Supporting Human Spontaneous Decisions in Autonomous Cars" (2017). All Dissertations. 1943. https://tigerprints.clemson.edu/all_dissertations/1943 This Dissertation is brought to you for free and open access by the Dissertations at TigerPrints. It has been accepted for inclusion in All Dissertations by an authorized administrator of TigerPrints. For more information, please contact [email protected]. A VOICE AND POINTING GESTURE INTERACTION SYSTEM FOR SUPPORTING HUMAN SPONTANEOUS DECISIONS IN AUTONOMOUS CARS A Dissertation Presented to the Graduate School of Clemson University In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Automotive Engineering by Pablo Sauras-Perez May 2017 Accepted by: Dr. Pierluigi Pisu, Committee Chair Dr. Joachim Taiber Dr. Yunyi Jia Dr. Mashrur Chowdhury ABSTRACT Autonomous cars are expected to improve road safety, traffic and mobility. It is projected that in the next 20-30 years fully autonomous vehicles will be on the market. The advancement on the research and development of this technology will allow the disengagement of humans from the driving task, which will be responsibility of the vehicle intelligence. In this scenario new vehicle interior designs are proposed, enabling more flexible human vehicle interactions inside them. In addition, as some important stakeholders propose, control elements such as the steering wheel and accelerator and brake pedals may not be needed any longer. However, this user control disengagement is one of the main issues related with the user acceptance of this technology. Users do not seem to be comfortable with the idea of giving all the decision power to the vehicle. In addition, there can be location awareness situations where the user makes a spontaneous decision and requires some type of vehicle control. Such is the case of stopping at a particular point of interest or taking a detour in the pre-calculated autonomous route of the car. Vehicle manufacturers’ maintain the steering wheel as a control element, allowing the driver to take over the vehicle if needed or wanted. This causes a constraint in the previously mentioned human vehicle interaction flexibility. Thus, there is an unsolved dilemma between providing users enough control over the autonomous vehicle and route so they can make spontaneous decision, and interaction flexibility inside the car. ii This dissertation proposes the use of a voice and pointing gesture human vehicle interaction system to solve this dilemma. Voice and pointing gestures have been identified as natural interaction techniques to guide and command mobile robots, potentially providing the needed user control over the car. On the other hand, they can be executed anywhere inside the vehicle, enabling interaction flexibility. The objective of this dissertation is to provide a strategy to support this system. For this, a method based on pointing rays intersections for the computation of the point of interest (POI) that the user is pointing to is developed. Simulation results show that this POI computation method outperforms the traditional ray-casting based by 76.5% in cluttered environments and 36.25% in combined cluttered and non-cluttered scenarios. The whole system is developed and demonstrated using a robotics simulator framework. The simulations show how voice and pointing commands performed by the user update the predefined autonomous path, based on the recognized command semantics. In addition, a dialog feedback strategy is proposed to solve conflicting situations such as ambiguity in the POI identification. This additional step is able to solve all the previously mentioned POI computation inaccuracies. In addition, it allows the user to confirm, correct or reject the performed commands in case the system misunderstands them. iii DEDICATION Papa, Mama, Borja, Carlos ... Family … Friends … This is for you. Gracias! iv ACKNOWLEDGMENTS I would like to express my gratitude to all the teachers that I had the honor to learn from during this journey. A complete set of learning experiences in courses of automotive, electrical, civil and industrial engineering have been invaluable resources to expand my view of what the future of mobility will look like. My special gratitude to Dr. Pierluigi Pisu, my academic advisor, for all his support, time, challenging questions, and guidance, that helped me so much in shaping this research. Dr. Joachim Taiber has been there from the beginning. His vitality and passion made him the perfect mentor. Special thanks also to Dr. Mashrur Chowdhury and Dr. Yunyi Jia for their invaluable feedback during this process. The seed of this dissertation started with a really successful team work. Team CUICAR Tigers: I cannot thank you enough for that time and our friendship. In this regards, I am also thankful to Dr. David Smith for his support during that process. I want to express my gratitude to all the friends I made during this time (so many of them that I rather not to personalize here). You made my life here a little bit more enjoyable during these years. Finally I would like to thank to my family and friends from Spain for their unconditional support and the excitement that they demonstrated to me during this fulfilling experience….Gracias! v TABLE OF CONTENTS Page I. CHAPTER ONE. INTRODUCTION ......................................................................... 1 A. Objective .............................................................................................................. 1 B. Motivation ............................................................................................................ 2 C. Research Contributions ........................................................................................ 5 D. Broader Impacts ................................................................................................... 7 E. Research scope ..................................................................................................... 8 F. Dissertation organization...................................................................................... 8 II. CHAPTER TWO. AUTONOMOUS VEHICLES BACKGROUND ...................... 11 A. Levels of Vehicle Automation ........................................................................... 11 B. Autonomous Vehicles Societal Challenges........................................................ 12 C. Autonomous Vehicles Approaches. The Interaction Flexibility vs. User Control/Spontaneity dilemma. ...................................................................................... 14 1. OEMs approach .................................................................................................. 14 2. Waymo approach................................................................................................ 15 III. CHAPTER THREE. LITERATURE REVIEW ....................................................... 18 A. Secondary Functions Control ............................................................................. 19 1. Device Control ................................................................................................... 19 2. Situated Awareness Interactions ........................................................................ 21 B. Primary Functions Control ................................................................................. 22 1. Autonomous cars control ................................................................................... 23 2. Mobile robots guidance ...................................................................................... 24 C. Dialog feedback systems .................................................................................... 25 D. Finite State Machines (FSM) for autonomous vehicles ..................................... 26 IV. CHAPTER FOUR: SYSTEM DESIGN ................................................................... 27 A. Flow diagram...................................................................................................... 27 B. Interaction command structure ........................................................................... 30 vi Table of Contents (Continued) Page C. System Architecture ........................................................................................... 32 D. Simulation Framework ....................................................................................... 37 1. 3D virtual environment ...................................................................................... 37 2. System processing environment ......................................................................... 37 V. CHAPTER FIVE: POINT OF INTEREST COMPUTATION ................................ 40 A. Introduction ........................................................................................................ 40 B. POI computation without using external sensor information............................. 42 1. Method ............................................................................................................... 42 2. Simulation Results.............................................................................................