Living with Harmony: a Personal Companion System by Realbotix™
Total Page:16
File Type:pdf, Size:1020Kb
Living with Harmony: A Personal Companion System by Realbotix™ Kino Coursey, Susan Pirzchalski, Matt McMullen, Guile Lindroth, and Yuri Furuushi Abstract Existing personal assistants and agents are by design limited in their abil- ity to form or encourage close personal bonds. The Harmony system is designed to be a customizable personal companion agent capable of close personal interaction via the user's phone, virtual reality headset, as well as through a physical interactive android body. In this chapter, we will describe the history that led to Harmony’s creation, the unique challenges and the overall system design. We will also look at user reactions to the system and anticipated future developments. Keywords Androids · Personal assistant · Virtual reality · Realbotix · Embodied agent · Companion agent K. Coursey (*) AI Research, Realbotix, Inc., Dallas, TX, USA e-mail: [email protected] S. Pirzchalski Hardware and System Engineering, Realbotix, Inc., Dallas, TX, USA e-mail: [email protected] M. McMullen Realbotix, Inc., San Marcos, CA, USA e-mail: [email protected] G. Lindroth Content and AI Research, Realbotix, Inc., Curitiba, Paraná, Brazil e-mail: [email protected] Y. Furuushi Application Developement, Realbotix, Inc., Curitiba, Paraná, Brazil e-mail: [email protected] © Springer Nature Switzerland AG 2019 77 Y. Zhou, M. H. Fischer (eds.), AI Love You, https://doi.org/10.1007/978-3-030-19734-6_4 78 K. Coursey et al. 1 Introduction Androids (robots designed to resemble a human in appearance and behavior) have been a staple of both popular culture and a field of active research and development. They have been the topic of movies and television and have been used to simulate individuals both living and dead. However, until recently the technology to produce an attractive android has been prohibitively expensive and fragile for the average consumer or researcher. Parallel to the increased interest in androids is the develop- ment of voice interactive personal assistant agents and chatbot technology. Unfortunately, policy restrictions on the topics of conversations have limited some interaction possibilities for such personal agents. That is, they are limited in their ability to form close bonds with their users by design. The Harmony system (Fig. 1) is designed to be a customizable personal compan- ion agent capable of interaction via the user’s phone, a virtual reality (VR) headset, as well as through a physical interactive android body. The agent creation system allows the user to define both the visual appearance of the agent’s avatar and person- ality traits. These traits are made active across all interaction platforms. The conver- sational system is designed to address mature topics that other conversational agents are designed to shy away from. Finally, the agent can interact continuously with the user via a phone app, then transition to controlling the android head on a physical body. In addition to being a consumer product, Harmony also provides a physical plat- form for exploring human–robotic interaction. In this chapter, we will describe the history that led to Harmony’s creation and the overall system design. We will also look at user reactions to the system and anticipated future development. Fig. 1 Harmony v0.8 Living with Harmony: A Personal Companion System by Realbotix™ 79 1.1 Realbotix™ and Its Prehistory Harmony was created by Realbotix™, a joint venture made up of three organiza- tions: Abyss Creations/RealDoll, Daxtron Labs, and NextOS. Abyss Creations/ RealDoll produces high-end life-size silicone love dolls for close to 20 years and helped define the higher end of quality in the market over that time span. RealDoll was originally conceived as fashion mannequin that could pass a “fast visual Turing test,” in that at a glance an unobservant observer might not notice that the manne- quin was not human (Fig. 2). Daxtron Labs is an advanced technology consultancy focused on AI and communications solutions. Prior to Realbotix™, Daxtron pro- vided support for Hanson Robotic’s Bina48, PKD2, and Sophia androids. Since 2008, Daxtron Labs has provided assistance to Hanson Robotics in the area of con- versational processing, speech processing, and integrated general artificial intelli- gence technology. Daxtron adapted the CogBot system developed for the OpenCog project to provide conversational interaction and to act as an interface to external systems such as question answering engines and cloud services. NextOS is a continuation of Guile3D of Brazil which has provided desktop and mobile agents for a number of years. Most recently, NextOS produced the Denise desktop assistant with home automation and Internet of Things (IoT) features (Fig. 2). Various media outlets such as movies and science fiction have generated high expectations for the eventual creation of adult-oriented humanoid robotics. In 2010, there was a media frenzy over the then attempted introduction of an interactive doll called Roxxy by True Companions (Wikipedia contributors, 2018a). Given a background in embodied chatbot development, in 2010 Daxtron Labs approached Abyss for obtaining a doll with modifications for sensor placement. Matt McMullen (CEO of Abyss) expressed interest and support, and over the ensu- ing years ideas and possible designs were shared. This resulted in a prototype head with voice interaction, orientation sensing, and vision (face detection, face tracking, and object recognition). Daxtron also developed an embedded mini-Beowulf cluster processor design called the Quad-Processor Extended. Fig. 2 Various RealDolls including portrait-based “Wicked” line and NextOS Denise personal assistant 80 K. Coursey et al. Fig. 3 Protype circa 2013–2014 Fig. 4 Harmony V0 with solid eyelids and large format eyes In 2015, The New York Times (Canepari, Cooper, & Cott, 2015) produced a docu- mentary discussing the impact of robotics on society, and the Realbotix™ project was formed with the addition of NextOS (who had independently contacted Abyss in the past and whom Daxtron was familiar with from their work in conversational agents). Daxtron developed the physical hardware and electronics (Fig. 3), while NextOS produced the conversational front-end, and together developed a communications protocol allowing the phone-based avatar to animate the physical robot. Daxtron continued experiments with vision processing using the Nvidia TX1 and TX2 embedded processors. Abyss modified an existing BoyToy doll for the body and face of the “V-zero” version of Harmony (Fig. 4). After the V0 was developed, the working relationship between the groups was formalized, and a general system architecture was defined. Living with Harmony: A Personal Companion System by Realbotix™ 81 1.2 Harmony Harmony is the result of a common vision of the three organizations that make up Realbotix™. Harmony is designed to be an interactive and functional one-to-one scale model/action figure of the future. As such she is designed to lead the imagina- tion of users, developers, and hopefully researchers. She is also designed to be what other assistants cannot be, which is personal, allowing the user to define her appear- ance and personality. The physical hardware and electronics are designed to provide an easy to interface and control a physical avatar. The appearance is also designed to exploit Abyss’ experience with producing artistic dolls that are typically viewed as being outside the uncanny valley. 2 Related Work Highly expressive and lifelike androids have been and are currently being devel- oped. Hanson Robotics has produced a series of robotic heads with high expressiv- ity. PKD v1 and PKD v2 (Hanson et al., 2005), HUBO Einstein (Hanson et al., 2006), Jules, BINA48 (Harmon, 2010; Wikipedia contributors, 2018b), Han, and Sophia and are some examples of their continuing work to merge artistic polish with advanced faces exhibiting high degrees of freedom and advanced AI techniques. Many of the Hanson Robotics systems have taken the stance of being “personality portraits,” being dynamic models of the persona of either existing people or created characters. The Hiroshi Ishiguro Laboratories at ATR have also produced a number of humanoid robots (Liu, Glas, Kanda, Ishiguro, & Hagita, 2014; Mori, MacDorman, & Kageki, 2012). Their Geminoid androids (Becker-Asano & Ishiguro, 2011; Nishio & Hagita, 2007) were developed to explore the requirements for natural human interaction with androids and teleoperation. This research has continued with the development of the ERICA (Glas, Minato, Ishi, Kawahara, & Ishiguro, 2014) android which advances research in conversational interaction. A number of chatbots and personal assistants already exist. One can reference the Loebner Prize (Home Page of The Loebner Prize in Artificial Intelligence, n.d.) and the various chatbots that participate in the contest. However, these agents are purely textual. In contrast, there are intelligent personal assistants like Apple Siri (n.d.), Amazon Alexa & Echo (n.d.), Clarity Labs open source Sirius/Lucida (n.d.), Google Now (n.d.), Hauswald et al. (2015), Microsoft Cortana (n.d.), and Samsung Bixby (n.d.). All of these agents provide factual assistance. As assistants, their goal is to provide task-focused help and interface to a number of web services. Most have dual text and speech interfaces, with some accepting visual input. Even though they are in constant contact with their users, they are not focused on idle chit-chat, open- ended conversations or relationship building. They excel in focused task-oriented dialogs and interactions, by providing a natural interface to question answering ser- vices, recommendation services, and performing user-directed actions. 82 K. Coursey et al. 3 System Description The project’s inspiration can be summarized by a simple question: “What if the OS’s of the movie Her (Wikipedia contributors, 2018c) could not only interact with their users via phones or their desktops but they could also use VR or they had access to their own bodies?” The Harmony project aims to be a consumer purchas- able simulation of that experience (Fig. 5). The software system supports simulating a basic relationship between a user and their personal assistant/companion.