First Steps toward Natural Human-Like HRI
M. Scheutz, P. Schermerhorn, J. Kramer, and D. Anderson Artificial Intelligence and Robotics Laboratory Department of Computer Science and Engineering University of Notre Dame, Notre Dame, IN 46556, USA {mscheutz,pscherm1,jkramer3,danderso}@cse.nd.edu
Abstract not even close. Yet, we believe that is not too early to start a discussion of possible requirements for NHL-HRI, given Natural human-like human-robot interaction (NHL-HRI) current achievements in HRI and knowledge of robotic ar- requires the robot to be skilled both at recognizing and chitectures. Specifically, we believe that it will be critical producing many subtle human behaviors, often taken for for future robotic efforts in HRI to investigate architec- granted by humans. We suggest a rough division of these tural structures, principles, and concepts that necessarily requirements for NHL-HRI into three classes of proper- (or even potentially) have a role in robots capable of NHL- ties: (1) social behaviors, (2) goal-oriented cognition, and HRI. (3) robust intelligence, and present the novel DIARC ar- In this paper, then, we will start with a modest reflection chitecture for complex affective robots for human-robot on three classes of properties that we deem crucial to suc- interaction, which aims to meet some of those require- cessful natural HRI, with an eye towards NHL-HRI. We ments. We briefly describe the functional properties of then present a brief overview of our own attempts at defin- DIARC and its implementation in our ADE system. ing a Distributed Integrated Affect, Reflection, and Cog- Then we report results from human subject evaluations nition architecture (DIARC), as a first step towards ar- in the laboratory as well as our experiences with the robot chitectures for NHL-HRI. After discussing the functional running ADE at the 2005 AAAI Robot Competition in organization of DIARC and the role of affect in the in- the Open Interaction Event and Robot Exhibition. tegration of its various subsystems, we briefly describe some of the implemented components, and compare re- sults from testing DIARC in human subject experiments 1 Introduction in the laboratory with our experiences of the robot’s inter- actions with people in the real-world context of the AAAI We take the ultimate goal of human robot interaction 2005 robot competition, specifically the Open Interaction (HRI) to be the achievement of natural and human-like Event and Robot Exhibition. Finally, we provide a sum- (NHL) robot behavior as it relates to human contact. By mary of the current state of DIARC and conclude with this, we mean that our intention is to establish a robotic some thoughts on the role of affect in NHL-HRI-capable architecture for HRI such that any restrictions on possible robots. interactions are due to human capacities (i.e., the limita- tions of human perceptual, motor, or cognitive system), and not the a priori functionality of the robot. For ex- 2 The DIARC Architecture ample, an interaction that would require humans to speak at ten times the normal speech rate would be excluded, Natural human-like human-robot interaction places many as would interactions that required humans to hear ul- complex demands on a robotic architecture. As a first trasound or communicate in some non-human language. rough cut, we can divide them into three main categories: However, we do want to include interactions that can oc- (1) social behaviors, (2) goal-oriented cognition, and (3) cur in any typical human setting, such as the ability to use robust intelligence. The first category includes all aspects language freely in any way, shape, or form; to make ref- of human communicative acts, the second is concerned erence to personal, social, and cultural knowledge; or to with all forms human cognition and teleological behavior, involve all aspects of human perception and motor capa- and the third centers around various mechanisms that en- bilities. sure the reliable, long-term, fault-tolerant autonomy and Clearly, NHL-HRI is not an achievable goal for any survival of the robot. We will now briefly expand on each robotic system in the foreseeable future; in fact, we are of these three categories.
1 First and foremost, it is immediately clear that robots humans in social interactions; for instance, speech recog- must be capable of natural language processing if hu- nition, even when perfect, makes up only one part of the mans are to be free to use (spoken) language whenever meaning of an utterance. Expressions of affect (e.g., via and in whatever form they please. In addition to speech gestures, facial expressions, or prosodic characteristics, recognition and production, natural language interactions see Ekman, 1993) augment or modify the explicit seman- will require methods for semantic processing of language tic content of statements (e.g., a sarcastic tone might indi- structures and natural language understanding. Moreover, cate that the speaker’s belief is the opposite of the spoken knowledge of dialog structures, dialog progression, and content, or emphatic gestures might indicate the strength teleological discourse is required for the robot to be able of the speaker’s belief). To create accurate representations to engage in natural communicative interaction patterns of others’ mental states based on conversation, it is nec- (e.g., Grosz & Sidner, 1990). This also includes the recog- essary to “pick up on” these cues. Moreover, humans ex- nition of human affect and the appropriate expression of pect and look for these cues when processing the robot’s affect on the part of the robot (e.g., in response to rec- speech output, so the inclusion of affect expression capa- ognized affect), as well as mechanisms for recognizing bilities should enhance the degree to which humans feel and producing other non-verbal cues (e.g., gestures, head they can accurately assess the robot’s belief states (the movements, gaze, etc.) that accompany human discourse utility of affect expression will be demonstrated in Sec- and social interactions. tion 3.1). Second, genuine natural interactions require the robot Affect can also be an effective way for higher-level to communicate, instill, and prompt the ascription of in- deliberative mechanisms in an agent architecture to con- tentionality (i.e., the human ability to treat systems as if nect to and utilize motivational mechanisms of lower- they had their own intentions). Human interlocutors will level non-deliberative components. Specifically, delib- automatically watch for behaviors that convey intent (e.g., erative mechanisms can alter the states of these compo- as established by non-verbal cues) and assume that the nents (e.g., by injecting new “force” into “affective cir- robot has the ability to recognize and utilize such behav- cuits” or by suppressing output of those circuits) to cre- iors in others. As part of being able to present itself con- ate and modify existing goals, directly influence the con- sistently and over extended periods of time as a purpose- trol of action (e.g., by changing the preferences in the driven entity, the robot will require genuine purposes, rep- agent’s action selection mechanism), and drive learning resented as goals and implemented in internal goal and based on internally generated valuations and value sig- task management mechanisms.1 Their absence will have nals (Scheutz, 2000). Thus, affect may serve the purpose disruptive effects on the natural flow of conversation and, of integration and management of multiple processes re- eventually, the overall interaction. Ultimately, the robot quired for the effective functioning of an autonomous sys- has to behave in a way that supports a consistent human tem (Ortony, Norman, & Revelle, 2005); affect allows for “theory of robot minds,” which is the human ascription of motivational signals originating not from changes in the human-like beliefs, intentions, and desires that make the external environment detected via sensors, but from com- robot predictable to humans. ponents within the architecture itself (e.g., from deliber- Third, the architecture must include mechanisms to re- ative subsystems). Such signals can then influence vari- cover both from failures within the system (e.g., acous- ous other parts of the architecture and modify goal man- tic, syntactic, semantic misunderstandings, dialog fail- agement, action selection, and learning (e.g., see Scheutz, ures, etc.) as well as failures of the system itself (e.g., 2004, where we isolated 12 functional roles of emotions crashes of components, internal timing problems, faulty in an agent architecture). hardware, etc.). Finally, whereas a complex cognitive evaluation of a situation may provide a more accurate assessment (and 2.1 The Utility of Affect for Controlling In- likely a better subsequent action selection) than that pro- vided by an affective evaluation, it may prove too costly formation Flow in Agent Architectures or time-consuming to be practical for real-time use. For Affective robotic architectures can directly address many fast, low-cost approximations, humans seem to use affec- demands for HRI, in addition to providing many bene- tive memory (Bless, Schwarz, & Wieland, 1996), which fits for other robot tasks. Affect plays a critical role for encodes implicit knowledge about the likelihood of occur- rence of a positive or negative future event (Clore, Gasper, 1Note that the emphasis here is on both “consistency” and “extended & Conway, 2001). Robotic agents can use such affective time period” as humans can sometimes be tricked into believing that states as subjective probabilities in affective evaluations something has a purpose because it seems to exhibit purposeful behavior of potential actions. This can be useful in quickly deter- for short periods of time. However, the deception will typically not last for long (e.g., see the repeatedly failed attempts at convincing humans mining an appropriate reaction to catastrophic failures in that a computer is a human in the Loebner prize competition). the system (e.g., the failure of the vision subsystem).
2 Sensors Perceptual Processing Central Processing Action Processing Effectors
Battery Affect Threat Reflexes charge detection appraisal PX AA RF Bumper Collision Affect Action Motion Wheel sensors detection appraisal selection control motors PX AA AS MC Sonar Proximity Affect Plan sensors detection appraisal execution PX AA AS Laser rng. Motion Affect Affect finder detection appraisal expression OJ AA AX Leg Object Localiz− Path Action Camera Camera detection recogn. ation planning selection control motors OJ OR LZ LZ AS CC Wheel Visual Visual Face encoders STM Attention Tracking OR VP OT Motion Face Visual Visual Object Camera detection detection Learning LTM tracking framegr. OJ OR VP VP OT Object Affect High−level Goal detection recogn. goals manager OJ AR HG GM Task LTM Affect Task STM Reflexes appraisal (scripts) (goal stk.) AA ME TM RF Affect Declar. Concept. Task Affect appraisal LTM LTM Manager expression AA ME ME TM SR Affect Diction− Semantic Text−to− Sound Speakers recogn. ary analysis speech control AR SP SP SR SR Micro− Sound Voice Word Sentence Sentence phones detection detection recogn. parsing production SO SO SP SP AX
Figure 1: A high-level view of the functional components in the proposed DIARC architecture for complex human-like robots. Boxes depict concurrently running components of varying complexity. Solid arrows depict information flow and dashed arrows depict control flow through the architecture (the latter via different affective processes). Only links pertaining to affect processing are shown. Labels (black ovals) are included to relate architectural components to their counterparts in the implementation diagram shown in Figure 2.
2.2 A Brief Overview of DIARC ceived threat, rather than considering the costs and bene- fits of stopping, comparing those to the costs and ben- Figure 1 depicts a partial view of the functional orga- efits of alternative actions, and selecting the best action nization of the proposed affective architecture for com- overall). The central processing step performs action se- plex robots: DIARC, the Distributed Integrated Affect, lection based on the perceptual input, concept memory, Reflection, Cognition architecture. Sensors, effectors, cost-benefit analysis, etc. At the level of action process- and perceptual, central, and action processing compo- ing, directives from the perceptual and central processing nents are separated into columns. All boxes depict au- stages generate commands for the effectors, with conflicts tonomous computational units that can operate in paral- generally resolved in favor of commands from perceptual lel and communicate via several types of communication processing (e.g., the reflexive STOP command overrides links. Names of components denote their functional roles movement commands generated by plan execution). This in the overall system. is accomplished by priority-based action selection mech- At a high level, an agent that implements DIARC op- anisms (e.g., Scheutz & Andronache, 2004). erates as follows: sensory information is gathered via the various sensors and passed on the the appropriate per- 2.3 The Vision System ceptual processing components. In perceptual processing, raw sensory input is (potentially) parsed into meaningful The vision system has two components: a social compo- perceptual data. Much of these data are accessed by el- nent for detecting and tracking faces and facial features ements of the central processing group for goal and task and a recognition component for object recognition, learn- management. In addition, perceptual data are routed to af- ing, and memorization. The social subsystem, for exam- fective appraisal modules, which make fast evaluations of ple, provides information about detected faces using his- the potential (positive or negative) effects they may imply togram methods from Yang, Kriegman, and Ahuja (2002) for the robot. This may lead, in turn, to “reflexive” re- as well as skin color, color of clothes, and the camera actions that bypass the entire central processing step (e.g., pan and tilt angles, which are used in conjunction with causing the robot to stop immediately in response to a per- information about the distance of an object (as reported
3 by sonar and laser sensors) to identify and track people in 2.4 Natural Language Processing a room (Scheutz, McRaven, & Cserey, 2004) and deter- The natural language processing subsystem integrates and mine some of their salient features such as height, based extends various extant components. Speech recognition is on distance and camera tilt angle (Byers, Dixon, Good- performed using SONIC (Pellom & Hacioglu, 2003) and ier, Grimm, & Smart, 2003) for future identification. Fea- Sphinx (The Sphinx Group at Carnegie Mellon Univer- ture extraction of the eyes and mouth is performed using a sity, 2004); parsing is handled by the link parser (Sleator swarm-based exploration of a parameter space to find pa- & Temperley, 1993). Verbnet (Kipper, Dang, & Palmer, rameters for a Canny edge detector that produce the best 2005) and Framenet (Fillmore, Baker, & Sato, 2002) are features within a face region given by the OpenCV Haar used for semantic processing, in conjunction with a mod- cascade (Middendorff & Scheutz, 2006). Motion of the ified version of Thought Treasure for natural language tracked features gives insight into emotional changes of understanding and speech production (Mueller, 1998). the subject, as seen in Ekman and Friesen (1977). For Speech synthesis is handled by the University of Edin- example, a sudden raise in the eyebrows can signal hap- burgh’s Festival system (Festival, 2004), augmented by piness or surprise, whereas downward movement can in- an emotional output filter (Burkhart, 2005). In addition, dicate frustration or confusion. The general shapes and our own components are employed for affect expression positions of features are maintained in a hash table keyed in spoken language (e.g., “angry”, “frightened”, “happy”, by face and leg positions, allowing emotional states to be “sad”, and their gradations, such as “halfangry”, which tracked simultaneously for several people. The emotion indicates a somewhat elevated state of anger). recognition component currently only classifies faces as “happy”, “sad” or “neutral”. Object recognition was implemented using scale- 2.5 Action Interpretation and Selection invariant feature detection (SIFT) points for the extraction The action control subsystem (Scheutz, Schermerhorn, of distinctive features from images (Lowe, 2004). SIFT Kramer, & Middendorff, 2006) is based on a novel affec- keypoints are invariant with respect to image scale, rota- tive action interpreter, which interprets scripts (Schank tion, change in 3D viewpoint and change in illumination, & Abelson, 1977) stored in long-term memory. Scripts and can be used to perform a variety of tasks including encode the robot’s procedural knowledge of certain con- reliable object recognition, even in situations where the versation “templates” as well as complex action se- scene is cluttered or the object is partially occluded. quences for task performance. These scripts can be com- Object recognition is a two-phase process for unknown bined in hierarchical and recursive ways, yielding com- objects, composed of learning an object and subsequent plex behaviors from basic behavioral primitives, which memory recall. When learning an object, the SIFT key- are grounded in basic skills (Ichise, Shapiro, & Lan- points of an image are translated to an ASCII represen- gley, 2002). Moreover, several spatial maps are used 2 tation and a distinct identification token is stored in a for the representation of locations of the robot, peo- database. During recall, the system generates an ASCII ple, and other salient objects in the environment, as representation of the keypoints in the current scene. This well as for path planning and high-level navigation. As representation is compared to entries in the database us- a trivial example, the following script (activated from ing a nearest-neighbor algorithm, allowing the system to a higher-level script as notify-beverage-done(self, answer queries such as “Is object X in the scene?” and Jim, coffee, desk3)) instructs the robot to move to lo- “Which objects in the database are in the scene?” Once cation desk3 (retrieved from a map in long-term memory) objects have been positively identified, it is possible to shift its focus of attention to the human there, and tell the perform additional queries concerning their spatial rela- human that his coffee is ready: tionships (e.g., “Is object X to the left of object Y?” etc.). SIFT-based object recognition performs most effec- ====notify-beverage-ready//serve.V tively when attempting to identify rigid objects with a dis- role01-of=robot| tinctive pattern. It reliably differentiates between the cov- role02-of=human| role03-of=beverage| ers of various textbooks, assorted computer components role04-of=loc| and the boxes of various household products. However, timeout-of=600sec| fewer keypoints are detected on curved surfaces, with a event01-of=[move-to robot loc]| correspondingly lower recognition rate for objects such event02-of=[shiftFOA robot human]| as balls or aluminum cans, made worse if the objects lack event03-of=[say-to robot human distinctive markings and surface features. [human your beverage is ready.]]|
2The ASCII representation is generated with the binary program dis- The action interpreter substitutes the arguments passed for tributed on Lowe’s website http://www.cs.ubc.ca/∼lowe/keypoints/. each of the roles in the script and begins executing the
4 first event. A behavioral primitive like move-to(robot, 2.6 System Infrastructure loc) then has a particular meaning to the robotic system. There are certain aspects of a complex robot that are In this case, the action interpreter passes the action on to critical for long-term, safe, and flexible operation. The the navigation system, which interprets it as a command computational demands of various sub-systems require to move the robot to the coordinates (x,y) of loc (repre- distributing the architecture across hosts, allowing con- sented in a discrete, topological map). The high-level nav- current operation while retaining system reactivity. Fur- igation system generates a plan which translates the action thermore, the importance of monitoring and maintain- into commands for the low-level navigation system, even- ing system integrity and health (including error detec- tually causing the robot to move in a particular direction, tion, system reconfiguration, and failure recovery) can- if possible (e.g., it will not move there if obstacles block not be overstated. DIARC is implemented within ADE, the location, although it will attempt to move around ob- the Architecture Development Environment (Andronache stacles that obstruct the path to the final location). & Scheutz, 2006; Scheutz, 2006), which provides a multi- In addition to action primitives or references to agent based infrastructure. ADE is not an architecture it- other scripts, scripts also support conditional execution self, but a framework for developing, debugging, and de- whereby the next step is determined by the outcome of ploying complex agent architectures based on the APOC the present event. In this way, failure recovery actions universal agent architecture formalism (Scheutz & An- can be encoded directly in the scripts. For example, sup- dronache, 2003; Andronache & Scheutz, 2004); ADE in- pose there were no human at desk3 in the above exam- corporates various tools and mechanisms that promote re- ple. In that case, the shiftFOA(human) goal cannot be liable and flexible system operation (Kramer & Scheutz, achieved, so a more appropriate action would be to end 2006b, 2006a). notify-beverage-ready in its failure state (which can The basic component in ADE is the ADESERVER, in turn be detected and addressed appropriately by the which is comprised of one or more computational pro- calling script), rather than delivering the message to no- cesses that serve requests. Accessing the services pro- body and indicating successful completion. vided by an ADESERVER is accomplished by obtaining A failure such as the one described above causes an ad- a reference to the (possibly remote) server, forming a lo- justment to the robot’s affective state, which subsequently cal representation that is referred to as an ADECLIENT. allows for an additional context-based adaptation of goals, The ADEREGISTRY, a special type of ADESERVER, me- preferences, attitudes, and, ultimately, behavior. Affec- diates connections among servers and the processes that tive states do not explicitly influence action selection (i.e., use their services. In particular, it organizes, tracks, and there is no branching in scripts based on affective states). controls access to servers that register with it, acting in a Instead, affect influences the priorities assigned to the role similar to a white-pages service found in multi-agent goals currently held by agent. For each goal currently systems. The ADEREGISTRY provides the backbone of held by the agent, there is an associated script interpreter an ADE system; all components must register to become that manages the execution of script events to achieve that part of the architecture. A set of components may contain goal. These script interpreters execute concurrently, so multiple registries that mutually register with one another multiple goals may be advanced at the same time. How- to provide both redundancy and the means to maintain dis- ever, when conflicts arise (e.g., when two scripts require tributed knowledge about the system. the same physical resource, such as motor control), they All connected components of an implemented ar- are resolved in favor of the goal with the highest priority. chitecture (that is, ADESERVERs, ADECLIENTs, and importance A goal’s priority is based on its (i.e., benefits ADEREGISTRYs) maintain a communication link during minus cost scaled by affective evaluations, see Scheutz et system operation. At a bare minimum, this consists of a al., 2006) and its urgency, which reflects the likelihood periodic heartbeat signal indicating that a component is that there will be sufficient time remaining to complete still functioning. A server sends a heartbeat to the registry the script. By mediating the importance component of with which it is registered, while a client sends a heart- goal priority, affect can effectively alter the robot’s per- beat to its originating server. The component receiving ception of a goal’s utility. Positive affect leads to more the heartbeat periodically checks for a heartbeat signal; “optimistic” assessments of utility (and, hence, higher pri- if none arrives, the sending component receives an error, ority), whereas negative affect leads to more “pessimistic” while the receiving component times out. An ADEREG- 3 assessments of utility. ISTRY uses this information to determine the status of servers, which in turn determines their accessibility. An 3The current implementation of the action interpreter is still some- what impoverished, as variables for other scripts have not been imple- scripts and match them against the preconditions Xi. Also, the current mented yet. For example, it is not possible to add “variable actions” to implementation only supports detection of failures, but not “recursive” scripts such as “pick any script that satisfies preconditions Xi and exe- attempts to recover from them – “recursive”, for recovery actions might cute it”, which would cause the action interpreter to search through its themselves fail and might thus lead to recovery from recovery, etc.
5 Perceptual Action W − Wheel Encoder Sensors Processing Central Processing Processing Effectors Ba − Battery LZ Bu − Bumper Device W RF MC Mo So Ba − Sonar Device PX L − Laser Device Bu HG GM Mo − Motor Device AA So OT CC C C − Camera Device L VP TM Mi − Microphone OR C SR Sp − Speakers OJ AS Sp Mi SP − Architectural Link SO AR ME AX PX − Proximity AA − Affect Appraisal Abstract Agent Architecture OR − Object Recognition ADE Components OJ − Object Detection LZ Mapping and SO − Sound Detection Logger Localization SP AR − Affect Recognition Sentence Parser LZ − Localization HG − High−level Goals VP OT CC SO SP GM − Goal Manager Image Processing ADERegistry Speech Recognition VP − Visual Processing C Mi TM − Task Manager SP − Speech Processing PX OJ OR HG GM TM ME − Memory Leg/Obstacle − Reflexes ME AS RF L Action Interpreter MC − Motion Control and Execution OT − Object Tracking CC − Camera Control AR AA PX RF MC AX SR SR − Speech Production Affect Recognition Robot Speech Production AS − Action Selection Mi Sp AX − Affect Expression W Ba Bu SoMo ADE Components − ADEServer Hardware/Network − Heartbeat Only − Data and Heartbeat − Data Wire − Network Laptop−1 Laptop−2 Onboard PC
Figure 2: Three representations of the proposed DIARC architecture for complex human-like robots. The bottom level depicts the system (or hardware), the middle level depicts the multi-agent system (or ADE components), and the top level depicts a simplified view of the DIARC agent architecture shown in Figure 1. Note that the ADEREGISTRY and “Logger” components are part of the infrastructure, not part of DIARC itself.
ADESERVER uses heartbeat signals to determine the sta- ent subsystems of DIARC as well as the viability of the tus of its clients, which can then determine if the server’s overall architecture. For example, the robot was prepared services remain available. to perform an object recognition task (Section 3.2) which Figure 2 shows a “3-level” view of the architecture; the tested the vision system and its short- and long-term mem- top depicts a partial view of the abstract DIARC architec- ories in conjunction with natural language processing . ture, while the middle and bottom depict its breakdown The robot was asked questions about objects in its visual into components in ADE and the hardware on which it field (e.g., “What is this?”, “How many coke cans do you executes, respectively. The robot platform used is an Ac- see?”); if an object was unknown, the robot was told what tivMedia Peoplebot with an on-board computer and two it was and could successfully re-identify it at a later time. additional laptops (all running Linux with a 2.6.x ker- Another relatively basic task demonstrated on the robot nel). Available hardware includes a pan-tilt-zoom camera, was the take orders task, where the robot was able to per- a SICK laser range finder, three sonar rings, two micro- form a limited set of actions to confirm operability of the phones, two speakers, a local Ethernet network, and one natural language processing, localization, and motor con- wireless link to the “outside world.” trol (e.g., “Move forward 2 meters”, “Turn left”). More ambitious was the Open Interaction task (also de- 3 DIARC Applications scribed in Section 3.2), which tested system cohesion and performance. In this task, the robot wandered an open Over the last two years, DIARC has been used in vari- area, detected nearby people, approached them, and initi- ous applications, from informal laboratory evaluations, to ated conversations. human subject experiments, to public demonstrations and The waiter task further explored the functionality of the robot competitions to evaluate the functionality of differ- entire system, with a focus on high-level task and envi-
6 ronment knowledge, in addition to cognitive scripting ca- induced in subjects by virtue of a warning message ut- pabilities. In particular, a set of scripts encoding typical tered by the robot: “I just noticed that my battery level is interactions in a waiter/patron scenario were defined and somewhat low,
7 The results then suggest that the content of the mes- in the lab prior to the competition. Although not perfect, sage was insufficient, in itself, to trigger in the no-affect the robot performed reasonably well in that controlled en- subjects the belief that the robot might be “stressed,” even vironment. It would wander and find people, engage in though the robot was in exactly the same internal (stress) conversation, and then move on, and it would execute state with respect to its goal priorities and deadlines as its demonstration tasks fairly consistently. However, the in the affect conditions (the only architectural difference practical realities of the conference site proved more trou- between the two conditions was that affective modulation blesome than anticipated. of the robot’s voice based on its internal states was sup- pressed in the no-affective condition). Rather, the affect Open Interaction. The large crowd (and attendant con- modulation of the robot’s voice seems to have contributed versational hubbub) was problematic for the system, as to the attribution of stress to the robot by subjects in the were many physical features of the exhibition area (e.g., affect condition. mirrors and tablecloths). The robot traversed the open in- In sum, we believe that the results based on objective teraction environment fairly well, but was unable to con- and subject evaluations demonstrate that appropriate af- sistently engage people in conversations. fect expression (that is congruent with people’s own affec- The lack of automatic failure recovery mechanisms in tive states) can help humans in construing a mental model the infrastructure led at times to extended periods of in- of a robot, which makes the robot’s “mental states” trans- activity while manual recovery was performed. Although parent to the human and its behavior predictable. Such ADE did include facilities to allow the user to restart in- mental models might motivate subjects to help the robot dividual components without bringing the entire system (e.g., if they think it is stressed and maybe overwhelmed) down, the lack of reliable wireless access in the competi- or to try harder at achieving a task. Moreover, they might tion area made it impossible to connect remotely, necessi- 4 become more aware of the way they interact with the robot tating time-consuming system restarts. and automatically adapt their interaction patterns so as to During the open interaction, the vision system was facilitate interactions and performance, as suggested by challenged by several factors. While the vision sys- our results. tem worked well if run alone, running other concurrent compute-intensive processes negatively impacted perfor- mance. Lighting conditions were also problematic. One 3.2 The Robot at AAAI in 2005 aim of the employed swarm mechanisms as part of the vision system was to adapt gracefully to changing light, The experiments above demonstrate the robot’s compe- however, the swarm system was in its early stages at the tence under controlled laboratory conditions, in particular time of the competition. As such, feature extraction was with respect to category (1). However, this competence less reliable than anticipated, leading to sometimes spo- does not necessarily translate directly to competence in radic performance by components dependent on this out- the real world, where it is impossible to control all po- put. tentially confounding variables. Hence, it is important to The robot was able to detect legs using the laser, and verify the robot’s performance in unstructured environ- used them as a first pass in detecting humans during ments, such as the AAAI 2005 robot competition. Our the open interaction; however, it was often the case that entry (ND-Rudy) was prepared for two competition cat- other “leg-like” structures (e.g., the ruffles of a tablecloth) egories: the Open Interaction Event and the Robot Exhi- would be identified as legs. Moreover, the leg detection bition. The Open Interaction configuration would cause did not distinguish the fronts of legs from the backs of the robot to approach people and attempt to initiate sim- legs, leading the robot to “initiate a conversation” with a ple template-based conversations (e.g., “Hello, my name human’s back at times. Overall, we found that the people is Rudy. Would you like to chat?”). A limit was placed on detection system placed too much weight on the laser evi- the length of a conversation, although the robot could ter- dence relative to face detection in concluding that a person minate the conversation early if it noticed the human had was present and facing the robot. not responded in a while. The difficulties experienced in natural language pro- Several demonstrations of the robot’s abilities were pre- cessing were attributable mostly to the ambient noise level pared for the Robot Exhibition. To demonstrate visual in the exhibition hall. Although Sonic performed reason- short-term memory, the robot was asked to recall (with- ably well in the controlled environment of the lab, it was out looking) how many faces were immediately surround- probably not optimally configured. Given the noise of the ing it. The ability to identify objects visually was demon- crowd at the competition, therefore, speech recognition strated, including the ability to learn the names of pre- was very limited. Attempts to improve performance by viously unknown object types and subsequently identify them correctly. 4Automatic failure recovery procedures have since been added to the Each of these capabilities was implemented and tested infrastructure to address this problem; see also Section 4.
8 reducing the size of the dictionary had only limited effect designer to specify failure recovery mechanisms for a va- and were insufficient to allow Sonic to reliably recognize riety of contingencies, ranging from the very generic to speech. Because all interactions were designed to be con- the specific. The ADE framework provides mechanisms ducted via spoken natural language, this was clearly the to allow multiple disparate components to be combined most significant limitation of the robot. In the open inter- into a functional robotic system, and its failure recovery action, the robot often seemed confused, responding inap- capabilities, although somewhat lacking at the time of the propriately to what was said and sometimes prematurely competition, have been greatly improved, thereby signifi- ending a conversation because it mistakenly took the lack cantly increasing the system’s robustness. of intelligible output from Sonic as silence (and disinter- The problems experienced when moving from the con- est) on the part of the person. trolled environment of the laboratory to the real-world environment of the robot competition were primarily in Exhibition Demonstrations. Although some demon- three areas: infrastructure, vision, and language process- strations completed successfully during the exhibition, ing. The main limitation of the infrastructure was the lack there was insufficient time to perform others because of of automatic failure recovery. This has been addressed in delays due to software failures. Moreover, the crowd the meantime via the inclusion of failure detection mech- noise was often so high that judges were unable to hear anisms that allow ADE to notice when a component has the robot’s speech. failed, a resource specification scheme by which it can The ADE framework proved highly successful for dis- determine whether there is a target machine matching the tributing components across hosts; at the competition, resource needs of the failed component, and a recovery various components were relocated to different comput- mechanism that allows ADE to restart the failed compo- ers in an effort to optimize overall performance. How- nent. The system has been expanded to include mecha- ever, the uncontrolled environment exposed limitations in nisms for reasoning about the state of the system, allowing the visual object recognition system. The SIFT mecha- ADE to make intelligent decisions concerning the place- nism is susceptible to “pollution” by keypoints that are ment of system components (Kramer & Scheutz, 2006a). actually part of the background instead of the object be- The vision subsystem lacked the robustness required ing identified. It proved virtually impossible to avoid this for effective operation in uncontrolled environments, pollution in real world scenarios, resulting in a number of making feature detection unreliable. The system’s perfor- 5 misidentifications. mance has since been enhanced substantially by extending Given the language understanding subsystem’s inabil- the swarm systems to allow for hierarchical swarms that ity to cope with the noise, it is unsurprising that the robot can track features at different levels of granularity and in had trouble in many cases responding with correct behav- different parameter spaces. ior. Moreover, the ambient noise level also impacted the Language processing was by far the biggest problem other side of language processing—speech production— encountered at the competition, because it comprises the often making it impossible to understand the speech out- main interface between the robot and the humans in its put due to underpowered speakers. At one point during environment. Sonic has been replaced by Sphinx 4 as the exhibition judging we were forced to fall back to us- the main speech recognition platform used by the sys- ing the keyboard of an attached laptop for (typed) natural tem, improving recognition rates somewhat. Moreover, language input and its LCD for (printed) natural language new techniques for dynamically narrowing the dictionary output. to the target task when the domain is sufficiently limited have also improved performance. However, these mecha- nisms are not practical for tasks like the open interaction, 4 Discussion and Related Work where it can be difficult to predict the subject of conversa- tion. We are exploring different hardware configurations In the laboratory environment, at least, DIARC addresses to improve performance (e.g., a noise cancelling micro- the first and the third requirements set forth in Section 2, phone array and a wireless microphone headset). How- although the performance of many components could be ever, speech recognition remains a major bottleneck for improved. The robot is – to some extent – able to interact overall system performance. via natural language, detect emotions, and produce non- Other competition participants are also pursuing goals verbal cues, such as shifting gaze to indicate focus of at- that relate directly to the requirements above. The tention. Moreover, the action interpreter allows the script UML Robotics Lab, in investigating HRI for teleoper- ated robotics, have developed a robust infrastructure us- 5 This problem has since been addressed by applying a stereo vision ing sliding scale autonomy by which the robot can ac- algorithm to isolate an object in the foreground. Preprocessing the stereo image increases the likelihood that any of the determined keypoints ac- cede some control to the remote user when it does not tually belong to the object held to the camera. know how to proceed; the robot continues to operate in
9 some reduced capacity instead of stopping and waiting which uses a dual representation of positive and negative for human intervention (Desai & Yanco, 2005). Hansen affect that is influenced by various components in the ar- Robotics is developing a realistic animated face that is chitecture and used for the calculation of the importance, capable of generating subtle visual cues (Hanson et al., and consequently the priority, of goals.6 2005). The LABORIUS project confronts a substantial Despite significant advances in several areas of HRI, portion of the requirements in various subprojects. For none of the systems that competed in 2005 (including natural language processing, they have developed a sys- ours) has demonstrated the second requirement for natural tem for separating voices in order to understand multiple interaction with humans in real-world environments: the sentences simultaneously (Yamamoto et al., 2005). Their ability to demonstrate and recognize intent. It is in this architecture for socially interactive robots includes moti- general area that human-robot interaction is most lack- vational (although not explicitly affective) states that in- ing. As humans, we take for granted many of the subtle fluence behavior (Michaud et al., 2005). And their Tito (even subliminal) cues present in any human-human in- project implements nonverbal communication through the teraction. The ability to integrate non-verbal information use of arm gestures, head shaking, and affect expression with explicitly communicated information (such as sar- via smiling and raising eyebrows, and has the ability to casm or impatience) comes very naturally for humans; it recognize nonverbal cues, such as gaze detection, in hu- can, in fact, be very difficult to specify what led one to mans (Michaud, Duquette, & Nadeau, 2003). a particular conclusion regarding another person’s intent There are also other groups not represented at AAAI when it was not explicit in the spoken word. Similarly, 2005 that are working on HRI projects similar to ours, humans instantly and often subconsciously make infer- using affect mechanisms to improve interactions. Here ences of others’ private mental states based on external we can only review the two closest architectures in terms clues. When one member of a group shares information, of using emotions for internal state changes and action we infer immediately that all group members now have selection. Murphy et al. (2002) implement emotional that new knowledge. Also, we are often able to make use- states with fixed associated action tendencies in a ser- ful inferences based on what someone does not say (e.g., vice robot as a function of two time parameters (“time- when the situation clearly requires some comment, but to-refill” and “time-to-empty”) plus two constants. Ef- none is offered). These are only a few simple examples fectively, emotion labels are associated with different in- of an array of “rules of thumb” that humans use and ex- tervals and cause state transitions in a Moore machine, pect their interlocutors to use to infer intent in the course which produces behaviors directly based on perceptions of normal conversations. Without a systematic approach and emotional states. This is similar to the way urgency that exposes and incorporates this type of implicit knowl- is calculated in our action manager, but different from edge, robots will continue to miss critical, if not constitu- the explicit goal representation used in our architecture, tive, parts of human social interactions; hence, the second which allows for the explicit computation of the impor- requirement will remain a road block to NHL-HRI, even tance of a goal to the robot (based on positive and negative if all the other problems are solved. affective state), which in turn influences action selection We believe that affect will play a pivotal role in remov- (e.g., urgency alone may or may not result in reprioritiza- ing this road block. The intentionality requirement is the tion of goals and thus changes in affective state). More- furthest from fulfillment in part because of the tremen- over, the robots in Murphy et al., 2002 do not use (spoken) dous complexity of goal-driven cognitive architectures. natural language to interact with humans nor do they de- However, as we argue above, one function of affect is to tect human affect. integrate and manage many diverse cognitive processes, The architecture in Breazeal, Hoffman, and Lockerd eliminating the need for complex centralized control. The (2004) extends prior work (Breazeal, 2002) to include nat- result is an architecture in which multiple states (both in- ural language processing and some higher level deliber- ternal and external) influence the prioritization and selec- ative functions, most importantly, an implementation of tion of goals, leading to predictable intentional behavior “joint intention theory” that allows the robot to respond to that humans can use to develop a “theory of mind” for human commands with gestures indicating a new focus of the robot. DIARC is novel in this regard; affective states attention, etc. The system is intended to study collabora- reflective of past experiences influence the operation of tion and learning of joint tasks. One difference is that our the goal-based control system. The system responds to robot lacks the ability to produce gestures beyond simple events in a more “human-like” manner than non-affective nodding and shaking by the pan-tilt unit (although it is systems, allowing humans to “relate” better and making mobile and fully autonomous as opposed to the robot in them more likely to ascribe the property of intentional- Breazeal et al., 2004). More importantly, the mechanisms ity to the robot. As such, DIARC is the among the most for selecting subgoals, subscripts, and updating priorities 6The details for reprioritization of goals were not provided in of goals seem different in our affective action interpreter, Breazeal et al. (2004).
10 advanced architectures available for NHL-HRI. virtual and robotic agents. In P. Petta & J. Muller¨ (Eds.), Best of at2ai-4 (Vol. 20). Bless, H., Schwarz, N., & Wieland, R. (1996). Mood 5 Conclusion and the impact of category membership and indi- viduating information. European Journal of Social In this paper, we have proposed three main cate- Psychology, 26, 935-959. gories of requirements for natural human-like human- Breazeal, C., Hoffman, G., & Lockerd, A. (2004). Teach- robot interaction: (1) social interaction abilities, such ing and working with robots as a collaboration. In as natural-language production and understanding, situ- Proceedings of aamas 2004. ational knowledge, and expression and recognition of af- Breazeal, C., & Scassellati, B. (1999). How to build fect and other non-verbal cues; (2) goal-oriented cogni- robots that make friends and influence people. In tion, which requires the robot to act in a purpose-driven Iros (pp. 858–863). manner, allowing humans to predict the robot’s behav- Breazeal, C. L. (2002). Designing sociable robots. MIT iors based on ascribed beliefs, intentions, and desires; and Press. (3) robust intelligence, the ability to recover from failures Burkhart, F. (2005). Emofilt: the simulation of emotional within the system as well as failures of the system itself. speech by prosody-transformation. In Proceedings The DIARC architecture introduced in the AAAI 2005 of interspeech 2005. robot competition is a first attempt at meeting some of Byers, Z., Dixon, M., Goodier, K., Grimm, C. M., & these requirements. While DIARC– and every other cur- Smart, W. D. (2003). An autonomous robot pho- rent robotic system – fails at achieving, even in part, the tographer. In Proceedings of iros 2003. Las Vegas, second requirement for NHL-HRI (intentionality) in nat- NV. ural environments, it does make substantial progress to- Clore, G., Gasper, K., & Conway, H. (2001). Affect as ward integrated social behaviors and fault tolerant cogni- information. In J. Forgas (Ed.), Handbook of affect tion; it is able to recognize and express affect at a coarse- and social cognition (p. 121-144). Mahwah, NJ: grained level, in addition to its natural language under- Erlbaum. standing and production capabilities, and ADE’s auto- The CMU Sphinx group open source mated failure recovery mechanisms greatly enhance the speech recognition engines. (2004). autonomy of the robotic system. Although the system http://cmusphinx.sourceforge.net/html/cmusphinx.php. proved too fragile to demonstrate these abilities in the real-world environment of the 2005 AAAI Robot Com- Desai, M., & Yanco, H. A. (2005, August). Blending petition, its performance in experiments conducted with human and robot inputs for sliding scale autonomy. human subjects in a controlled laboratory setting is very In Proceedings of the 14th ieee international work- promising for categories (1) and (3). Moreover, the degree shop on robot and human interactive communica- to which affect is integrated into the architecture makes tion. Nashville, TN. DIARC also a promising platform towards meeting the Ekman, P. (1993, April). Facial expression and emotion. intentionality requirement (2) for NHL-HRI as the results American Psychologist, 48(4), 384–392. from human subject experiments demonstrate. Ekman, P., & Friesen, W. V. (1977). Manual for the facial action coding system (facs). Palo Alto: Consulting Psychologists Press. Acknowledgment The festival speech synthesis system. (2004). http://www.cstr.ed.ac.uk/projects/festival/. Centre The authors would like to thank Virgil Andronache, Chris for Speech Technology Research. Middendorff, Aaron Dingler, Peter Bui and Patrick Davis Fillmore, C. J., Baker, C. F., & Sato, H. (2002). The for their help with the implementation. framenet database and software tools. In Proceed- ings of the third international conference on lan- guage resources and evaluation (lrec) (pp. 1157– References 1160). Las Palmas, Spain. Grosz, B. L., & Sidner, C. L. (1990). Plans for discourse. Andronache, V., & Scheutz, M. (2004). Integrating the- In P. R. Cohen, J. Morgan, & M. E. Pollack (Eds.), ory and practice: The agent architecture framework Intentions in communication (pp. 417–444). MA: APOC and its development environment ADE. In MIT Press. Proceedings of aamas 2004. Hanson, D., Olney, A., Prilliman, S., Mathews, E., Zielke, Andronache, V., & Scheutz, M. (2006). ADE - a tool M., Hammons, D., Fernandez, R., & Stephanou, H. for the development of distributed architectures for (2005, July). Upending the uncanny valley. In Pro-
11 ceedings of the aaai-05 robot workshop. Pittsburgh, Pellom, B., & Hacioglu, K. (2003). Recent improvements PA. in the CU SONIC ASR system for noisy speech: Ichise, R., Shapiro, D., & Langley, P. (2002). Learning The SPINE task. In Proceedings of IEEE interna- hierarchical skills from observation. In Proceed- tional conference on acoustics, speech, and signal ings of the fifth international conference on discov- processing (ICASSP). ery science (pp. 247–258). Picard, R. (1997). Affective computing. Cambridge, Mass, Kanda, T., Iwase, K., Shiomi, M., & Ishiguro, H. (2005). London, England: MIT Press. A tension-moderating mechanism for promoting Rani, P., Sarkar, N., , & Smith, C. A. (2003). Affect- speech-based human-robot interaction. In Iros (p. sensitive human-robot cooperation-theory and ex- 527-532). periments. In Proceedings of ieee international Kipper, K., Dang, H., & Palmer, M. (2005, July). Class- conference on robotics and automation (icra) (pp. based construction of a verb lexicon. In Proceed- 2382–2387). ings of AAAI 2000. Schank, R., & Abelson, R. R. (1977). Scripts, plans, Kramer, J., & Scheutz, M. (2006a). Reflection and rea- goals, and understanding. Hillsdale, NJ: Lawrence soning for system integrity in an agent architecture Erlbaum Associates. infrastructure. (Under review) Scheutz, M. (2000). Surviving in a hostile multiagent Kramer, J., & Scheutz, M. (2006b). ADE: A framework environment: How simple affective states can aid for robust complex robotic architectures. (Under in the competition for resources. In H. J. Hamilton review) (Ed.), Advances in artificial intelligence, 13th bien- Lisetti, C. L., Brown, S., Alvarez, K., , & Marpaung, A. nial conference of the canadian society for compu- (2004). A social informatics approach to human- tational studies of intelligence, ai 2000, montreal,´ robot interaction with an office service robot. IEEE quebec, canada, may 14-17, 2000, proceedings Transactions on Systems, Man, and Cybernetics– (Vol. 1822, pp. 389–399). Springer. Special Issue on Human Robot Interaction, 34(2), Scheutz, M. (2004). Useful roles of emotions in arti- 195–209. ficial agents: A case study from artificial life. In Lowe, D. G. (2004). Distinctive image features from Proceedings of aaai 2004. scale-invariant keypoints. International Journal of Scheutz, M. (2006). ADE - steps towards a distributed Computer Vision, 60(2), 91-110. development and runtime environment for complex Michaud, F., Brosseau, Y., Cotˆ e,´ C., Letourneau,´ D., robotic agent architectures. Applied Artificial Intel- Moisan, P., Ponchon, A., Ra ievsky, C., Valin, J., ligence, 20(4-5). Beaudry, E.,´ & Kabanza, F. (2005). Modularity Scheutz, M., & Andronache, V. (2003). APOC - a frame- and integration in the design of a socially interactive work for complex agents. In Proceedings of the robot. In Proceedings IEEE international work- aaai spring symposium. AAAI Press. shop on robot and human interactive communica- Scheutz, M., & Andronache, V. (2004). Architectural tion (pp. 172–177). mechanisms for dynamic changes of behavior se- Michaud, F., Duquette, A., & Nadeau, I. (2003). Char- lection strategies in behavior-based systems. IEEE acteristics of mobile robotic toys for children with Transactions of System, Man, and Cybernetics Part pervasive developmental disorders. In Proceedings B, 34(6), 2377-2395. ieee conference on systems, man, and cybernetics. Scheutz, M., & Logan, B. (2001). Affective versus delib- Middendorff, C., & Scheutz, M. (2006, June). Real- erative agent control. In S. Colton (Ed.), Proceed- time evolving swarms for rapid pattern detection ings of the aisb’01 symposium on emotion, cogni- and tracking. In Proceedings of artificial life x. tion and affective computing (pp. 1–10). York: So- Mueller, E. T. (1998). Natural language processing with ciety for the Study of Artificial Intelligence and the thoughttreasure. New York: Signiform. Simulation of Behaviour. Murphy, R. R., Lisetti, C., Tardif, R., Irish, L., & Gage, Scheutz, M., McRaven, J., & Cserey, G. (2004). Fast, re- A. (2002). Emotion-based control of cooperating liable, adaptive, bimodal people tracking for indoor heterogeneous mobile robots. IEEE Transactions environments. In IEEE/RSJ international confer- on Robotics and Automation, 18(5), 744-757. ence on intelligent robots and systems (IROS). Ortony, A., Norman, D., & Revelle, W. (2005). Effec- Scheutz, M., Schermerhorn, P., Kramer, J., & Midden- tive functioning: A three level model of affect, mo- dorff, C. (2006). The utility of affect expression in tivation, cognition, and behavior. In J. Fellous & natural language interactions in joint human-robot M. Arbib (Eds.), Who needs emotions? the brain tasks. In Proceedings of the ACM conference on meets the machine. New York: Oxford University human-robot interaction (HRI2006). Press. Sleator, D., & Temperley, D. (1993). Parsing english
12 with a link grammar. In Proceedings of the third international workshop on parsing technologies. Yamamoto, S., Nakadai, K., Valin, J., Rouat, J., Michaud, F., Komatani, K., Ogata, T., & Okuno, H. (2005). Making a robot recognize three simultaneous sen- tences in real-time. In Proceedings IEEE/RSJ inter- national conference on intelligent robots and sys- tems (pp. 897–902). Yang, M.-H., Kriegman, D. J., & Ahuja, N. (2002, Jan- uary). Detecting faces in images: A survey. IEEE Transactions on Pattern Analysis and Machine In- telligence, 24(1), 34–58.
13