Experiences in Deploying Test Arenas for Autonomous Mobile Robots

Experiences in Deploying Test Arenas for Autonomous Mobile Robots Adam Jacoff, Elena Messina, John Evans Intelligent Systems Division National Institute of Standards and Technology Gaithersburg, MD 20899 ABSTRACT and henceforth, the RoboCup Federation [3] will use the arenas to host their newly formed The National Institute of Standards and Technology RoboCupRescue league competitions. A has created a set of reference test arenas for evaluating discussion of the details of these competitions is the performance of mobile autonomous robots contained in Section 3 of this paper. performing urban search and rescue tasks. The arenas are intended to help accelerate the robotic research There are three sets of customers for the arenas. community’s advancement of mobile robot The first are researchers, who need testing capabilities. The arenas have been deployed in two opportunities. The repeatable obstacles (sensory competitions thus far and are also being used by researchers to test their systems’ capabilities. We and physical) that are focussed towards mobile describe the arenas, their use in competitions and our robotic perception and intelligent behavior near-term and long-term plans for the arenas. provide them with challenges for their robots. The second are the sponsors of research. They can use the arenas for validation exercises to 1. INTRODUCTION objectively evaluate robots in structured, repeatable, representative environments. The The National Institute of Standards and arenas can be used to validate robotic purchases, Technology (NIST) has been collaborating with identify strengths and weaknesses in systems, other government agencies and university and compare the cost effectiveness of different researchers to develop methods of evaluating and approaches. Finally, the end users of the robots measuring the performance of robotic and other can benefit from the resulting performance intelligent systems. The community agrees that it metrics. The eventual goal is to develop standard would benefit from having uniform, reproducible performance metrics from the arenas that can be means of measuring capabilities of their systems used by purchasers to evaluate mobile robot to evaluate which approaches are superior under capabilities. which circumstances, and to help communicate results. One of the efforts in the performance There were several motivating factors for metrics program at NIST is the creation of building the arenas. The first was the desire to be reference test arenas for autonomous mobile able to compare “apples to apples” in a robots. The first set of arenas was modeled after technological sense. When researchers publish the Urban Search and Rescue (USAR) results, they typically describe the performance application and was designed to represent, at of their systems in their laboratory or varying degrees of verisimilitude, challenges demonstration environments, making it difficult associated with collapsed structures. This is a to compare and contrast with others researchers’ domain that is very dangerous for rescue results. Isolating tests for sensing, behaviors, and personnel and in which robots will likely be able other robotic capabilities – and making these to provide increasing levels of assistance in tests reproducible – allows the research searching for survivors. [1] The arenas were first community to make meaningful comparisons of deployed at the American Association for algorithms, sensors, platforms, and other Artificial Intelligence (AAAI) Rescue Robot independent items. A standardization of these Competition in 2000. In 2001, the arenas were challenges, through use of the arenas, enables a used at the International Joint Conference on direct comparison of approaches. Artificial Intelligence (IJCAI). They will again be used at AAAI-2002. Additionally for 2002 A second desire was being able to “teach to the in Fig. 1. For a more in-depth discussion of the test.” The arenas provide an objective set of design considerations for the arenas, see [2]. measures for evaluating different robotic At the lowest level is the locomotion capability implementations. The arenas are not idealized of the robot’s physical platform. Although two “blocks world” tests. They provide some fairly of the three arenas provide some challenges for realistic challenges that mobile robots must be locomotion and require general agility of the able to address to be considered capable in this robots, our emphasis (and that of the AAAI domain. We hasten to add that the USAR competitions) is on algorithms. So the arenas domain is extremely challenging. Although the attempt to isolate and test the higher elements of arenas do provide some elements of what may be robot autonomy and do not address locomotion encountered in a collapsed building, they are not directly. representative of the reality of a disaster scene. Rather, they provide a step-wise abstraction of The element just above the hardware such challenges in an attempt to isolate and implementation of locomotion and sensors is repeatably test specific robot capabilities. sensory perception. The robot has to sense what is in its environment in order to navigate, detect Another concern of research sponsors and of hazards, and identify goals (simulated victims researchers themselves is the slowing of progress and their locations). Sensor fusion is an due to re-invention of the wheel. When building important capability, as no single sensor will be a robot, numerous hardware and software able to identify or classify all aspects of the subsystems are required and it is not possible (or arenas. The simulated victims in the arenas are very difficult) to reuse any work done by other represented by a collection of different sensory organizations. By highlighting successful signatures. They have shape and color approaches negotiating well-known obstacles, it characteristics that look like human figures and is hoped that others will better understand and clothing. They have heat signatures representing adopt these approaches, and expedite their body heat, along with motion and sound. The progress into other areas of research. Finally, practice makes perfect: arenas that are COLLABORATION available to researchers year-round should enable AUTONOMY them to repeat experiments and therefore debug and improve their systems. The arenas are set up PLANNING near the NIST campus in Gaithersburg, KNOWLEDGE Maryland, and can be used by researchers year- round. Since robustness comes through repetition PERCEPTION and testing outside perceived limits, the three arenas provide increasing levels of difficulty, so SENSORS that researchers can move on to new challenges once they master the simpler sections. LOCOMOTION 2. DESIGN CONSIDERATIONS Figure 1: Constituent Elements of an Autonomous Mobile Robot 2.1. Elements of Robotic Capabilities arenas are also designed to pose challenges to The primary goal of the test arenas is to provide typical robot navigation sensors. For example, reproducible measurements and tests of acoustic-absorbing materials confuse sonar autonomous mobile robots. There are several sensors. Laser sensors have difficulty with elements that come together to create a fully shallow angles of incidence, smooth surfaces, autonomous mobile robot. Recognizing that and reflective materials. Highly regular striped there are going to be different levels of wallpaper and other types of materials pose autonomy implemented in mobile robots, the challenges to stereo vision algorithms. arenas are designed to isolate the different Compliant objects that may visually look like capabilities that may be available on any rigid obstacles require the robots to apply tactile particular robot. They are shown schematically sensors or other means of verifying that they can Figure 2: Model of the Reference Test Arenas for Autonomous Mobile Robots indeed push them aside (e.g., open doors or of victims and hazards. Ideally, they would curtains). Manipulation of rigid obstacles, such provide humans a map of the environment they as closed doors or debris, provide more advanced have explored, with the victims’ and hazards’ challenges. Robot localization is another locations marked. The environment that the essential capability derived from sensing. robots operate in is three-dimensional, hence Different flooring materials affect localization they should reason, and be able to map, in three schemes based on wheel encoders. Additional dimensions. The arenas may change dynamically cues from the environment need to be employed during a competition (as a building might further to help localize the robot in an effort to generate collapse while rescuers are searching for and maintain correct maps. Since the arenas victims). Therefore the ability to create and use represent collapsed structures and buildings, maps to find alternate routes is important. GPS is not considered to be available. The planning or behavior generation components Knowledge representation is the next element. It of the robots build on the knowledge encompasses the robot’s ability to model the representation and the sensing components. The world, using both a priori information (such as robots must be able to navigate around obstacles, might be needed to recognize certain objects in make progress in their mission (that is to explore an environment) and newly acquired information as much as possible of the arenas and find (obtained through sensing the environment as it simulated victims), take into account time as a explores). In the mobile robot competitions

Load more