II-NEW: Multi-Robot Lab for Quality of Life Applications 1 Introduction

II-NEW: Multi-Robot Lab For Quality of Life Applications

1 Introduction

NSF funds will support the purchase of ?? small humanoid robots to establish a multi-robot research and education facility. Our vision of the future is one in which humans interact with a large number of devices and robots in a human-robot community. In order to develop the research and educational foundation for such a future, we propose establishing a multi-robot research and education facility at CMU. We will focus on research and education about how robots can help everyone, with additional emphasis on helping children with developmental disorders and older adults. To work towards a future of human-robot community, we need to develop techniques for and educational materials on multi-robot perception, control, navigation, manipulation, reasoning, learning, and human-robot interaction. We need to explore how multiple robots can help each other perform better, and ex- change information effectively to achieve useful tasks. We have extensively explored how individual robots operate in human environments and have investigated and developed multi-robot systems in customized environments. We are now ready to experiment with multiple robots in our daily human environments. Fac- ulty who have expressed an interest in such a research facility include faculty from the Robotics Institute (Atkeson, Kanade, Likhachev, Rybski, Sheikh, Siegel, Simmons, Steinfeld, Thorpe), Computer Science De- partment (Touretzky, Veloso), Human-Computer Interaction Institute (Forlizzi, Cassell), Machine Learning Department (Gordon), Language Technologies Institute (Rudnicky), and Mechanical Engineering (Mess- ner). Choset will participate in his role as Director of the Robotics Undergraduate Minor. We will also integrate this multi-robot research and education facility into the research and education programs of our NSF Engineering Research Center on Quality of Life Technologies (QoLT) which is a collaboration with the University of Pittsburgh. Example research projects include studies on how the sensors on all robots can be most effectively combined to monitor a large space, how to allocate resources to handle errors and emergencies (a human or robot accident such as a spill), and how such a system can be used to present a consistent interface to humans traveling through the space. We also contemplate experiments in which the robots are installed in other facilities such as homes, student dorms, and nursing homes or other care facilities, where we would do studies as to how a team of robots could make residents happier, monitor residents, and reduce the burden on caregivers. We expect the underlying foundational issues of multi-robot perception, control, and interaction to be very similar at our multiple study sites. We also expect the multi-robot facility to play a major role in education. We have several courses that already use small numbers of simple robots to work towards these goals (including 15-491: CMRoboBits: Creating Intelligent Robots, 16-264: Humanoids, and 16-362: Mobile Robot Programming Laboratory). With the availability of this multi-robot facility, we will see inclusion of hands on robotics in courses on AI, machine learning, HCI, speech (LTI), optimization, mechanical engineering, and programming. We also expect extensive involvement in Andrew’s Leap, the SAMS program, and other K-12 and minority outreach programs. We believe two possible applications will be wildly motivating for undergraduates. The first is team sports (Figure 1). We find that the element of competition is highly motivating for students. We also find that interactive games and theater is a rich realm for developing and exploring human-robot interaction. With a set of robots, video games can become physical games. One proposed educational project is a robot play or musical. We have found that assistive and story-based projects are especially attractive and engaging for female students. We are focusing on humanoid robots for two reasons. The first is that there are complete humanoid robots now available in the market, which are aiming to serve both research and education, and for which we are able to negotiate a significantly reduced price. The second reason is that the humanoid robot artifact

1 Figure 1: Left: Prof. Manuela Veloso currently uses NAO robots in her studies of teams of intelligent agents in complex, dynamic, and uncertain environments, in particular adversarial environments. We will build on this experience base. Right: Schematic of NAO robot. offers compelling research and educational features, including a complex and motivating articulated body that allows for a rich walking motion, body poses, and gestures, as well as on board sensing, cognitive, and actuation capabilities that permit sophisticated interaction with humans. We believe others will be able to replicate this multi-robot laboratory as the cost of small robots decreases and the entertainment robotics market matures. The proposed facility is a new facility, and we estimate its lifetime as at least 10 years. The compelling new research and education opportunities are described below, and include use of multiple state of the art humanoid robots in classes that teach kinematics, dynamics, control, machine perception, artiﬁcial intelligence and reasoning, machine learning, systems integration, and human-computer interaction, and research in how communities of humans and robots can work together so that robots assist humans in their daily tasks, robots help older adults, robots help children with developmental disorders, and working with robots helps us understand how the brain works. Researchers, educators, and students from CMU and from the University of Pittsburgh will be able to use our facility and our robots. Most of our faculty teach and do research in synergistic areas, so we expect a great deal of cross fertilization between teaching and research, and between different areas. We expect this facility and its resources to be a major magnet in drawing students into research. CMU and the investigators involved have committed to supporting any infrastructure modiﬁcation necessary to support robot use across several buildings such as indoor navigation aids and automatic door and elevator controls accessible from the computer network. We will operate and maintain the facility using CMU contributions such as CMU Robotics Institute funding of our Robotics Education Lab, and support from the research grants of the users (as we do now for our shared research facilities such as high performance computer clusters).

2 The Multi-Robot Research and Education Facility

With the NSF funding, we will create a multi-robot research and education facility at CMU. It will include ?? small humanoid robots. The robots will be housed in our 800 square foot Robot Education Lab (Newell Simon Hall xxx). Faculty and students will be able to use them there, as well as check them out. Manuela Veloso’s and Chris Atkeson’s laboratories will provide additional support.

2 Figure 2: Some challenging dynamic tasks we programmed on a Sarcos humanoid robot.

We will also instrument a large indoor space spanning across three connected buildings (the Gates- Hillman Complex, Newell Simon Hall, and Wean Hall) to support robot use. These buildings are connected by robot-navigable bridges and have been used extensively in the past for individual robot studies. Robots would be able to work in the Wean and NSH 4th floor hallways and GHC floors 3-5 without opening doors and using stairs, and with assistance or additional technology will be able to pass through doors and ride elevators to go further. This space includes classrooms, offices, laboratories, hallways, restrooms, lounges, kitchens, and larger scale commercial eating facilities. It is our expectation that much use of the robots will occur in these buildings. We will also support use of the robots in the student’s homes or dorm rooms, as well as, use in nursing homes or other care facilities as part of the research and educational activities of our NSF ERC on QoLT, We have selected NAO H25 humanoids manufactured by the Aldebaran company (www.aldebaran- robotics.com) as the primary robot in our facility. We have negotiated a $7000 price for the full humanoid. The NAO robot is one of the most widely-used humanoid robot for academic purposes worldwide. It is also widely used in Secondary Education STEM activities. The robot is a 58 cm tall humanoid robot with a body with 25 degrees of freedom (DOF), each actuated by electric gearmotors with Hall effect sensors (36) and dsPICS microcontrollers. The neck has 2 DOF, each arm has 4, the pelvis has 1, each leg has 5, and each hand has 2. There are many sensors: 2 CMOS video cameras, 4 microphones, a sonar distance sensor with two channels, 2 IR emitters and receivers, 2 single axis gyros, one 3 axis accelerometer, 9 contact sensors, and 8 FSR pressure sensors to measure force. There are various output devices, including 2 high quality speakers and a voice synthesizer, and 33 LED lights. Power is provided by a 55 Watt-Hours battery, giving the robot 1.5 hours without recharging or more depending on the usage. The robot weights approximately 5 kg. The robot is fully programmable with a x86 AMD GEODE 500MHz CPU with 256MB SDRAM and 2GB Flash Memory. The CPU (located in the head) runs a Linux kernel and supports ALDEBARANs own proprietary middleware (NAOqi). There is a second CPU (located in the torso). There is both wired and wireless ethernet, as well as infrared local communication. The operating system is a version of Embedded Linux (32bit x86 ELF) using a custom OpenEmbedded based distribution. It supports many programming languages including C++, Urbi script, Python, and .Net. The NAO software provides good support for beginning users, including visual editing of robot programs as graphical objects. The company provides software including general behavior programming, simulation, walking, falling and whole body motion control, vision software than can track and recognize faces and objects, audio software to localize and track sound sources, recognize sound, and recognize speech, interfaces to tactile sensing, global communication using ethernet and Wifi, local communication using infrared signals, middleware, and OS functions. Much of this software is open source and students and researchers can replace it with their own software as needed. We have experience with the NAO robots, as Manuela Veloso’s lab currently uses ??? of these robots for research in robotics including participation in robot soccer. MANUELA: More stuff on current use of NAO robots goes here.

3 3 Humanoid Robots And Teaching

MANUELA: talk about your course. Chris Atkeson has taught an undergraduate course on Humanoids several times. A focus on humanoids allows the class to survey many aspects of robotics, computer science, and engineering. The course has used donated inexpensive toy robots (the Wowwee Robosapien RS Media) for its educational laboratory. Enough robots (40) were donated so that every student could take one home and work with it in their dorm rooms. This was tremendously successful and motivating for the students. Many of them published Youtube videos of the robots in action (search for 16-264 on Youtube). Unfortunately, the Wowwee company no longer supports these robots, and the onboard Java machine no longer works with current versions of Java. Furthermore, the robots have much more limited mechanical, sensing, and computational capability when compared to the NAO robots. It would be a tremendous improvement to move to the NAO robot lab. The course focuses on freshmen and sophomores, with the goal of exposing students to a wide range of technology associated with intelligent machines. It surveys perception, cognition, and movement in humans, humanoid robots, and humanoid graphical characters. Topics covered include kinematics, dynamics, control, mechanical design, artiﬁcial intelligence (planning, reasoning, learning, decision making), vision, hearing, tactile sensing, robots and AI in literature, and philosophical and ethical issues. We have a lot of fun with this, and try hard to get students involved in research projects after the semester is over. The educational lab makes a huge difference in getting students involved and excited about the course material. Enabling each student to work with their “own” robot for a semester in their dorm gives them a sense of pride and ownership of the material. It also exposes this material to a much wider range of dorm mates and fellow students by putting the robot in the dorms. NAO robots are widely used in other universities for educational work on localization, mapping, and navigation, perception, cognition, action, including manipulation and locomotion. Since most control software is open source, students can easily access the lower levels of the system, and replace software components. The visual programming software NAO provides is quite good, and includes a timeline, movement recorder, and curve editor to tune movements. NAO also makes it easy to connect other devices, such as the Kinect, to the robot control system.

4 Multiple Humanoid Robots Helping Everyone

MANUELA: Talk about Cobots. Cobots is a focus of National Robotics Initiative.

5 Multiple Humanoid Robots And Human-Robot Interaction lengthen interaction from minutes to weeks. Yaser Sheikh: I’d prospectively be interested in using your robot community to see if we can make the robots display socially appropriate behavior (look in the right area at the right time by taking cues from what humans are looking at). Of course, I’m interested in the realtime perception and scene modeling part of it, and less equipped to do the realtime planning and control of the robots. Robots are becoming increasingly functional in carrying out everyday tasks. As they begin to enter our social environments, we will expect them to seamlessly collaborate with not only individuals but groups of people, as well. In many scenarios, such as disaster recovery, logistical planning, and support at large events (sports, concerts, etc.), robots will have to co-habit a space with a group of humans and directly interact with them to cooperatively accomplish tasks. To achieve this, robots must ﬁrst understand the social

4 Figure 3: Air Hockey. A humanoid robot playing air hockey with a human. dynamics of humans in a variety situations and environments. The goal of this research is to understand the social dynamics of humans from cameras directly mounted on members of a social group (e.g., workers in a factory, friends at a party, employees in an ofﬁce space, journalists at an event). This will allow co-robots in a social environment to gain an empathic understanding of the dynamics in the group, by being aware of where each member is, what they are cognitively attending to, and what, as a group, they are trying to accomplish. We aim to develop predictive models of social dynamics and discover algorithms to estimate a time-varying social importance map that associates social importance to each location in 3D space. Aaron Steinfeld: Steinfeld is currently exploring how robot behaviors and characteristics impact human trust in robots (IIS-0905148). This work is focused on real-time, task-oriented interactions rather than social interaction. The research is currently utilizing non-humanoid rovers to examine trust during remote navigation. Research in other aspects of human-robot interaction strongly suggest that humanoid robot forms lead to differences in human interpretation of robot performance and acceptance. Therefore, there are opportunities to examine how the humanoid form alters operator trust and use of robots during task-oriented activities.

6 Multiple Humanoid Robots Helping Children With Developmental Disor- ders

Justine Cassell: My current work with humanoid graphical agents suggests that they are effective in helping children with high-functioning autism to learn contingency, reciprocity and other social skills in communication. Recent results from my lab even suggest transfer such that children who learn some social skills with the virtual humans later do better at those social skills with other children. Other work in the ﬁeld suggests that robots can also help children with autism but transfer has not been shown. I intend to compare graphical agents (virtual humans) to humanoid robots in their ability to scaffold the core social communication skills of children with autism.

7 Multiple Humanoid Robots Helping Older Adults

Atkeson writes this. Dan Ding: I am interested in integrating the humanoid robots into smart environments. We’d like to investigate how the smart environment and robots can compensate each other to promote maximum human function, i.e., to what extent a smart environment equipped with embedded sensors, actuators, networked appliances and objects can facilitate the operation and reliability of the robots? We would like to use the robots in the cueing kitchen which can help monitor user actions (which can complement the ﬁxed cameras in the environment) and deliver prompts and cues with different modalities (e.g., voice or any sound, a

5 display attached to it, or projecting the information onto any surface) possible at the most opportune moment and location to assist people with cognitive impairment to complete their daily tasks. I also think the other two projects in HCHW (health kiosk and dwellsense) could use the robots to provide the health monitoring feedback and functional assessment feedback to users, essentially serving as a 24/7 caregiver, which could be more acceptable to the end-users. We could also deploy a robot to the wounded warrior concept homes which can be designed as a mobile reminder for the residents, or an assessment center for stress and anxiety which are common in service members (could be REU student projects).

8 Humanoid Robots and Understanding The Brain

Andy Schwartz: I am interested in looking at embodiment, using humanoid robots. We are especially interested in haptic exploration and tool use. In our brain activity/ robotic research, we have found that the motor areas of the brain respond when anthropomorphic robot arms/hands perform natural-appearing movement. This further validates ﬁnding that there is an robust observation/mimicry network in the primate brain and that this is a primary learning mechanism. Anthropomorphic robots could be “taught” using the same principles and we could derive behavioral principles from how they learn to interact with objects in the environment. We could then use these principles to build better, brain-controlled interfaces for high- performance humanoid effectors, in our effort to build prosthetic systems for paralyzed individuals.

9 Modular Robots

Jodi Forlizzi: I would be interested in developing a general modular robot that we could modify in a number of iterations for the purposes of conducting studies in the lab and in the ﬁeld. I can supply more information if needed.

10 Impact

Impact: Describe impact. What is transformative? One transformative element of our proposed research is ... Education and outreach: Revise this to ﬁt this proposal: We will create an environment for training of undergraduates and graduate students by: 1) involving them in the research, 2) developing and requiring an appropriate curriculum, and 3) disseminating educational and research materials. In addition to his undergraduate teaching of a course on humans and humanoid robots, The CMU Robotics Institute already has an aggressive outreach program at the K-12 level [?], and we will participate in that program. Dan Ding: The robots would also be useful for many of the QoLT outreach activities including the QoLT Ambassador program, TechLink, BodyScott activities, and Creative Tech night for girls etc., for the K-12 students and general public to explore how robots can be programmed to assist people with reduced capabilities. Reid Simmons: Yes, I’d be interested. I am discussing with Anne Mundell and Michael Chemers in Drama about doing outreach to K-12 students using humanoid robots to teach them about the connections between art and technology. The robots would do an interactive presentation with human actors and (per- haps) answer questions from the students. We did a preliminary version of such a presentation recently to drama students and got very good feedback. Diversity: Revise this to ﬁt this proposal: One of the co-investigators is female. We will make sure to include women and minority students. CMU is fortunate in being successful in attracting an unusually

6 Figure 4: Our Sarcos humanoid robot responding to a perturbation while standing (video frames are shown every third of a second) and testing our Sarcos humanoid on lifting tasks. high percentage of female undergraduates in Computer Science. In terms of graduate students, we have had success working with each of our respective graduate admissions programs.

11 Project Management Plan

A detailed project management plan, with timeline, to create and deploy th e new or enhance the existing research infrastructure. Is the project management plan, including timeline, costs, and personnel, realistic? CMU and the investigators involved have committed to supporting any infrastructure modiﬁcation necessary to support robot use across several buildings such as indoor navigation aids and automatic door and elevator controls accessible from the computer network. We will operate and maintain the facility using CMU contributions such as CMU Robotics Institute funding of our Robotics Education Lab, and support from the research grants of the users (as we do now for our shared research facilities such as high performance computer clusters). * Does the proposing institution(s) provide a convincing case of their commi tment to maintain and operate the infrastructure for its useful life?

12 Prior Work

Atkeson was involved in the following Equipment Grant: NSF award number: ANI-0224419, Total award: $1,015,000, Duration: 3/15/03 - 2/28/07, Title: Collaborative Research Resources: An Experimental Plat- form for Humanoid Robotics Research, PI: Hodgins, Atkeson co-PI, 5 other co-PIs. This grant is supporting the development of a humanoid robot, which is used as a testbed for humanoids research. Figure 4 shows the completed robot. This equipment grant directly led to $5,122,897 in research funding, helped us get our

7 $3,666,905 IGERT program, and helped us get our NSF Engineering Research Center in Quality of Life Technology. The equipment grant was a great investment both for us and for the NSF. This robot was used by Atkeson for education and research supported by the following grants. NSF award number: ECS-0325383, Total award: $1,466,667, Duration: 10/1/03 - 9/30/09, Title: ITR: Collaborative Research: Using Humanoids to Understand Humans, PI: Atkeson, Hodgins co-PI, 2 other co-PIs. This grant supported complementary work on programming human-like behaviors in a humanoid robot. NSF award number: DGE-0333420, Total award: $3,666,905, Duration: 10/1/03 - 9/30/09, Title: IGERT: Interdisciplinary Research Training in Assistive Technology, PI: Atkeson. Atkeson directed an IGERT educational program in which several students used the Sarcos Robot. NSF award number: ECCS-0824077, Total award: $348,199, Duration: 9/1/08 - 8/31/12, Title: Ap- proximate Dynamic Programming Using Random Sampling, PI: Atkeson. This grant supports work on better planning approaches. NSF award number: IIS-0964581, Total award: $699,879, Duration: 7/1/10 - 7/1/13, Title: RI: Medium: Collaborative Research: Trajectory Libraries for Locomotion on Rough Terrain, PI: Hodgins, Atkeson co- PI. This grant supports work on behavior library approaches to generating behavior. DARPA M3 Program:Total award: $2,608,152, Duration: 2/1/11 - 1/30/15, Title: Achieving Maximum Mobility and Manipulation Using Human-like Compliant Behavior and Behavior Libraries, PI: Atkeson. This grant supports work on agile behavior. Publications that used the Sarcos robot include [?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?].