Using Robot Competitions to Promote Intellectual Development

AI Magazine Volume 21 Number 1 (2000) (© AAAI) Articles Using Robot Competitions to Promote Intellectual Development

Robin R. Murphy

This article discusses five years of experience using into the classwork of the Introduction to three international mobile robot competitions as Robotics and Computer Vision (junior-senior the foundation for educational projects in under- level), Advanced Robotics and Computer graduate and graduate computer science courses. Vision (graduate level), and Field Session (com- The three competitions—(1) AAAI Mobile Robot, puter science design practicum) courses offered (2) AUVS Unmanned Ground Robotics, and (3) by the Department of Mathematical and Com- IJCAI RoboCup—were used in different years for an introductory undergraduate robotics course, an puter Sciences. The intent of these efforts is to advanced graduate robotics course, and an under- improve students’ design competence and graduate practicum course. Based on these experi- intellectual maturity by taking advantage of ences, a strategy is presented for incorporating the opportunities afforded by robot competi- competitions into courses in such a way as to fos- tions. ter intellectual maturation as well as learn lessons This article reviews experiences associated in organizing courses and fielding teams. The arti- with these courses and their design teams over cle also provides a classification of the major robot a five-year period. The design teams met the competitions and discusses the relative merits of objectives of each course, earned 1 first-place each for educational projects, including the expected course level of computer science stu- and 4 third-place awards plus $5000 in prizes, dents, equipment needed, and costs. and produced 7 papers. Based on these positive experiences, I encourage others to integrate competitions into classes and offer recommen- esign competitions are becoming dations on how to facilitate the process. increasingly common, especially in the Although the focus of this article is specific to Dfield of mobile robots. The sponsorship classroom education, it is expected that these of such competitions ranges from local clubs of observations will also provide insight into the enthusiasts to large professional organizations, incorporation of design competitions into the such as the American Association for Artificial general educational experience. A competition Intelligence (AAAI), which sponsors the annual can reinforce key concepts of a course and AAAI Mobile Robot Competition and Exhibi- bring reality to an AI robotics project as well as tion as part of its annual conference. Awards produce a viable entry. Other teaching oppor- for competitions vary from certificates of merit tunities not covered in class, such as budgeting to significant prize money, such as the Associ- and improving communication skills, are not ation for Unmanned Ground Vehicle Systems discussed in this article. (AUVS) Annual Ground Robotics Competition The article begins by contrasting the ob- awards of $5000 to its first-place winner, $3000 jectives of competitions and educational expe- to the second place, and $2000 to the third. riences. The conclusion is that certain portions The Colorado School of Mines (CSM) has of competitions can be an important tool for been a leader in fostering the intellectual fostering intellectual maturity, as defined by development of its students through design ac- the Perry model (Pavelich and Moore 1996). tivities (Pavelich and Moore 1996). One out- Next, a classification of the major internation- come of this emphasis has been the incorpora- al robot competitions is presented. Back- tion of national design competitions directly ground information on the courses is present-

ed and how this information influenced the numbers of demonstrations of software agents matching of classes to a specific competition. playing competitive games. How the courses were organized to take advantage of the competition topics is discussed Competitions and the Perry Model next, with particular emphasis on grading and Although competitions clearly offer something class size, the spectrum of philosophies for to the research community, they also offer a managing competitions, and the difficulties more immediate payoff to the general educa- with cooperative learning. A section on field- tional development of the individual student. ing a team by integrating the class projects is A competition provides additional extrinsic also included. The article concludes with a syn- motivation for the students to mature, accord- thesis of the experiences at CSM into a general ing to the Perry model of intellectual develop- strategy for incorporating a competition into a ment (Culver, Woods, and Fitch 1990). robotics course. Perry’s model defines nine stages of increasing complex reasoning: Positions 1 and 2, dual- ism, reflect a student’s attitude that right and Competition versus Education wrong answers exist for all problems. At posi- Competitions The goals of a competition often have a surface tion 3, early multiplicity, students realize that intersection with the topics addressed by edu- knowledge includes methods for solving prob- focus on cation. However, competitions focus on win- lems and that there can be more than one right winning ning (although some define winning in terms of answer to a problem. Students reach position 4, (although design rather than performance), whereas edu- late multiplicity, when they begin to think and cation concentrates on teaching the methods analyze about the diversity of possible solu- some define that ultimately lead to success. Instructors use tions. At position 5, relativism, students evalu- winning in a variety of techniques in the classroom, most ate solutions from different contexts. At posi- notably laboratory exercises and projects, to tions 6 through 9, commitment within relativism, terms of ensure that each student has the opportunity the students are able to take into account that design to meet the learning objectives. The Perry the world is a changing place. model suggests that a competition can aid the A competition involves a clearly de- rather than intellectual maturity of students who are be- fined—but open-ended—problem to which performance), ginning to accept that there might be more there are many possible solutions. The students whereas than one correct answer to a problem. The pri- must be largely self-reliant to understand the mary issue for an instructor is how to identify problem and apply their knowledge and prob- education and integrate the appropriate aspects of a com- lem-solving methods; they must function at concentrates petition into the laboratory and project assign- the least at the early multiplicity position. ments. Working voluntarily with other students on teaching encourages the student to identify and evalu- the methods Competition Goals ate a variety of opinions stemming from the The sponsors of robot competitions generally scientific literature, the professors, and other that cite the following reasons for the competition: students, which should move the student to a ultimately to evaluate progress of the community in some late-multiplicity position. The typical team aspect of mobile robotics, for example, walking organization of an adviser plus student mem- lead to platforms; to encourage undergraduate stu- bers lends itself to a realization of relativism, success. dents to get more experience with mobile whereby the teacher serves as a consultant robotics, thereby improving both the quality of instead of an arbiter. The goals of the competi- students going into the work force and the tion, as opposed to other potential applications number of students willing to make a career in of robotics, serve to stress the contextual aspect robotics; to expose undergraduate students to of applying knowledge. faculty and research issues, thereby encourag- In addition to the general intellectual matu- ing them to consider graduate school; to ration of the student, a competition provides a encourage graduate students and their advisers forum in which to acquire and exercise specific to consider pressing but unsolved issues facing job-related skills. Mobile robot competitions, robotics; and to have fun. In effect, they are in particular, require sophisticated software, mini–grand challenges of a field to encourage interdisciplinary interactions, and teamwork. interdisciplinary interactions between academ- It is difficult for the students to make progress ic researchers in the hopes that such endeavors without applying good software-engineering will become commonplace. principles, especially testing and debugging The potential impact of these competitions strategies. The students will probably need to on research should not be underestimated. The collaborate with students from other disci- AAAI national conference hosts increasing plines; for example, the smooth control of a

78 AI MAGAZINE Articles mobile robot depends both on the power train and have a high potential for being completed. and on how fast the software can update what Shelter students from extraneous con- the next move should be. A good competition cerns. Competitions often involve many is designed so that it is virtually impossible for activities that would not be encountered in a a one-person team to succeed; so, interpersonal typical laboratory experience: acquiring-main- and project management skills are continually taining hardware, raising money, and so on. challenged within the team and refined. Although these activities are likely to be part of a student’s future career, they can take valuable Issues in Using Competitions time and attention away from the objectives of Given that a competition offers an opportunity the course. Learning how to do a budget can- for intellectual growth, the real guidelines in not replace learning how to program with reac- capitalizing on the educational opportunity tive behaviors in an introduction to AI robotics include the following: course or replace the reinforcement from Identify where the students are on the Per- applying techniques covered in class to a real- ry model scale and what aspects of the com- world project. petition will help them to reach the next These guidelines maximize the learning stage. Each class is different, but a useful experience in terms of advancing intellectual heuristic is to assume that the majority of the maturity. However, the professor will have to undergraduates are at position 3 (early multi- spend additional time restructuring a competi- plicity). Internal assessments of graduating tion into “class-sized” pieces as well as support- The faculty seniors at the Colorado School of Mines indi- ing all the remaining responsibilities. This cate that the majority have not reached posi- restructuring can be too much for professors adviser tion 4 (late multiplicity) by graduation. To attempting to balance research, service, and must have a reach students at positions 3 and 4, the instruc- other classes with some semblance of a person- strategy tors should identify topics within the competi- al life. Therefore, the following item is added to tion and focus on how the methods covered in the list of issues: for both class apply to solving these topics rather than Minimize the burden on the faculty advis- incorporating on how they generate novel solutions. One er. In addition to the effort required to inte- approach is for the instructor to specify a small grate the competition into a class project(s) the set of approaches taken by previous entrants to and supervise progress, the competition is like- competition one aspect of the competition (for example, ly to necessitate extracurricular activity and topological navigation), have the group of stu- travel, which places additional demands on the into the dents read the associated papers, and present faculty member as well as ties up various classroom an evaluation of why these methods succeeded research resources (laboratories, workstations, and (or not). A position 4 group, for example, gifted robot platforms) and travel money. Therefore, undergraduates, might be given a more-open the faculty adviser must have a strategy for managing topic and only one or two starting places in the both incorporating the competition into the extra- literature. classroom and managing extracurricular Ensure that the appropriate educational demands. curricular component is integrated into the competi- demands. tion experience. A competition offers the potential for an educational experience, but it A Classification of Competitions doesn’t guarantee it. As with other team pro- The ROBOHOO web site (www.robohoo.com) has jects, negative events can reinforce a student’s a list of major competitions and links to their tendency to shun teamwork (“I could have home pages. Robotics and Autonomous Systems done it better by myself.”) or transfer responsi- publishes synopses of recent and upcoming bility to another agent (“It was my events in their Robotics Competition Corner partner’s/the hardware’s/the software’s/the section. competition sponsor’s fault”). The competition Robot soccer is currently very popular, both experience should be orchestrated so that the in its RoboCup and MIROSOT (also known as students have a reasonable chance to succeed if FIRA) forms. These competitions emphasize they proceed correctly, discern the cause of fail- facets of multiagent control, although ures as they occur, and make correction if pos- MIROSOT tends to be less about AI and more sible to prevent such situations from occurring about building microrobots. The Association in future projects. The relatively low level of for Unmanned Vehicle Systems (AUVS) spon- intellectual maturity of undergraduates favors sors three competitions: (1) ground, (2) aerial, the instructor providing the overall architec- and (3) underwater (a new addition). These ture and design so that the students work on competitions tend to pose grand challenges portions that are both relevant to the course that persist for four or five years until the state

SPRING 2000 79 Articles

Expertise Educational Support New Equipment Level Teams? Restriction ? Competition Hardware Software K U M P AAAI1 WSNWSSYN AUVS2 Aerial S M W W M S N Y BEAM3 SNSSNNYN AUVS2 Ground Robotics S W N M S W N Y Khepera N S W W S S Y Y MIROSOT S M W M S S Y Y RoboCup M S W W S S Y Y AUVS2 Underwater S W N N S S N Y SAE4 Walking S N N S M W Y Y

1. AAAI = American Association for Artificial Intelligence. 2. AUVS = Association for Unmanned Vehicle Systems. 3. BEAM = Biology, electronics, aesthetics, mechanics. 4. SAE = Society of Automotive Engineers.

Table 1. Rating of the Characteristics of Major Competitions as Strong (S), Medium (M), Weak (W), or Not at All (N) Applicable. Expertise is either on hardware (H) or software (S); the target participants are either K through 12 (K), undergraduates (U), Master’s candidates (M), or Ph.D. candidates (P). Does the event explicitly facilitate new teams (Y/N), and does the event require speciﬁc equipment (Y/N)?

of the art catches up. The AAAI competitions structing platforms, or does it require signifi- vary from year to year, but the events stress cant mechanical or electrical engineering software-oriented entries. A new series of skills? The AUVS Unmanned Ground Robotics events and competitions involving the Khep- Competition poses a vehicle-guidance problem era robot have gained in popularity. The Soci- that sounds perfect for AI and computer vision. ety of Automotive Engineers (SAE) Walking In practice, the majority of entries are from Machine Decathlon and the American Society mechanical engineering departments. of Civil Engineers (ASCE) Lunar Construction Although the course work can emphasize the competitions usually generate clever mechani- software portion, a computer science team cal solutions, ignoring software despite the might not be able to field a viable entry. Some clear need. BEAM (biology, electronics, aesthet- competitions implicitly favor Ph.D. students ics, mechanics) robotics and its cousin, the who are using the competition as a problem International Fire-Fighting Home Robot Con- domain for a research question (for example, test, are often too simple to reinforce anything using aerial vehicles as a test case for real-time beyond the most straightforward reflexive visual control), but others target providing K behaviors implemented in circuits. through 12 students with an opportunity to The various competition web sites often do work with circuits. not give a clear picture of what the competi- The Expertise and Educational Level tion entails. Murphy (1999) identifies five key columns in table 1 rate the major competitions questions and attempts to describe the spec- in terms of the real target participant. The trum of answers. expertise is divided into hardware and software Table 1 shows a subjective rating of each of and rated as either weak (W), medium (M), or the major competitions. These ratings are strong (S). The education level of the team based on an examination of the rules and actu- members is divided into K through 12 (K), al participation for the past years of the compe- undergraduate (U), Master’s (M), and Ph.D. (P) tition. participants. The participation for each competition is hard to categorize crisply and is treated Typical Participants more as a fuzzy set. For example, successful Possibly the first question an instructor should AAAI competition teams almost always consist ask is, “Is the focus of the competition appro- of Ph.D. and Master’s students, with one or two priate for the team’s intellectual level and undergraduate members. Undergraduate teams goals?” For undergraduate computer science have tended to fare unevenly. Two high school majors, one must ask, “Is the competition suit- teams have appeared at the competition, only ed for computer science majors who tend to be to quickly withdraw. comfortable with programming but not con- A related question is, “How much faculty in-

80 AI MAGAZINE Articles volvement is necessary?” The answer, of tition out of reach for a team, in spite of the course, depends somewhat on the team, but quality of the software. However, the right some competitions are harder than others. In equipment can be beyond the reach of anyone the AUVS aerial competition, the winners tend except researchers active in a particular field, to be teams led by one or two graduate stu- which also introduces the risk of using expen- dents whose Ph.D. thesis was a key element. sive research equipment for educational pur- The AAAI competition typically has a clear poses. Competitions such as RoboCup and graduate student flavor; the only undergradu- MIROSOT have small leagues where individual ate teams that do well (both in score and in robots can be constructed for approximately general respect from the robotics community) US$500. MIROSOT goes even further and per- have had a clear involvement from an adviser mits a hardware-oriented team to build vehi- active in research. Among the soccer competi- cles for a software-oriented team. Also, from tions, RoboCup has a strong research focus that time to time, robot manufacturers offer to loan suggests a high level of faculty supervision vehicles or sensors to certain teams. would be needed to field an undergraduate Regardless of the source, hardware must be team. available and reliable for the class to proceed smoothly. Given the demands of research and Novice Entries a general the lack of funding for the replace- An important criterion for an instructor to con- ment of expensive equipment, the use of Given the sider is whether the competition is amenable research equipment for education presents a demands of to new teams. Making a competition accessible fundamental conflict. Students new to the area is a difficult part of being an organizer. The need time on the robot, and faculty and research and AUVS competitions assume that the team has research assistants also require access to the a general been working on the entry for years; indeed, equipment. Classroom students can jeopardize lack of the latest aerial vehicle competition is so com- the research activity of a laboratory if they mis- plex, it expects a three-year commitment. One treat equipment. The dual use of equipment funding for of the biggest challenges for the AAAI competi- can literally bring research to a halt. I must the tion is ensuring the events are challenging note, however, that only minor damage has enough to promote interesting solutions but occurred in the research laboratory during edu- replacement not so hard that the robots look incompetent cational use. The students have generally been of expensive to the larger AI community. A new approach in protective of the robots and felt privileged to AAAI is to offer an entry-level event focused on be trusted with them. equipment, the state of the practice. Competitions such as Time management of robotics equipment the use of RoboCup explicitly encourage sharing code for both research and education can be compli- research and hardware designs. MIROSOT offers exten- cated. Student competitions can require that sive workshops in the summer for new com- the robot be unavailable for a sizable portion of equipment for petitors. Table 1 has a New column indicating time. For example, a robot was shipped to education whether the competition is generally support- RoboCup in Japan. The robot entered the ive of new teams. country with few problems and only a day late. presents a (In another competition, a competitor’s entry fundamental Equipment Restrictions arrived not only late but also upside down, Does the competition specify, restrict, or ex- damaging the robot beyond on-site repair. ) On conflict. pect certain types of hardware? Clearly, a Khep- return, a rubber wolf mask on the stereo cam- era competition requires Khepera entries. era pair, to make the robot look more animal Other competitions do not specify the hard- like, was not explicitly listed on the manifest. It ware, but the rules implicitly favor a certain resulted in more than a two-month delay type. For example, the rules of the AUVS before the robot arrived back in the United Ground Robotics Competition were originally States, significantly interfering with another written with a golf cart in mind as the typical research project. Contention for equipment vehicle. Smaller vehicles based on a children’s also hampers evaluation. Rarely was a class able battery-powered jeep were able to take advan- to access the CSM robots long enough to pro- tage of favorable obstacle layouts. Likewise, duce a statistically significant analysis of the recent winners of the AUVS Aerial Vehicle robot’s behavior. Because evaluation is critical Competition have all been hobby helicopters in science, this lack of access is especially using differential general problem solving for disturbing. control. Table 1 has a column entitled Equip- ment Restrictions that shows competitions Costs and Rewards that have significant hardware requirements. Finally, what’s in it for the team? Using a com- The wrong equipment can put a compe- petition for a classroom exercise is one thing,

SPRING 2000 81 Articles

but it is another set of issues to actually enter Advanced Robotics course. This section the competition. Entering a competition describes each of the courses, justifies the means traveling to it as well as shipping the match to a competition, and reports the robots. Some competitions offer travel scholar- results. It also provides a discussion of what ships (although it appears only AAAI main- lessons were learned in matching and how tained this practice for 1999); others offer they might transfer to larger classes. The next prizes. To maximize the amount of travel mon- section details the incorporation of the ey for students traveling, as well eliminate the competition into the course organization. week or two associated with shipping, a robot was shipped only once to a competition. Typi- Introduction to Robotics cally, the students and adviser drive to the Introduction to Robotics is an upper-level com- competition, often spending between 18 and puter science course that attracts approximate- 36 hours together in a van. The first year’s trav- ly 15 to 30 undergraduates. The undergradu- el from Denver, Colorado, to Washington, ates are usually juniors or seniors in computer D.C., was in an non–air conditioned van. The science or engineering. The course is a survey, van got so hot, the circuit boards on the robot concentrating on software issues. AI was not a Travel costs got soft and flexible to the touch. After that, prerequisite, although C++ programming expe- the team had an air conditioned van! Travel rience and a course in software engineering for the eight costs for the eight teams discussed in this arti- were. Engineering students typically met these teams cle averaged $2200 for the robot and four stu- prerequisites by virtue of being enrolled in the dents attending a competition in the United computer science elective program. The objec- discussed in States and $4000 for an international event, tives were the same each year: this article including customs fees. At the end of this course, the student averaged The rewards to the team also vary by compe- should be able to demonstrate a knowl- tition and individual preferences. Many stu- edge of the basic AI paradigms in robotics, $2200 for the dents are eager to compete to travel. Others are the ability to design and implement a re- robot and four motivated by prize money. In 1997, a group of active behavior, familiarity with common advanced students were given the choice of techniques for navigation, competence students working on three different competitions: (1) with common industrial vision and ma- attending a AUVS Ground Robotics held outside Detroit, chine-perception techniques, and the abil- Michigan, offering a $5,000 purse; (2) AAAI in ity to integrate sensing and action. competition in Rhode Island, which was considered the most Undergraduate students at CSM are required desirable competition because CSM had a pos- the United to take a six-week practicum called Field Ses- itive history of participation; (3) and a new sion during the summer after their junior year. States and competition, RoboCup, held in Japan. Funds Students who have taken Introduction to were guaranteed for travel to Detroit or Rhode $4000 for an Robotics were eligible to work with permission Island but not Japan. The majority of the stu- in the CSM Mobile Robotics–Machine Percep- international dents chose the AUVS competition; the next tion (MR–MP) Laboratory for their field ses- most popular was AAAI, and one took a chance event, sion. Generally the best performers in Intro- that funding would become available for a trip duction to Robotics opt to do so, and for the including to Japan. Ultimately, participation in all three purposes of this article, Field Session is treated customs fees. accommodated all students’ tastes. as an extension of Introduction to Robotics. This was the equivalent of offering a special Matching Courses topics or senior design course for the best stu- to Competitions dents the following semester. Introduction to Robotics has been taught five times. Each time, Three different courses resulted in eight teams motivating examples and class projects were for three different competitions over a five-year taken from the AAAI Annual Mobile Competi- period (1993–1997). Two of the three courses tion and Exhibition, and the students attended are taught annually during the spring semester: the competitions. In 1997, when there were 32 Introduction to Robotics and Computer Vision students, a group of undergraduate students and Advanced Robotics and Computer Vision. who had already done robotics laboratory work A third course is a follow-up to Introduction to were permitted to work on a RoboCup entry as Robotics and is taught for six weeks immediate- their project. In all cases, I served as both the ly following the spring semester. The AAAI class instructor and the team sponsor. Mobile Robot Competition and Exhibition and The AAAI competition was a good match for RoboCup World Cup were matched with the the introductory course on the conceptual lev- Introduction to Robotics course and the AUVS el because the events generally involve both Ground Robotics Competition to the behaviors and navigation, allowing the stu-

82 AI MAGAZINE Articles dents to use a hybrid deliberative-reactive architecture. Events typically concentrate on having a robot act as delivery person, going from one room to another, which is routine enough to look easy but challenging in practice. Furthermore, it emphasized software development and integration of existing research ideas as well as putting research into practice. Finally, I have an active research program in similar domains, providing the students with an architecture and reusable code rather than having to write a complex system from scratch. The competition was also a good match from the hardware perspective because it tended to favor indoor research robots that are either slow and bulky (for example, DENNINGs) or small and mostly harmless (for example, PIO- NEERs). Either extreme is attractive when teaching undergraduates who might not correctly A B program the acceleration and velocity gains. Speed is generally not an issue, and the competition arena is similar to a robotics laboratory. The team used a DENNING-BRANCH MRV4 named CLEMENTINE, shown in figure 1a, for four years until she was replaced with CRISBOT, a Nomadic 200, shown in figure 1b. The primary disadvantage of the AAAI Mobile Robot Competition and Exhibition was that in most years, the specifics of the rules were not posted until the middle of the spring semester. Another disadvantage was that the competition often favored teams that were able to purchase or engineer hardware. For example, in 1995, our team tied with the first-place winner on computer vision performance but took fifth because of a penalty for not having a manipulator on CLEMENTINE. The 1993 and 1994 teams did not place in the competition because of hardware failures. C The 1995 team placed fifth in the Recycling Trash event and was the only team composed solely of undergraduates. In 1996, the Intro- Figure 1. Robots Used by the Introduction to Robotics Course. duction to Robotics team placed sixth in the A. CLEMENTINE, a Denning-Branch MRV4. B. CRISBOT, a Nomadic 200. C. SILVER BUL- Office Navigation event. From 1993 to 1996, LET, a custom platform. the teams used CLEMENTINE. In 1997, the Introduction to Robotics team competed in two events because the larger class size necessitated more project topics. The 1997 The Introduction to Robotics team also par- team placed third in Technical Challenge and ticipated in the expert demonstration league of third overall in the Hors d’oeuvres, Anyone? the RoboCup World Cup Competition, held in event using CRISBOT. The team earned a third conjunction with the 1997 International Joint place technical challenge award in the Find Conferences on Artificial Intelligence. Original- Life on Mars event before a frame-grabber fail- ly, the team was to compete in a large-size ure prevented it from competing in the final league of Nomadic 200 bases, but the other round. The robot, SILVER BULLET, was built by an teams did not materialize. The RoboCup entry interdisciplinary undergraduate team spon- was able to reuse computer vision routines for sored by the National Science Foundation segmenting and tracking a ball, allowing the through the Research Experiences for Un- students to concentrate on docking and apply dergraduates Program (figure 1c). results of evidential reasoning research.

SPRING 2000 83 Articles

ics were better suited to taking care of an outdoor platform (they literally were more likely to bring it in out of the snow and rain) and developing software for a more challenging environment. Advanced Robotics could have used the AAAI competition as well, but because both the Introduction and Advanced Robotics are always taught in the same semester, there would have been significant contention for the robot, and the undergraduates might have been stifled by the graduate students. The objective of the Unmanned Ground Robotics Competition is to have a robot follow a road (white lines painted on a golf course), avoid obstacles, climb ramps, and so on. Because the competition is outdoors, a standard indoor research robot platform cannot be used, the computer vision component is much more difficult, and testing is more involved. The MR-MP lab was loaned an outdoor robot by Omnitech Robotics, Inc., a local Colorado firm. A In 1995, the team was given the robot out- right. Figure 2 shows the robot, C2, at the 1997 competition. The competition was a good match concep- tually but a poor match in practice. The competition participants were largely drawn from mechanical engineering departments. Design documentation was later required by the competition, but the design phase was sponsored and judged by the Society of Automotive En- gineers and did not consider software. In addition, the competition was held outdoors, and in 1997, a frame grabber was damaged because of rain. The graduate courses at the University of South Florida (where I have recently moved) do not make use of the Unmanned Ground Robotics competition, given the focus on hardware, the lack of participation by teams from B schools with a strong AI program, risk to hardware, and no incentive for innovative AI and computer vision. Figure 2. C2 Robot Used by the Advanced Robotics Course. The 1994 Unmanned Ground Robotics team A. On the field during the 1997 Unmanned Ground Robotics Competition. B. A took first place and a $5000 prize, and the 1995 Close-Up View. and 1997 teams placed fourth and eighth, respectively. All teams are considered success- Advanced Robotics ful, but the 1995 team was the most notable. Advanced Robotics is a graduate course with This team used a more sophisticated approach five to eight students, mostly from computer based on a thorough evaluation of the major science. Half of the students are undergradu- approaches to vehicle guidance and produced ates who are taking the course as a technical two papers and a software library for use in fur- elective. The course is an advanced treatment ther research endeavors in vehicle guidance. of material presented in Introduction to Robot- The 1997 team was plagued by hardware fail- ics, concentrating more on AI techniques used ures. Rain fell almost continuously for the in mobile robotics and situated agency. three days of the competition, and the frame Advanced Robotics used the AUVS Un- grabber shorted out because of moisture. The manned Ground Robotics Competition as team did produce an interesting application of motivation. The students in Advanced Robot- visual attention.

84 AI MAGAZINE Articles

Course Organization presentations based on outside readings se- lected by the instructor (40 percent). The pro- The Introduction to Robotics and Advanced ject was also subdivided into milestones: litera- Robotics courses were similar in organization, ture search and evaluation, implementation, with the competition serving as a theme for and final report. the course. Experiences with assigning multiple groups the same subtask were mixed; it Class Size and Competing Groups appeared to hurt performance in the in- A large number of class members makes it more troduction course but produce good results in difficult to ensure that each student partici- the advanced course. Both courses were man- pates. In the Introduction to Robotics classes, aged with a coach style of faculty interaction, the competition task was divided into indepen- which assumed that the students were at a dent subtasks and assigned to each student or position 3 or 4 level of maturity. This coaching group of students as a project. These groups are style was a balance between the instructor intentionally small and consist of two to three micromanaging the project and true coopera- students. If possible, engineering students were tive learning. A true cooperative learning paired with computer science students to pro- approach was deemed to have serious draw- mote interdisciplinary interactions. In two of backs, especially given the intellectual level of the five years, the Introduction to Robotics the students. classes had many more students than subtasks. Grading Scheme In these instances, more than one group was given the same subtask, and the results of each The Introduction to Robotics and Advanced group were compared. As expected, different Robotics courses used their associated competi- groups did produce some diversity in tions as a theme in all aspects. First, the com- approaches and results, but the duplicate petition served to motivate classroom discus- assignments also appeared to disenfranchise sions of how a particular technique would be some groups. Felling less critical to the overall applied to a real-world problem. Second, the project allowed them to be more indifferent to competition supplied the domain for home- it: If their code didn’t work well, some other work assignments, for example, program a group’s code would. One solution is to use mul- potential field to represent the X behavior. If tiple events or competitions for the project. the specifics of the competition for the class were not available, the event of the previous Duplicating subtasks among groups pro- year’s competition served as the sources for duced different results with the Advanced motivating examples and assignments. In addi- Robotics course. It was common in that course tion, the students used the classroom discus- to have groups implement a different method sions and assignments as scaffolding for their on the same platform and then compare final project on an assigned topic relevant to results. This approach worked well because the the competition. students were curious to see which technique Grading for Introduction to Robotics was would work the best. Although the groups were based on four components: (1) 6 to 10 home- competitive, loudly rooting for their algorithm work assignments (30 percent), 3 tests (20 per- to outperform the other groups during trials, cent), outside readings (10 percent), and a the groups worked together to fix control prob- group project (40 percent). The individual lems, develop common code segments, and so homework assignments often involved pro- on. gramming. They were intended to ensure that The possible negative impact of competition each student had tackled each of the major in the classroom is described in Johnson, John- topics of the course, including potential fields, son, and Smith (1991) and should certainly be sequencing behaviors, topological and metric avoided. However, in both courses, students path planning, and computer vision. The were divided into groups only to implement or group project was typically subdivided into evaluate cutting-edge research results, not to equally weighted milestones: simulation of the rank their individual performances. A group is reactive behavior, implementation of behavior evaluated on its understanding of the with ultrasonic perception, incorporation of approach, the quality of the implementation, deliberation, replacement of ultrasonic percep- and follow-up analysis of the performance. As tion with computer vision, and final report. a result, all groups usually receive an A, re- The instructor posted the list of the topics for gardless of which technique shows superior the project and let the groups decide which performance. Competing groups learned as one they wanted to tackle. much as if they had all been working together. Grading for the Advanced Robotics course This observation is consistent with the study was based on a group project (60 percent) and by Sherman (1989) on competition versus

SPRING 2000 85 Articles

increasing instructor management increasing student autonomy

Micro-manager Coach True cooperative learning

Figure 3. Spectrum of Styles of Managing Competitions.

cooperation when teaching biology. age intellectual development) and intercedes As an experiment in motivation, a bonus of to make adjustments as needed. a half letter grade was awarded to the group in the 1995 Advanced Robotics class that showed Difficulties with Cooperative Learning the highest performance in each of two In the cooperative learning approach, the demonstrations. In the first demonstration, the adviser serves as a consultant, or benign student groups had similar techniques, and the remote manager, and the students are expected bonus grade served only to motivate the stu- to self-organize, identify the problems, gener- dents to keep adjusting thresholds on the algo- ate potential solutions, and evaluate the rithm during the two-hour–long demonstra- results. This expected level of interaction is tion. In the second demonstration, one essentially at position 5, relativism, of the Perry technique was clearly superior to the other, model. However, undergraduate students are and the bonus grade served no purpose. It did often at position 3 or 4, introducing a severe not appear to demoralize the second team mismatch between expectations and ability. because it knew that the difference between an The cooperative learning model was used A and an A+ was negligible. The bonus grade with two teams (1993 and 1994 Introduction was deemed to be superfluous and was discon- to Robotics) with uneven results. The students tinued in future classes. simply did not have the technical knowledge and methodology necessary to identify prob- Cooperative Learning Spectrum lems and good solutions until the end of the In addition to organizing the project to accom- semester. Even during the follow-up practicum modate class size, the classroom instructor course, after they had been exposed to the the- needs to consider how to best manage the ory, they still did not have enough hands-on learning experience. My experience and discus- experience to effectively apply this knowledge. sions with other advisers suggest that there is a Formal cooperative learning methods did not wide range of styles for managing projects seem to enhance the educational process. The related to competitions, as shown in figure 3. students were immediately experiencing the There is the true cooperative learning philosophy benefits of cooperative learning: engaging in (Johnson, Johnson, and Smith 1991) that the material, acting as peers with other stu- emphasizes student involvement and interac- dents and professionals, and developing and tion and little to no instructor involvement. refining problem-solving skills. Major obstacles The cooperative learning approach is not rec- to cooperative learning included a lack of prac- ommended, despite its popularity among edu- tice and self-confidence in applying their skills. cators for reasons stated later. At the other end In general, they had not yet internalized the of the spectrum, there is the micromanager procedure of identifying a problem, generating approach, where the adviser structures every many solutions, determining evaluation crite- aspect of the experience. This technique was ria, and applying these criteria to the solutions. not used for any of the teams because its edu- In spite of the mismatch between man- cational merit and use of time were question- agement styles and the students’ level of intel- able. Most teams were managed from some- lectual maturity, the 1993 Introduction to where in the middle of the spectrum with good Robotics team produced a solid entry results. The middle of the spectrum is the coach (although the entry did not place because of a style, where the instructor closely supervises hardware failure) primarily because its task was the activities of the team (intended to encour- simpler. The winning team from the Jet Propul-

86 AI MAGAZINE Articles sion Laboratory/California Institute of Tech- approach of trial and error, where the students nology consisted of one person who pro- would ideally try their approach and see it fail grammed the robot the night before. The 1994 compared to others, did not seem applicable. team was never able to grasp the increased Because students were clearly violating a fun- complexity of that year’s competition, and damental principle of computer science in both the learning experience and the entry applying a less optimal solution, the adviser were weak. In contrast, the 1995 to 1997 teams and practicum instructor each spent two hours operated satisfactorily under a more structured attempting to review complexity analysis and strategy. These classes were given an overall algorithm design with the team in an attempt design for the competition entry and tasked to to get the students to change their minds. In work on modules that were educationally rele- the end, the team had to be ordered to use the vant to the class. Experience indicates that stu- better algorithm. dents are best motivated when they can see the Based on these experiences, perhaps compe- “big picture” at the beginning of the semester titions are too challenging for the students to before they actually begin work. Unfortunate- function cooperatively at the relativism level ly, students are generally not knowledgeable and higher without significant support from enough to design such a framework at the the instructor. This is particularly noteworthy beginning of a course. More structure, com- because it appears that many advisers of under- bined with improved hardware, contributed to graduate teams treat them as if the members four third-place awards. were at position 5. Experience also suggests that the cooperative This assumes that the students have had suf- learning tenet of getting local experts involved ficient coursework and have a high level of is impractical and inappropriate. In one varia- intellectual maturity. Experience tion of the cooperative learning approach Other instructors are genuinely concerned also suggests reported by Tribus (1993), students were that they will stifle their students’ creativity. In expected to contact professionals in the field the worst case, this assumption results in a that the and solicit opinions about what was impor- “sink or swim” attitude. cooperative tant, in addition to the other activities. This Providing an overarching architectural more extreme cooperative learning paradigm, framework, software modules to work on, and learning tenet having the students interact with professionals the set of methods to evaluate did not appear of getting to research the aims of the course and its pro- to interfere with student creativity. Creative jects, is not necessarily appropriate. Mobile students found (or made) opportunities to local experts robotics is a new field, and the competitions infuse the project with their ideas, especially in involved is use scenarios that require significant innova- the follow-up course. By tailoring the class to impractical tion. There are few experts in the field, and the positions 3 and 4, weaker students consistently team adviser is likely to be the most knowl- did well and often included their project as and edgeable person. In addition, the relationship portfolio materials for interviews. inappropriate. between the university and local industry can be strained by having students constantly Integration of Projects query an industrial expert. However, a robot In the Introduction to Robotics courses, inte- competition sponsored by a national profes- gration of the class projects was deferred until sional society could be interpreted as being the follow-up practicum course. Integration of cooperative learning because the competition software modules and software with hardware allows professionals and students to interact is essential to producing an entry in a compe- (although in a more controlled fashion). tition, but it involves a great deal of work unre- Cooperative learning’s most important lated to the class. It can be deferred until after down side is group dynamics. If allowed to the class has been completed. If the instructor’s work autonomously, both the undergraduate framework is reasonable, and the students have and graduate students tended to generate only addressed the critical intellectual issues during one solution to a problem and then “fall in the class, the smaller set of volunteers or mem- love” with it. The 1994 AAAI team generated a bers of an independent study course should be provably more computationally complex algo- able to readily integrate the modules. Further- rithm to searching a graph that represented a more, because integration is extracurricular, topological map, ignoring the well-known sin- members of the class who were not high per- gle-source shortest-path algorithm developed formers or were not committed to the team will 20 years ago for this specific class of problems. drop out, and the team will be more produc- Because this algorithm was one of several soft- tive. ware components, the degradation in perfor- Another advantage of deferring integration mance might not have been noticeable. The is that the instructor can tailor management

SPRING 2000 87 Articles

styles. During class, the instructor acts more the first six weeks of class in place of assign- like a coach, setting up practice drills, calling ments. These inexpensive kits are well suited for the plays, and so on. Based on interactions illustrating reactive behaviors and simple per- with the students during the semester, the ception, and the Kheperas support a project instructor can decide how much autonomy the with advanced navigation and computer vision. follow-up group is able to handle. It is anticipated that many students will pur- The major disadvantage of deferring in- chase their own kit to keep, further reducing the tegration is that gratification is also deferred. cost of the laboratory to the department. Some students might not be able to participate on the team beyond their contributions in class, even if they are high performers. Fielding the Entry This article argues that a high degree of structure Recent Modifications is useful for integrating the intellectual chal- The equivalent to Introduction to Robotics has lenges of the robot competition into a course. As been taught once at the University of South a result, a significant portion of the work needed Florida with an enrollment of 40 students, to field the entry must be done outside class or almost all computer science majors. Because of in a follow-up class or summer Research Experi- the larger class size and initial laboratory limita- ence for Undergraduate format. In the case of tions, no competition was integrated into the Introduction to Robotics, the instructor invited classroom. Two modifications are being added to high performers to work on the competition as the course to compensate for the larger class size part of their practicum course. At USF, an inde- and to reintroduce competitions into the course. pendent study course is offered. The first modification is the incorporation of Competition activities can provide an addi- Khepera robots as dedicated educational tional opportunity for intellectual development. robots, and second, the use of a soccer For example, students can now consider hard- competition is tentatively planned. Dedicated ware. Hopefully, they will have acquired enough hardware for the course reduces the problems intellectual maturity to identify the real issues to associated with using research robots for edu- generate and evaluate multiple solutions, even cation. Two Khepera robots have successfully if the solutions have little to do with the class been used during a USF Introduction to Robot- material. This section describes issues associated ics project during the last six weeks of the with fostering intellectual development during course. Kheperas are relatively inexpensive the extracurricular activities. (US$2000 for a basic model to US$8500 for one The biggest observed impediment to addi- with color vision and a gripper) and reliable. tional intellectual development during An excellent UNIX-based simulator is available extracurricular activities has been group (a WINDOWS version is under development) that dynamics. The group norms are partially estab- allows the students to test their code before lished during class, but student attrition allows physically interacting with the robot and pos- the norms to be malleable. Two aspects of the sibly damaging it. Their small size (smaller class organization influence the final group than a soda can) makes it easy to physically set dynamic: (1) overall structure of the course and up and create test domains. Initially, the class (2) milestones. In addition, having students was allowed to program the robots individually work on the team in subsequent years is also an using the simulator. Preliminary knock-out advantage. Traditional group performance rounds were held on the simulator, and semi- modifiers, such as writing mission statements, finals and finals were conducted on the physi- have produced mixed results. cal robots during class time. In the future, The structure of the projects during class groups of two or three students will program a gives the students an implicit standard op- team of soccer robots using a MIROSOT or erating procedure on how the adviser expects RoboCup small-league arrangement. Another decisions to be made outside class: identify option is to use Tucker Balch’s TEAMBOT JAVA sim- problems, generate solutions, and evaluate. ulator, which can be customized for use with Milestones are essentially deadlines. any robot platform. The dynamics of groups in the class were Although the use of a simulator helps manage influenced by project deadlines. These dead- the contention for hardware resources in a large lines put pressure on the students and caused class, it has the negative consequence of elimi- the groups to “form” and “storm,” using Tuck- nating true hands-on experience. Therefore, the erman’s terms (Gray and Starke 1988). This ear- second modification will be to regain exposure ly state of group development presents a prob- to hardware. Lego Mindstorms and Rug Warrior lem at the end of class, when a small fraction of kits will be used in laboratory exercises during the students in the class elect to work on the

88 AI MAGAZINE Articles team, either for course credit or as a volunteer. dents of this fact. The author’s mission has The students who stay are often from dif- always been to derive the most educational val- ferent project groups, leaving the roles and ue from the competition. Winning is never norms within the team unclear. Milestones part of the mission as such, but students are developed jointly with the sponsor and the reminded that because they have addressed the team serve to put pressure on the team to per- competition with intellectual rigor, winning is form and began evolving toward a favorable anticipated. dynamic. It can be difficult to arrange respon- An important aspect of fielding an entry is sibilities so that everyone is equally involved. the follow-up. Students will be reluctant to For example, if a student is in charge of hard- compose an analysis of what happened and ware, but the hardware does not break during why. If the competition is an extracurricular the competition, the student does not feel as if activity, students cannot be forced to submit a he/she contributed. A possible solution is write-up. The value of the follow-up report assigning each student a primary and a sec- comes from encouraging students to think in ondary area of responsibility. Therefore, a stu- analytic terms. The instructor generally knows dent in charge of hardware would be knowl- what happened and why and how to avoid the edgeable enough to help test and modify problems encountered later. Also, the instruc- navigation code unless a hardware matter tor is the one most likely to do the bulk of the Mentoring arose. writing for any publication. under- Continuity in the teams between years is Fentiman and Demel (1995) present a formal helpful on many levels. A senior can lecture a methodology for having students document graduates junior on the importance of software engineer- their design projects and present results. Many has now ing, and the junior learns to have faith in the of the documents and steps involved would be instructor. The instructor is reassured that the useful if the sole objective were the competition. become senior actually learned something. The methodology appears significantly time a tradition An unexpected positive outcome of two consuming, so much so that it is unclear classes using the same robot platform because whether students would complete it voluntarily. among of hardware problems has been that the under- An alternative strategy is to request each student graduate graduates in Introduction to Robotics have had produce a one-page report. Because the follow- students, a chance to mingle with the graduate students up report is an important educational tool, it is in Advanced Robotics. The graduate students worth the additional time to repeatedly urge and the provide role models and peer reinforcement. In students to submit it. Even if the report is never under- five cases, the graduate students spontaneously written down, it will stimulate reflection on the “adopted” an undergraduate student and student’s role in the project. graduates encouraged him/her to go on to graduate are school (three did). Mentoring undergraduates has now become a tradition among graduate Conclusions frequently students, and the undergraduates are frequent- In conclusion, design competitions can effec- referred to ly referred to as “virtual grad students.” In tively be integrated into coursework. The Intro- as “virtual addition, one talented but unruly undergradu- duction to Robotics course consistently earned ate was reportedly told by a graduate student a much higher rating by the students than the grad that his attitude toward the robot was un- departmental average for courses. At the same students.” acceptable, and his behavior subsequently time, the course received only a slightly above- improved without further intercession. average rating on work load, indicating that Despite their popularity in business, mission the project was perceived by the students to be statements written by the teams either during within their capabilities. The Advanced Robot- class or during the extracurricular portion have ics course received higher ratings than the rarely had any effect on group dynamics. department average, and one year it was given Undergraduates at CSM have been exposed to a 4. 0 rating for the course overall. mission statements but largely reject them as a A design competition can provide both moti- creation designed for the instructor. Graduate vating examples of how abstract concepts can students tend to be more amenable to writing be transferred to practice and meaningful mission statements, but they still tend to dupli- assignments and projects. The added stimulus cate the mission of their adviser. Writing a mis- of the competition encourages the students to sion statement appeared to aid the positive put forth their best effort in identifying prob- group formation of only one team. What did lems, generating and evaluating solutions, seem to work was for the instructor to have a working in a group, and applying knowledge. A strong mission statement that included stu- competition provides an opportunity to further dent roles and to continually remind the stu- the intellectual development of a student, par-

SPRING 2000 89 Articles ticularly because the traditional team relevant challenges of the competition tees who help to make competitions organization favors students at the rel- into class projects that are implement- useful. ativism stage of Perry’s model. Under- ed and evaluated. An overarching graduate students are unlikely to be at framework or system architecture References that stage, and the adviser should help must be provided to facilitate stu- Culver, R. S.; Woods, D.; and Fitch, P. 1990. the students move toward relativism. dents’ ability to visualize the overall Gaining Professional Expertise through Follow-on courses or extracurricular challenge. Limitations on modifi- Design Activities. Engineering Education 80(3): 533–536. experiences can lead to a competitive cations to the hardware or other sys- entry, as seen by the five awards and tems must also be explicit to maintain Fentiman, A. W., and Demel, J. T. 1995. Teaching Students to Document a Design seven publications garnered by the student focus on solutions that sup- Project and Present the Results. Journal of CSM teams. Experiences at CSM were port the educational objectives. Fol- Engineering Education 84(4): 329–333. gained through small-sized classes but lowing the competition and outside Gray, J. L., and Starke, F. A. 1988. Organiza- should not serve as a deterrent from class, follow-up reports should be tion Behavior: Concepts and Applications. 4 trying the described methods. encouraged. ed. Columbus, Ohio: Merrill. Many advisers approach teams with This strategy provides some certainty Johnson, D. W.; Johnson, R. T.; and Smith, a sink or swim attitude, assuming that that the competition itself did not dis- K. A. 1991. Active Learning: Cooperation in the students have had enough course- tract from the educational enterprise the College Classroom. Edina, Minn.: Inter- work and experience to succeed on and that each student should experi- action Book. their own or learn from their failures. ence intellectual growth. The burden Murphy, R. R. 1999. Upcoming Competi- This assumption places students at the on the adviser is reduced in the long tions for 1999. Robotics and Autonomous relativism position already. Observa- run because the strategy minimizes the Systems 26: 233–234. tions of five undergraduate and gradu- frustrations owed to mismatched com- Pavelich, M. J., and Moore, W. S. 1996. ate classes and eight competition teams petitions, erroneous expectations of Measuring the Effect of Experiential Educa- suggest that it is better to expect the student intellectual maturity, and lack tion Using the Perry Model. Journal of Engi- neering Education 85(4): 287–292. students to operate at the early multi- of focus on the course material. Indeed, plicity stage. Rather than expecting stu- the strategy does more than facilitate Sherman, L. W. 1989. A Comparative Study of Cooperative and Competitive dents to be autonomous, a more ap- the integration of a competition into a Achievement in Two Secondary Biology propriate strategy is to treat the robotics course. Classrooms: The Group Investigation Mod- intellectual challenges inherent in the el versus an Individual Competition Goal competition as opportunities to prac- Acknowledgments Structure. Journal of Research in Science tice problem-solving skills and inter- Portions of the extracurricular activi- Teaching 26(1): 55–64. nalize these skills under favorable con- ties and equipment described in this Tribus, M. 1993. Quality Management in ditions. As the students reinforce their article were made possible through Education. The Journal for Quality and skills under supervision, they should National Science Foundation grants Participation 16(1): 12–21. move toward a position of relativism. CDA-9531730 and CDA-9617309. The Incorporating a design competition author would like to thank Julian Mar- Robin R. Murphy into course work requires a significant tinez and Bill Hoff for coadvising the received a B.M.E. in investment of time and thought. The 1994 AAAI and 1995 Unmanned mechanical engineering instructor must plan well in advance Ground Robotics teams, respectively, (1980) and an M.S. (1989) and a Ph.D (1992) to identify the most suitable competi- and the many student participants in computer science tion and adapt the relevant aspects to who make teaching worthwhile. NSF, (minor, computer-inte- the class population. The importance Nomadic, Omnitech Robotics, Inc., grated manufacturing of access to hardware and its timely DENNING-BRANCH International Robot- systems) from the Geor- maintenance cannot be overempha- ics, the Colorado Space Grant Consor- gia Institute of Technology, where she was sized. Assignments and projects need tium, and the CSM Foundation pro- a Rockwell International Doctoral Fellow. to be done on a robot to expose stu- vided hardware and travel support. From 1992 to 1998, she was an assistant dents to a realistic experience. If there AAAI and RoboCup also provided professor in the Department of Mathemat- is a hardware breakdown, learning is generous travel support. Jean Bell and ical and Computer Sciences at the Col- orado School of Mines. Since 1998, she has disrupted, and the students become Mike Pavelich provided many helpful been an associate professor of computer sci- frustrated. discussions on the lessons learned ence and engineering at the University of Integrating the opportunities from these experiences and especially South Florida. Murphy has received numer- afforded by a design competition into the anonymous reviewers. ous teaching awards, including a Colorado a classroom situation should be highly Mary Parrish provided invaluable Institute Gender Equity Excellence Award structured. The instructor must identi- assistance in editing. Tom Henderson for her efforts at the college and K through fy the aspects of the competition that and Franz Groen offered their encour- 12 levels in encouraging women and apply directly to the educational agement and positive reinforcement minorities to pursue careers in the sciences. She is finishing a textbook, Introduction to objectives of the class and defer the for the columns in Robotics and Au- AI Robotics (The MIT Press, 2000). Her e- other aspects to extracurricular activi- tonomous Systems. Many thanks to the mail address is [email protected] ties. The instructor must formulate the volunteers on the organizing commit-

90 AI MAGAZINE