Autonomous Robotic Reconnaissance Missions in Extreme Space Environments

Luminary Presentation 2013 PHM Conference, New Orleans, LA

Prof. Dr. Wolfgang Fink1,2 Associate Professor Edward & Maria Keonjian Endowed Chair AIMBE Fellow

1Depts. of Electrical & Computer Engineering, Biomedical Engineering, Systems & Industrial Engineering, Aerospace & Mechanical Engineering, and Ophthalmology & Vision Science Visual and Autonomous Exploration Systems Research Laboratory, University of Arizona, Tucson, AZ 2California Institute of Technology, Visual and Autonomous Exploration Systems Research Laboratory, Division of Physics, Mathematics and Astronomy, Mail Code 103-33, Pasadena, CA

Carl Byington

Andrew Hess Tim Wilmering Karl Reichard Ginger Shao

Jose Celaya Shankar Sankararaman Indranil Roychoudhury Yvonne Lopez Roxane Rose et al. © Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Acknowledgements

Collaborators:

Mark A. Tarbell Visiting Scientist and Senior Research Scientist Visual and Autonomous Exploration Systems Research Laboratory Caltech/University of Arizona

Wayne Waller Former Director Digital Media Center Caltech

Colleagues at various Institutions

Ophir Chasma terrain image courtesy ESA/DLR/FU Berlin (G. Neukum) © Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Example of Recent/Transient Events: Dark Slope Streaks

[Images courtesy of NASA/JPL/MSSS]

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Example of Recent/Transient Events: New Impact Crater (also: Volcanoes, Fires, Floods)

[Images courtesy of NASA/JPL/MSSS]

Mars Reconnaissance Orbiter (courtesy of NASA)

Mars Science Laboratory Rover Mission (courtesy of NASA) Phoenix Lander Mission (courtesy of NASA)

Patent number: US 6,990,406 Title: “MULTI-AGENT AUTONOMOUS SYSTEM” Authors: Wolfgang Fink et al.

Patent number: US 7,734,063 Title: “MULTI-AGENT AUTONOMOUS SYSTEM” Authors: Wolfgang Fink et al.

Patent number: US 7,742,845 Title: “MULTI-AGENT AUTONOMOUS SYSTEM AND METHOD” Authors: Wolfgang Fink et al.

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Tier-Scalable Reconnaissance Paradigm and Robotic Test Bed Featured in SCIENCE 30 July 2010 Vol 329

NEWSFOCUS NEWSFOCUS

Which rock? On Mars, the robot Opportunity Sensorweb, a group of half a dozen NASA ceed. He dreams of robots that can experi- solution (look at the fi nal digit) with no guid- needed some human help to spot this meteorite. satellites that monitor Earth’s atmosphere. ment with their own “neural networks”— ance. And by focusing on one bit, the com- Some scan large sections of Earth’s surface their internal architectures for taking inputs, puter freed 23 other bits for different tasks. Venus, for example, 450°C surface temper- and pick out a fl ood or a volcanic plume from processing information, and producing out- In other contexts, such self-confi guring net- atures and pressures comparable to those space. They beam the data to ground-based puts—and can, like humans, form their own works have helped scientists design circuits a kilometer deep in the ocean will destroy computers, which in turn direct higher- rules for exploring. and new drugs. the onboard computers of any lander within resolution satellites to focus on the event— McGuire says certain architectures have Ultimately, Fink says, he hopes to instill 5 hours, tops. To get anything done, the all without human input. Chien hopes to advantages in different applications. With a something like curiosity in robots. That kind lander will need to perform experiments, expand the work to other planets. But the so-called Hopfi eld neural network, for exam- of programming would go far beyond algo- such as sampling soil, without human input. instruments can spot only a short list of ple, a computer can recognize an entire pic- rithms his team has developed to help robots Rendezvous missions with comets or predefined events; they cannot find any- ture stored in memory after seeing only a frac- calculate the best angle to stretch out an arm asteroids and landings on distant moons thing interesting or new on their own. Asked tion. Many robots come equipped with mul- to grasp an object or scoop soil. “We’re after would also benefi t from more autonomous whether the system could shift its attention tiple lenses and cameras that take pictures on the intent to deploy the arm. How does the robots, researchers say. On Saturn’s moon on its own between the two most notable different scales, so the capability to tag small spacecraft know where it wants to dig? This Titan, radio waves carrying scientists’ geological events of the past few months, the snippets as familiar would help make the is of interest to me.” instructions take 90 minutes to arrive from Eyjafjallajökull volcano eruption in March robot more effi cient in selecting which scenes EXPLORATION Earth. Yet a probe flying through Titan’s and the BP oil slick in the Gulf of Mexico to shoot or not shoot. The fi rst test? atmosphere would have to negotiate hazards in April, Chien groans: “I wish, I wish.” Smart, curious robot explorers wouldn’t have in real time, notes Wolfgang Fink, a computer To solve problems of data filtering and to work alone. Fink envisions a multitier Making Smarter, Savvier Robots scientist working at the California Institute interpretation, some researchers are work- scheme of robots with satellites, blimps or of Technology and the University of Arizona. What machines of the future really need to learn, say experts who ing to cultivate a robot’s taste for the unusual. balloons, and platoons of ground rovers. An “If it’s about to fl y into a mountain range, it Sometimes scientists want to study the most intelligent satellite would direct the blimps plan to have them explore the far reaches of the solar system, is more can’t say, ‘I’m fl ying into a mountain range. representative feature around, but more often to canvass certain areas. The blimps, in turn, independent behavior

Please advise,’ and wait 1½ hours.” on July 30, 2010

they are intrigued by anomalies. “If the whole would direct surface rovers to scout hydro- on July 30, 2010 Scientists also hope that greater intel- desert is smooth and one area is rough, that’s thermal vents or rappel down cliff faces with ligence will make robots more efficient, interesting,” says Chien. “If the whole desert a cable. Based on feedback between each tier, improving their “energy storage, memory, is rough and one area is smooth, that’s inter- the satellite would decide which sites to con- computational throughput, communication esting. If you really don’t know about the centrate on and how best to deploy the other downlink bandwidth, and heating and cooling environment, you have to fall back on some- machines. It would judge when to risk send- capability,” says Larry Matthies, a computer thing like outliers.” ing rovers into dangerous areas like active scientist at JPL. Opportunity (and its com- Patrick McGuire, a computer scientist and volcanoes, and when to stop collecting data, panion on Mars, Spirit) travel at such pokey geologist at the University of Chicago in Illi- Fink says. “A spacecraft could even leave a www.sciencemag.org paces—28 kilometers total in 6 years—partly nois, has developed a simple setup that can place and tell you, ‘There’s nothing interest- www.sciencemag.org because they rely on humans to spot danger- detect novel features in a landscape. A net- ing here. I’ll go somewhere else and I’ll tell ously loose sand or steep slopes. A smarter book laptop hooked up to a cell phone with a you when I get there.’ ” Fink and his team robot could zip around obstacles by itself and camera snaps a picture and compares its col- have started building a test site in Arizona On 18 July 2009, the Mars rover Opportunity and at universities, are working on improving travel up to 10 times as far each day, Matthies ors, textures, and shapes with other pictures with rovers, boats, and blimps for ﬁ eld exper- was scooting toward a distant martian crater robot explorers. But only a few dozen special- estimates. And the more work the rover can in its memory. The computer then compresses iments with rudimentary versions of such when it spied an anomaly amid the ripples ize in developing robots with true, high-level do alone, the more time it will have to collect the image with an algorithm. If the compres- robotic expeditions.

of red soil: a bruise-colored rock the size of independence. The main NASA lab, at the Jet good samples. Downloaded from

sion process is very similar to that of an earlier Chien imagines a different sort of team- Downloaded from a watermelon. It looked like a meteorite— Propulsion Laboratory (JPL) in Pasadena, image, the computer concludes that the new work: human explorers with fast-learn- potential evidence that the ancient atmosphere California, has a dozen people and a bud- Recipes for “eureka” image doesn’t contain much novel informa- ing robot assistants. A group led by David of Mars, like today’s, was thin enough for such get of about $4 million—a lower fi gure than In December and January, NASA took the tion and throws it out. Akin at the University of Maryland, Col- rocks to pass through without exploding. in the past. But scientists there see promis- fi rst steps toward making a spacecraft auton- McGuire has tested this system at rock lege Park, is testing a golf cart–size three- The strange rock was exactly the kind of ing signs. For one, NASA chief technologist omous when it uploaded four pieces of soft- outcrops in Utah and Spain that resemble the Mechanized teamwork. Caltech’s Wolfgang Fink wheeled rover, named Raven, to help astro- thing NASA sent Opportunity to find. But Robert Braun has begun a new, general Space ware to Opportunity. Tara Estlin, a senior barren landscape a probe might encounter on foresees robots exploring in tiered ranks. nauts explore planets. On tricky terrain such because Mars and Earth are millions of kilo- Technology Program that lists “machine intel- engineer at JPL, explains that, with the new a distant planet. He reported late last year that as loose soil or slopes, Akin says, the astro- meters apart and rotate out of sync, NASA ligence” as one thrust. software, “scientists can give us a single prop- the software performed equally well in both Even more ambitiously, Fink is develop- naut can simply say, “Follow my path,” and scientists didn’t see it until Opportunity had “There are compelling reasons to send erty or combination of properties—the larg- locations. In one case, the software immedi- ing systems to give robots freedom to change the robot will. driven 200 meters beyond it. They hit reverse humans into deep space,” says Steve Chien, est rock you can fi nd, or the darkest rock,” and ately recognized a patch of lichen as novel— their logical architecture—essentially to President Barack Obama’s stated goals of but had to wait three full days for Opportunity who develops autonomous space systems Opportunity will zero in on them. In March, and then, with the next picture, threw out an “rewire” their brains. A robot might make sending humans to an asteroid in the 2020s to backtrack to the spot. for JPL. “A smart scientist can do much bet- the software passed its fi rst test by discover- image of lichen on another rock as too similar a rule more complicated or simpler by add- and to Mars in the 2030s could help foster The researchers got their meteorite. But ter experiments. But it’s very expensive. By ing, all on its own, an angular, football-size to bother remembering, demonstrating that it ing or cutting steps, or combining the binary such partnerships. Chien says human-and- the near miss—and the frustrating delay— making the spacecraft much smarter, we can rock—ejected from a nearby impact crater— is a quick study. code of two rules and trying out their “off- robot teams could do a better job together underscored a defect of current exploration reduce the gap between human exploration in a fi eld of rounder boulders. (Paradoxically, spring.” If the new rules worked well, it adds than either could alone. Humans would make technology: Basically, robots are pretty dumb. and robotic exploration.” though, Estlin’s team still has to tell the rover Curious future them to its problem-solving repertoire. plans and be in charge, while robots slogged Now scientists across the world are striving to a day in advance when to be autonomous and Some scientists, including Fink, say better pro- Fink published a paper last year on a self- through the important but routine technician change that by developing intelligent robots Where to go when not to.) gramming alone won’t turn robots into inde- confi guring neural network to sort odd num- work. “It’s the classic apprentice thing,” Chien that can circumvent danger and spot enticing Robots with an IQ boost will be essential Earth-based systems have already dem- pendent explorers. “In planetary exploration, bers from even numbers. Working with num- says. “You want the biggest brainpower wor- features on their own. for fully exploring some locations in the onstrated signifi cant independence, within you’re in for surprises,” Fink argues, “and you bers 24 bits long—in the tens of millions in rying about the biggest problems.” –SAM KEAN CREDIT: DR. WOLFGANG FINK AND WAYNE WALLER, VISUAL AND AUTONOMOUS EXPLORATION SYSTEMS RESEARCH LABORATORY, CALIFORNIA INSTITUTE OF TECHNOLOGY OF INSTITUTE CALIFORNIA LABORATORY, RESEARCH SYSTEMS EXPLORATION AUTONOMOUS AND VISUAL WALLER, WAYNE AND FINK WOLFGANG DR. CREDIT: Hundreds of scientists, mostly at NASA solar system—including hostile spots. On limits. Chien works on the Earth Observing NASA/JPL NASA/JPL-CALTECH; (MAIN) CREDITS: will not always have a rule” on how to pro- decimal notation—the network hit upon the

508 30 JULY 2010 VOL 329 SCIENCE www.sciencemag.org www.sciencemag.org SCIENCE VOL 329 30 JULY 2010 509 Published by AAAS Published by AAAS © Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Public Exposure, Media Releases, and Awards

received NASA Board Award November 2009

Advantages of Tier-scalable Reconnaissance Architecture: • Overhead navigation affords deployment of less smart, therefore less costly, expendable ground agents • Deploy multiple expendable ground agents in target zones – reduced driving requirement • Mission safety and redundancy • Optimized target identification and obstacle avoidance • Efficient (overhead) commanding of multiple ground agents

Example (onboard) Instruments/Sensors: • Optical, IR, UV cameras • Hyperspectral cameras • Sonar-systems • Ground penetrating radar (GPR) • MEMS-based devices and sensors

Application Examples: • Operations in hazardous environments (incl. military, biochemical, radiation, or natural disasters) • Reconnaissance operations (incl. planetary, military, civilian) • Surveillance operations (e.g., Homeland Security), Stand-off Detection (IEDs)

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Tier-Scalable Reconnaissance Mission Test Bed: Proof-of-Concept Multi-Rover Test Bed

Fink & Tarbell, LPSC 2008 ~3 kg payload!

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Rover “CYCLOPS” featured in Popular Science 2010 and on National Science Foundation 2009 main webpage

nsf.gov - National Science Foundation - US National Science Foundation (NSF) 10/28/09 19:58

NSF Web Site

nsf.gov - National Science Foundation (NSF) News - Caltech Scientists Cr…ons in Testing Visual Prostheses - US National Science Foundation (NSF) 10/28/09 19:58 AUGUST 10 2010 Volume 277 #2

CYCLOPS

A Legally Blind Robotic Guinea Pig for Testing Artiﬁcial Eyes John B. Carnett Funding Opportunities Special Notice

Find Funding Opportunities Birthplace NSF Information Related to the American Recovery and Reinvestment ActNSF Web Site California Institute of Technology --- Dr. Wolfgang Fink, Mark A. Tarbell Upcoming Due Dates of 2009 Get NSF Updates Occupation How to Prepare Your by Email Simulates the visual experience of a blind person outfitted with a retinal Building a Sustainable Energy Future: U.S. Actions for an Effective Energy implant Proposal Economy Transformation Site Features Why We Need It Funding Trends Most vision implants are still too crude to test on humans. Program Areas Latest News See All How It Works Select One Cyclops, a $20,000, four-wheeled rover, is the worldʼs first stand-in for the visually impaired, allowing researchers to test and refine image-processing "The Music Instinct" Wins International News software for prosthetic eyes on a robot instead of a person. Mounted to Quick Links Recognition Cyclopsʼs head is a remote-controlled camera that can pivot to capture the Select One same view as a patient with that particular prosthesis would. If the robot Released October 26, 2009 For the News Media canʼt tell the difference between a stairwell and a fireplace, researchers will Press Release SearchNews Funding Opportunities News From the Field know they need to refine their algorithms. Special Reports

On The Job By Caltech Scientists Create Robot Surrogate for This year. Seeing Previously Invisible Molecules for the First Discoveries from NSF Blind PersonsTime in Testing Visual ProsthesesResearch About NSF Released October 22, 2009 Research Overviews General Information About NSF Press Release

Merit Review October 19, 2009 Speeches & Lectures

Broadening Participation/ Diversity Ancient Lemurs Take Bite Out of Evolutionary Tree Multimedia Gallery ReleasedScientists October at the 21, California 2009 Institute of Technology have created a remote- ViewNews Staff Directory Presscontrolled Release robot that is able to simulate the "visual" experienceNSF & Congress of a blind Search Staff Directory person who has been implanted with a visual prosthesis, such as an News From the Field Classroom Resources artificial retina. An artificial retina consists of a silicon chip studded with a CareerFor the Opportunities News Media varying number of electrodes that directly stimulate retinalNSF-Wide nerve Investments cells. It Now Showing: Film,is TV,hoped Museums that this and approach More may one day giveSee blindAll persons the freedom Contracting Opportunities Science and Engineering Special Reports of independent mobility. Full Story Visit NSF NSF supports great television, inspiring museum exhibits, Statistics breathtaking IMAX films, and compelling radio. ResearchNSF Organizations Overviews Source Search NSF Awards The Botany of Desire Office of Equal Opportunity California InstituteFeaturing of Technology Michael Pollan and based on his best-selling Follow NSF ProgramsNSF-Wide Investments book, this special takes viewers on an eye-opening RSS NoSpeeches FEAR Act & DataLectures exploration of the human relationship with the plant world -- seen from the plants' point of view. No FEAR Act Notice The National Science Foundation (NSF) is an independent federal agency Facebookthat supports fundamental research and education across all fields of science and engineering. In fiscal InspectorNSF Current General Newsletter Hotline Twitter year (FY) 2009, its budget is $9.5 billion, which includes $3.0 billion provided through the Budget and Performance Events Calendar See All Multimedia Gallery American Recovery and Reinvestment Act. NSF funds reach all 50 statesNeed through Help? grants Citizens' Report to over 1,900 Alluniversities Events and institutions. Each year, NSF receives aboutHelp Center 44,400 RecoveryNews Archive Act competitive requests for funding, and makes over 11,500 new funding awards. NSF also Advisory Committee Meetings How Do I Find...? awards over $400 million in professional and service contracts yearly. News by Research Area http://www.nsf.gov/ Get News Updates by Email Page 1 of 2 Arctic & Antarctic Useful NSF Web Sites: © Copyright 1998-2013 Visual and Autonomous ExplorationAstronomy & SpaceSystems ResearchNSF Home Page: Laboratory http://www.nsf.gov at Caltech/University of Arizona NSF News: http://www.nsf.gov/news/ Biology For the News Media: http://www.nsf.gov/news/newsroom.jsp Science and Engineering Statistics: http://www.nsf.gov/statistics/ Chemistry & Materials Awards Searches: http://www.nsf.gov/awardsearch/ Computing

Earth & Environment

Education

Engineering

Mathematics

Nanoscience

People & Society

Physics

http://www.nsf.gov/news/news_summ.jsp?cntn_id=115841 Page 1 of 2 Tier-Scalable Reconnaissance Mission Test Bed: Surface Explorers: Rovers

Fink et al., IEEE Aerospace 2011 Rovers: • Electric motors • Metal chassis and sensor platform • General-purpose, high-performance Unix workstation • Range up to 10 km on one battery charge • Up to 10 hours onboard computing • Wireless Internet capability • > 30 kg payload sustained • Onboard GPS • Onboard HD camera

Fink et al., IEEE Aerospace 2012 • Catamaran design • Electric motors • Very stable • Air-based propulsion system • 1.8 m long by 1.5 m wide by 0.5 m tall • General-purpose, high-performance Unix workstation • Mass: ~45 kg without sensor payload • Onboard HD cameras • > 68 kg payload capability • Onboard side-scanning sonar • Highly modular design • Wireless Internet capability (Earth applications) • Onboard GPS (Earth applications)

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory Finkat Caltech/University et al., IEEE Aerospace of Arizona 2012 World-wide Commanding & Sensor Data Transmission Infrastructures

Worldwide Control wireless via Cloud Computing!

Fink et al., CMPB 2009; SPIE 2009

Controllable worldwide via iPhone (demonstrated at MacTech 2010 Conference)

A new field for PHM?!

© Copyright 1998-2013 Visual and Autonomous Exploration Systems Research Laboratory at Caltech/University of Arizona Automated Global Feature Analyzer (AGFA, Fink et al., 2008) Operational Level 1 Diagram

[MER Image Courtesy NASA]

Color Angularity

Albedo Texture/Vesicularity

AGFA

Fink, IEEE WCCI 2006

Pre Geologic Process Situation

Geologic Process, e.g., fluvial

Post Geologic Process Evidence as “seen” by the Rover

“Seen” as 3 different deposits?

1 2 3

If so, geologic history of this region may be unclear/random!

“Seen” as 2 different deposits?

1 2 1

If so, geologic process may have taken place in the past, e.g., flood!

Mars Titan

Wolfgang Fink, Ph.D.

Visual and Autonomous Exploration Systems Research Laboratory

Depts. of Electrical & Computer Engineering and Biomedical Engineering University of Arizona

and

Division of Physics, Mathematics & Astronomy California Institute of Technology

[email protected] http://autonomy.arizona.edu