How Sensors are Literally and Figuratively Driving Mobile Robots

Timothy D Barfoot Canada Research Chair (Tier II) in Autonomous Space Associate Professor

ECE Workshop on Sensors for the Networked World November 21, 2014 Source: Google

2 Outline

• history of sensors in mobile robotics • early mobile robots • cameras • sonar • 2D laser ranging • 3D laser ranging

• examples of what on-board sensors let us do today • mapping • autonomous route following

Montreal 2014 3 Robot

Perception ? Action

World

4 Senseless Mobile Robots

Homer (Iliad, 800 BC)

Hero of Alexandria (70 AD)

Tesla radio-controlled boat (1898) Da Vinci (1478)

5 Light/Touch Sensors (late 1940s to early 1960s)

Beast Mod II Beast Mod I

Elmer and Elsie (Machina Speculatrix; 1948-9)

Beast Mod II (Johns Hopkins University, Applied Physics Lab; 1961-3)

6 Cameras (1960s to 1970s)

Shakey (Stanford Research Institute; 1966-72)

Autonomous Stanford Cart, 1979

Stanford Cart, 1979 Moravec with Cart

Shakey in ‘blocks world’ 7 early 1980s Sonar Rangefinder (1980s)

H Moravec and A E Elfes, "High Resolution Maps from Wide Angle Sonar," Proceedings of the 1985 IEEE International Conference on Robotics and Hans Moravec circa 1980 with two sonar-equipped robots Automation, March, 1985, pp. 116 - 121.

these classic Polaroid sonar sensors could detect range to objects in a 30-degree cone

CMU evidence grid circa 1990 built from sonar data 8 2D Scanning Laser Rangefinder (1980s)

DARPA ALV (CMU and SRI)

9 Cameras (mid 1980s to late 1990s)

CMU’s in 1997 VaMoRs, VaMP, Vita-2 (Bundewehr University, ; 1995)

Navlab 5 Navlab 1 Navlab 2

10 2D Scanning Laser Rangefinder (mid 1990s)

SICK intended laser for safety Marshall et al.: Autonomous Underground Tramming for Center-Articulated Vehicles • 401

classic combination of Pioneer and SICK laser rangefinder

Figure 1. 10-t-capacity Atlas Copco ST1010c LHD, with sensor layout.

obvious motivations to seek an autonomous tram- erating mines. An infrastructureless system is clearly ming (a.k.a. autotramming) solution. Yet several preferable but must easily allow for new routes to be factors make infrastructureless autotramming a added as the mine advances. challenging task. One aspect is the large inertia Some have argued strongly against the feasibility and characteristic center-articulated/hydraulically of approaches requiring precise pose estimation for actuated steering mechanism, which makes these mining vehicle automation, suggesting instead that vehicles difficult to control at high speeds (unlike “reactive” methods are preferable (Roberts, Duff, & laboratory robots that often behave approximately Corke, 2002, p. 131). However, in this paper we de- as kinematic systems). In this paper, we describe a scribe a robust localization method that draws on control architecture that effectively handles these current techniques from mobile robotics. Our tech- substantive dynamics. nique fuses laser range finder data and data from Moreover, a most notable challenge is the prob- odometric sensors to determine the position and ori- lem of precise and real-time underground local- entation of the vehicle with respect to a sequence of ization. A navigation system that requires the self-generated local metric maps of the underground installation and registration of fixed infrastructure for environment. Our approach contrasts existing sys- Rao-Blackwellized particle filterlocalization for would be costly and susceptible to the tems that either require infrastructure or often em- simultaneous localization and mappingoften harsh environmental conditions found in op- ploy topological methods and/or reactive algorithms (courtesy Cyrill Stachniss) mining automation (2006) 11 Global Positioning System (2000)

US Military ended selective availability in 2000, suddenly making it accurate enough for vehicle positioning

GPS constellation

12 Mars Exploration Rovers (2004)

Actual tracks showing autonomous avoidance of rocks on Mars 13 1st DARPA Grand Challenge (2004)

Grand Challenge Map

CMU’s Sandstorm Sandstorm has a setback before the race

14 2nd DARPA Grand Challenge (2005)

Beer Bottle Pass, on the 2nd Grand Challenge

Stanford’s

15 DARPA Urban Grand Challenge (2007)

MIT’s TALOS

CMU’s Boss Stanford’s Junior

16 3D Scanning Laser Rangefinder (2007)

17 Look Ma, No Lasers!

Source: Mercedes Benz

18 Q: How did mobile robots get to here?

A: Has a lot to do with sensors.

19 Common Sensors

stereo camera RGB-D camera 3D laser global inertial flash rangefinder positioning measurement system unit

omnidirectional compass 2D laser monocular camera inclinometer rangefinder camera

encoder infrared sun sensor sonar switch rangefinder (also star tracker) 20 State-of-the-Art Mapping

Zlot and Bosse, CSIRO, 2011

21 Autonomous Space Robotics Lab (ASRL)

safety and security

mining planetary rovers

22 Stereo Visual Odometry Pipeline

Devon Island 2008 Devon Island 2008

Previous frame Right Image Keypoint de-warp and image detection rectification Nonlinear Stereo Keypoint Outlier Pose numerical matching tracking rejection estimate Left Image Keypoint solution de-warp and image rectification detection • Moravec (1980), Matthies (1987) and extended by many others 23 Teach Phase

manual teach mobile robot relative map vehicle vehicle (reference) commands wheeled state chassis (the plant)

motion images stereo camera estimate + (sensor) relative mapper

Mistastin 2011 24 Repeat Phase

autonomous repeat mobile robot relative map vehicle vehicle (reference) path tracker + commands wheeled state safety checks chassis (controller) (the plant) pose relative to path

path localizer images stereo camera (state (sensor) estimator)

Furgale P T and Barfoot T D. “Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain”, ICRA 2010 Kuka Service Robotics Best Paper Award

Furgale P T and Barfoot T D. “Visual Teach and Repeat for Long-Range Rover Autonomy”, JFR 2010

25 Devon Island 2009 Visual Route Following (UofT, Aerospace)

Montreal 2012

26 Visual Route Following (UofT, Aerospace)

UTIAS 2014 Tim Barfoot [email protected] http://asrl.utias.utoronto.ca 27 Important Dates

Paper Submission: Feb 13, 2015 Accept. Notification: Mar 20, 2015 Revised Papers Due: Apr 17, 2015

th Organizers 10 Conference on Field and Service Robotics (FSR) June 24-26, 2015, Toronto, Canada General Chair: Tim Barfoot (UofToronto) Program Chair: David Wettergreen (CMU) FirstCall Callfor Papersfor Papers Local Arrange.: Jon Kelly (UofToronto)

We invite papers describing original work in a number of technical areas related to field and service robotics including (but not limited to): Book and Journal Special Issue

Mining, Agriculture and Forestry Robotics Intelligent Cars A book version of the Conference Construction Robots Cleaning, Floor and Lawn Care Robotics Proceedings will be published by Springer- Search and Rescue Robotics Security Robotics Verlag in their STAR series and selected Autonomous Vehicles for Land, Air and Sea Medical and Healthcare Robotics field robotics papers will be considered for Planetary Exploration Robots Entertainment Robotics publication in a Special Issue of the Journal Environmental Robotics Robots for Cultural Heritage of Field Robotics.

Sponsors !

fsr.utias.utoronto.ca!