Jack Lightholder

Machine Learning and Instrument Autonomy: Allowing Spacecraft To Do More, With Less

Jack Lightholder

This research was carried out by the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Round-Trip Human Com Delay Decision Reaction Time Time < Human Data Storage Analysis Time

Data Volume > Compute Bandwidth Power

Courtesy Dr. Lukas Mandrake, JPL ML ≠ Expert Replacement

• Eliminates drudgery

• Operates impossibly fast

• Focuses experts on interesting cases

• Enables larger human feats

Data Science asks: “Would you like to have the same output with 1 /6 the experts or x6 the output with your current experts?” Courtesy Dr. Lukas Mandrake, JPL Low Earth Orbit Examples Identified Clouds

Visual Salience: Identified areas of the image that differ from Preliminary Cloud Classification surrounding areas. results from EO-1

cloud cloud clear space clear limb /haze TextureCam: Pixel classification for cloud screening, downlink prioritization

[Chien et al. 2016 JAIS; Thompson et al., i-SAIRAS 2012; Wagstaff et al., GRL 2013; Bekker et al., Astrobiology 2014; Altinok et al. JFR 2015, Thompson et al. TGARS 2011] On-board Processing Science Investigation Observe, Thumbnails, transmit Manual Inquiry

Martian Orbit Unmapped / Ops Decision Support Changing Surface

Host of Scientists, Manual Selection Courtesy Dr. Lukas Mandrake, JPL Science Support Martian Orbit Data Mining Unmapped / changing surface

On-board Science Detect Transients, Ops Decision Support Summarize Content

Focus of Attention Tools Courtesy Dr. Lukas Mandrake, JPL Scene-Wide Terrain Landmark Labels Classification Identification

Courtesy Dr. Lukas Mandrake, JPL 8 Drs. Kiri Wagstaff Techniques Gary Doran • Salience Estimation Ravi Kiran • Created by Genetic Algorithm Lukas Mandrake • Finds optimal blend of leading Norbert Schorghofer techniques Alice Stanboli

• Landmark Classification Successfully ported to: • Naïve Bayes • Support Vector Machines • PDS / Planetary Image Atlas • Neural Network (deep learning) • IPEX: Atmel 400 MHz

Crater Impact ejecta

Salience Landmarks

Barchan dune Dark slope streak

Summarization

Landmark Classification 9 Drs. Alphan Altinok Brian Bue Alice Stanboli Kiri Wagstaff “Scalable Scene Analysis” System • Convolutional Neural Network • Implemented on PDS Atlas • Currently trained for Cassini & MSL Images

craters transients rings surface horizon clouds plume sky view starfield body types multiple objects phases artifact eclipse haze over exposure noise ripple camera distance 19 categories – 53 labels

10 Successfully ported to: TextureCam System • MSL VSTB Flight Testbed (RAD750) = ~100 HiRise • Random Forest based pixel classifier images/day • Extremely fast & parallelizable • EO-1: Mongoose V (M5) processor • IPEX: Atmel 400 MHz

cloud clear space

limb /haze

cloud

clear

IPEX Cube-Sat Feature Cima Lava Fields Collaboration MLIA & MV Identification & Cloud Mask

Drs. David Thompson, Alphan Altinok, Brian Bue, Gary Doran, Kiri Wagstaff 11 A Landed Mission Example Autonomous Exploration for Gathering Increased Science

• Target & Zap Rock ChemCam • Manually Scheduled Targets

• Round Trip Delays • Trouble hitting 1st time • Targeted science not possible right after drive

• Autonomy selects interesting targets • Refines targeting automatically

• ~30-100% additional ChemCam science targets on drive sols

Drs. Tara Estlin, Dan Gaines, Gary Doran, Raymond Francis, et al. 13 Mars Exploration Mars Science Rover (2009) Laboratory (2012)

Sol 1400

• Provides intelligent targeting and data acquisition by: – analyzing images of the rover scene

– identifying high-priority science targets (e.g., rocks)

Sol 1417 – taking follow-up imaging of these targets with no ground communication required [Estlin et al. 2011] A Deep Space Example Analyze data acquired onboard spacecraft and respond based on analysis

Spacecraft Points/ Slews Plan Data Execution Acquisition

Potential volatiles on Ceres

Data Re-Plan Analysis results Analysis in new imaging goal(s) jpl.nasa.gov Image Credit: NASA/JPL/Dawn Mission MSFC/JPL/LaRC/JSC/GSFC/NASA 6 Target

Reference stars

Target Detection and Approach Medium Field Imaging Close Proximity Imaging Ephemeris determination Shape, spin, and local environment Local scale morphology, terrain properties

Target Position Uncertainty Short Flyby Time Data Value Analysis and Sorting (<30 minutes)

Spacecraft Pointing and Uncertain Environment Short Time at Closest Approach Camera Limitations (<10 minutes)

Limited Downlink of 1 Kbps 7 Rosetta OSIRIS Narrow Angle Camera Detection of 2867 Steins Rosetta OSIRIS Narrow Angle Camera Detection of 2867 Steins Rosetta OSIRIS Narrow Angle Camera Detection of 2867 Steins New Horizons Long Range Reconnaissance Imager Detection of Pluto / Charon Image Credit: NASA, Cassini Mission Enceladus Comet 67P Io

Plumes gives scientists insights into the volatiles located throughout the solar system.

Unfortunately, they’re not scheduled. We have to react fast.

20 Image Credit: NASA, Cassini Mission (Left), ESA, Rosetta Mission (Center), NASA, New Horizons Mission(Right) Hartley 2 (EPOXI)

Edge Detection Convex Hull Thresholding

• Detects bright material beyond the limb

• Enables monitoring campaigns, target-relative data acquisition

• Detects most plumes with zero false positives Enceladus (Cassini)

[Thompson et al., PSS 2012, Wagstaff et al, ApJ 2014] 21 Target and plume detection using MRI-VIS on EPOXI. Tracking Comet Hartley 2 MOSAIC: Mars on-site shared analytics information and computing

Understand and maximize the effect of HPSC on Mars exploration

Goals: Research Tasks 1. Distributed computing for Mars • Resource-aware process scheduling across a network of agents 2. Quantify HPSC impact on missions • Model-based flight computing configuration for multi-processor / multi-robot systems 3. Explore trade space of HPSC designs • Optimize routing and storage of information across a network of agents • Extend Delay / Disruption tolerant networking for use in distributed systems

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC

1. Develop responsive, model-driven distributed computing stacks

1.1 Software Dispatch

1.2 Capability 1.3 Network Registry Status

2. Distributed World Model

4. PUFFER

Tasks • Benchmark existing flight software on a variety of computing hardware • Develop analytical models to estimate runtime, data, energy requirements as a function of HPSC config • Develop distributed process dispatcher (load balancing) based on above models • Develop distributed data product consensus over DTN

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC

1. Develop responsive, model-driven distributed computing stacks

Mars 2020 HPSC CubeSat

2 3

Working Example: • Can optimally solve Mars 2020 fast-traverse FSW allocation, given HPSC + network configuration • Output: minimum-cost allocation (time, power, etc) • See: “Dynamic Shared Computing Resources for Multi-Robot Mars Exploration” i-SAIRAS, 2018

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC

2. Understand impact of HPSC configurations and design on missions

Sand Trap

Locally shared mapping / Improved Path Planning Localization fix by navigation near computing resources sharing images

Tasks • Given HPSC configuration, solve optimal schedule (previous) to get runtime, data, energy requirements • Then, simulate effects on candidate missions

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC

Worked example for Mars 2020 rover mission

Jezero HiRise classifications

[1] Data-Driven Surface Traversability Analysis for Mars 2020 Landing Site Selection, Ono et. al. 4 hardware design points, path replayed in 3D

Mars 2020 is reaches its destination 19% sooner driving through Jezero crater when it has access to three or four cores of an HPSC, either onboard, or nearby with >=1 Mbps data rate. • Main gains are from better path optimization and better sensing • Secondary gains from decreased sensing and planning time required

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC: Mars On-Site Shared Analytics Information and Computing

Bandwidth X (Mbps)

What should this hardware look like? Given this mission

Methodology • Given prior models, iteratively “sample” HPSC / network config to evaluate metrics • Where possible, use “shadow cost” to determine choke point • (e.g., data transfer, communication bandwidth, onboard storage, or asymptotic runtime) • Not in isolation! Consider FSW algorithms, models of environment, etc.

Joshua Vander Hook (347N) [email protected] 818-354-5455 MOSAIC

3. Explore trade space of networked multi-processor configurations

4 Core Savings in Mars 2020 planning time (seconds) as Optimal Software Process Assignment: {not possible, Rover, Assisting CPU} 3 Core Gaain efunctiond Time as ofa F assistingunction of C CPUPU Fr espeedquency

10 Data Rate Avg time Plan Mbps sec Image DEM Analyze Terrain Path Drive 5 0.01 27 0 0 -1 0 0 0.1 27 0 0 -1 0 0

0 Time Saved 0 200 400 600 800 1000 1200 1400 1600 0.15 29.3 0 0 -1 0 1 Core 0.3 25.6 0 0 -1 0 -5 0.4 29.7 0 0 0 0 -10 2 0.9 28.2 0 0 0 0 Core 1 27.3 0 0 -15 100 15.3 0 0 Used actual RAD750 timings & HPSC predicted timings -20 Design Points (in order of coms bandwidth) -25 1. (rows 1-2) All onboard 2. (rows 3-4) Partially onboard -30 3. (rows 5-6) Partially onboard + Analyze Terrain 4. (rows 7-8) Maximally HPSC -35 Frequency of assisting CPU (max 1.6 Ghz)

From Mars 2020 analysis: • Main gains are parallelization (3,4 core is mostly level), even at low (8%) availability • Bottleneck is data rate, solution space is “level” w.r.t. compute

Joshua Vander Hook (347N) [email protected] 818-354-5455 NAVCAM ~Gbytes/Sol

O(1000) images/Sol)

Storage Gbytes ~ Tbytes

Downlink Engineering Data

0 - 100MBits/Sol (Fetch rover)

Masahiro Ono (347F) [email protected] 818-354-0930 34 Energy Optimal AutoNav Jezero Crater Preliminary Result Energy optimal vs time optimal • Used Jezero Crater’s DEM and terrain data • Simulation based on Fetch Rover design – Performed in collaboration with Austin Nicholas – Used Fetch Rover’s solar panel area, battery size, min charge level, nominal driving energy – Used MSL’s slip curve – Used MER/InSight’s dust accumulation model; assumed 100th Sol – Sun elevation > 10 deg – M2020 driving speed

Work by Kyon Otsu 35 New Way of Commanding AutoNav M2020: Command by waypoints MAARS: Command by costmap 34 33 33 32 31 30 30 31 32 33 35 34 34 33 32 31 31 32 33 34 36 35 35 34 33 32 32 33 34 35 37 36 36 35 34 33 33 34 35 36 38 37 37 36 35 34 34 35 36 37 39 38 38 37 36 35 35 36 37 38 40 39 39 38 37 36 36 37 38 39 40 39 38 37 37 38 39 40 39 38 38 39 40 41 40 39 39 40 41 42 47 46 44 43 41 40 40 41 42 43 47 46 44 43 41 41 41 42 43 44 48 47 45 44 42 42 42 43 44 45 47 45 44 43 43 43 43 44 45 46 48 46 45 44 44 44 44 45 46 47 49 47 46 45 45 45 45 46 47 48 50 48 47 46 46 46 46 47 48 49 51 49 48 47 47 47 47 48 49 50 52 50 49 48 48 48 48 49 50 51 53 51 50 49 49 49 49 50 51 52 • Uplink waypoint and KOZs only • Uplink global cost-to-go map • Plan min-time path to waypoint • Cost to the strategic goal from each cell Work by Kyon Otsu • Min local cost + global cost-to-go 36 Concurrent Path Planning & Scheduling

Sun simulation Mars

DEM (Mars24 by NASA GISS) Ground

Real-time execution on HPSC 4-D planning (x,y,time,battery) Uplink 34 33 33 32 31 30 30 31 32 33 35 34 34 33 32 31 31 32 33 34 36 35 35 34 33 32 32 33 34 35 Compressed34 33 33 32 31 30 30 31 32 33 37 36 36 35 34 33 33 34 35 36 35 34 34 33 32 31 31 32 33 34 38 37 37 36 35 34 34 35 36 37 36 35 35 34 33 32 32 33 34 35 39 38 38 37 364 353 353 362 371 380 30 31 32 33 37 36 36 35 34 33 33 34 35 36 40 costmaps39 39 38 375 364 364 373 382 391 31 32 33 34 38 37 37 36 35 34 34 35 36 37 40 39 386 375 375 384 393 4302 32 33 34 35 39 38 38 37 36 35 35 36 37 38 397 386 386 395 4304 4313 33 34 35 36 40 39 39 38 37 36 36 37 38 39 4308 397 397 4306 4315 4324 34 35 36 37 40 39 38 37 37 38 39 40 Bellman-Ford algorithm 39 38 38 37 36 35 35 36 37 38 47 46 44 43 41 40 40 41 42 43 39 38 38 39 40 41 47 46 44 43 410 4319 4319 4328 4337 4346 36 37 38 39 40 39 39 40 41 42 48 47 45 44 42 42 420 4339 4348 4357 37 38 39 40 47 46 44 43 41 40 40 41 42 43 47 45 44 43 43 43 43 44 4359 4368 38 39 40 41 47 46 44 43 41 41 41 42 43 44 48 46 45 44 44 44 44 45 460 4379 39 40 41 42 Few hours on desktop 48 47 45 44 42 42 42 43 44 45 49 47 46 45 457 456 454 463 471 480 40 41 42 43 47 45 44 43 43 43 43 44 45 46 50 48 47 46 467 46 464 473 481 491 41 42 43 44 48 46 45 44 44 44 44 45 46 47 51 49 48 47 478 47 475 484 492 5402 42 43 44 45 49 47 46 45 45 45 45 46 47 48 52 50 49 48 487 485 484 493 5403 5413 43 44 45 46 50 48 47 46 46 46 46 47 48 49 MTTT 48 46 45 44 44 44 44 45 46 47 53 51 50 49 49 49 49 50 51 52 51 49 48 47 47 47 47 48 49 50 49 47 46 45 45 45 45 46 47 48 52 50 49 48 48 48 48 49 50 51 Trade off: 50 48 47 46 46 46 46 47 48 49 53 51 50 49 49 49 49 50 51 52 (Mars 2020 Traversability 51 49 48 47 47 47 47 48 49 50 52 50 49 48 48 48 48 49 50 51 Analysis Tools) 53 51 50 49 49 49 49 50 51 52 Data size v.s. onboard computation, Heater model performance Terrain classification CFA, slip model Driving speed

37 Work by Kyon Otsu Provided by Austin Nicholas Jezero Crater Preliminary Planning Results Seasonal variation • Used Jezero Crater’s DEM and terrain data • Simulation based on Fetch Rover design – Performed in collaboration with Austin Nicholas – Used Fetch Rover’s solar panel area, battery size, min charge level, nominal driving energy – Used MSL’s slip curve – Used MER/InSight’s dust accumulation model; assumed 100th Sol – Sun elevation > 10 deg – M2020 driving speed • LS = 0.5 (sprint), 91 (summer), 189 (autumn), 274 (winter)

Work by Kyon Otsu 38 Vision-based Classification: Data Collection by Athena

Current sensor Camera image Camera Co-registration

Work by Federico Zechini, Jacek Sawoniewicz, Kyon Otsu 39 Energy-based Terrain Classification Clustering by HDP-HSMM algorithm

Uphill sand

Level bedrock

component of wavelet transformof componentwavelet

nd 2

1st component of wavelet transform Level sand Uphill bedrock Work by Federico Zechini 40 Single Wheel Testbed

Hard Soft

Motor current mA

s Work by Shoya Higa, Jacek Sawoniewicz 41 IR-based Terrain Classification: Proof-of-concept

Compact sand area Soft sand area

14 13 30 29 28 15 10 • Created two types of sandy area in Mars Yard: 12 11 27 26 25 8 7 23 22 9 24 4 • Compact (~80 kPa) and Soft (~30 kPa) 5 21 6 20 19 1 2 16 3 • Measured temperature and soil pressure at 30 locations 18 17 • Temperature was collected from 6:30am to 7 pm

Work by Yumi Iwashita 42 We have great new What compute, memory, power, capabilities! architecture do you need? Mission ML Systems Formulation Innovation

How much compute What new capabilities can we have? will this offer us?

What cache size, parallel architecture, Space and bit depth do you Computation We can build great need? space-based compute engines!

Bridging The Gap: Actual Performance Metrics

Machine Learning RAD HPSC EMU Capability SW Capabilities vs. (business as usual) (parallelized) (migrating threads) Onboard Requirements

Front Camera Rear Camera Side Camera

Sensors reporting Raw Pressure Grid Data • Context cameras • Pressure Grid • Force/Torque • Vertical Displacement Force (N) • Optical Flow • IMU (accelerations)

Torque (Nm) Sink (mm) jpl.nasa.gov