Teleoperation and Visualization Interfaces for In‐Space Operations

Will Pryor, Balazs P. Vagvolgyi, Simon Leonard, Louis L. Whitcomb, and Peter Kazanzides Johns Hopkins University, Baltimore, MD, USA

Overview Teleoperation Interfaces

• Ground‐based teleoperation for on‐orbit satellite servicing is complicated by communication • Keyboard and GUI delay and non‐ideal camera views • Conventional NASA interface • We have evaluated teleoperation interfaces: • Visualization on multiple monitors • Direct teleoperation using da Vinci master console • 3D monitor or HMD possible • Programmed motion primitives, where a single motion command is issued via a GUI • Operator enters robot commands via • Model‐mediated teleoperation, where the operator’s interaction with a simulated keyboard environment is replicated remotely • Operator uses GUI to change • Interactive path planning, where the user operates a virtual robot, evaluates the motion, visualization configuration and then executes on the real robot

• da Vinci Master Console: • Stereo visualization • Operator uses master tool manipulators https://sspd.gsfc.nasa.gov/restore‐L.html (MTMs) to move robot • Operator uses MTMs or space mouse to change visualization configuration

Experimental Tasks: Cutting multi‐layer insulation (MLI) Visualization Augmented

RRM Task Board

Restore‐L • Virtual objects on real • Real camera images on • Virtual models (Landsat7 MLI hat) camera images virtual models

Scene Modeling and Registration

Physical setup Cutting tool • Manual construction • Virtual fixture plane for satellite surface Model‐Mediated Teleoperation • Virtual fixture line for cut path • Adjusted by operator or by sensor feedback

• Image survey • Manual feature identification on 2D images • Register to known geometry • Reconstruct unknown • User‐defined model with virtual fixtures on master and hybrid motion/force control on slave geometry • Registration update (Update Transform) based on actual motion profile • Task monitor based on friction model of MLI cutting task

Interactive Path Planning • Define a path as a series of waypoints, connected with collision‐aware path planning Publications • Path can be previewed, modified, and executed when operator is satisfied 1. “Augmented Reality Environment with Virtual Fixtures for Robotic Telemanipulation in Space”, IROS 2012. • Support multiple visualization devices, such as 2D and 3D monitors, da Vinci stereo console, 2. “Model‐Based Telerobotic Control with Virtual Fixtures For Satellite Servicing Tasks”, ICRA 2013. and HoloLens HMD 3. “Strategies and models for cutting satellite insulation in telerobotic servicing missions”, Haptics 2014. • Waypoints can be placed from multiple user input devices, including keyboard/mouse GUI 4. “Experimental Evaluation of Force Control for Virtual‐Fixture‐Assisted Teleoperation for On‐Orbit Manipulation of Satellite Thermal Blanket Insulation”, ICRA 2015. and da Vinci master manipulators 5. “Registration of planar virtual fixtures by using augmented reality with dynamic textures”, ICRA 2015. 6. “Preliminary Study of Virtual Nonholonomic Constraints for Time‐Delayed Teleoperation”, IROS 2015. 7. “Parameter Estimation and Anomaly Detection while Cutting Insulation during Telerobotic Satellite Servicing”, IROS 2015. 8. “Task Frame Estimation during Model‐Based Teleoperation for Satellite Servicing”, ICRA 2016. 9. “Augmented virtuality for model‐based teleoperation”, IROS 2017. 10. “Scene modeling and augmented virtuality interface for telerobotic satellite servicing,” ICRA 2018 (RA‐L). 11. “Experimental Evaluation of Teleoperation Interfaces for Cutting of Satellite Insulation”, ICRA 2019.

Acknowledgments • NASA NNG15CR66C

The GUI (left) offers one interface for defining a path plan, while the 2D Rviz visualization (right) offers one visualization interface. The • Brian Roberts and the Satellite Servicing Projects Division (SSPD) team at NASA Goddard operator can switch seamlessly between editing the plan with GUI controls and with an Rviz interface using Interactive Markers. Here Space Flight Center (SSPD) the operator is previewing a path (the semitransparent UR10 model) made up of two waypoints (shown as green end effector models). • The da Vinci Research Kit (dVRK) is supported by NSF NRI‐1637789