Teleoperation and Visualization Interfaces for In-Space Operations Will Pryor, Balazs P. Vagvolgyi, Anton Deguet, Simon Leonard, Louis L. Whitcomb, and Peter Kazanzides Johns Hopkins University, Baltimore, MD, USA

Overview Teleoperation Interfaces • Ground-based teleoperation for on-orbit satellite servicing is complicated by communication delay and non-ideal camera views • We have evaluated teleoperation interfaces: • Direct teleoperation using da Vinci master console • Programmed motion primitives, where a single motion command is issued via a GUI • Model-mediated teleoperation, where the operator’s interaction with a simulated environment is replicated remotely • Interactive Planning and Supervised Execution (IPSE), where the user operates a virtual robot, evaluates the motion, and then supervises execution on the real robot

• Keyboard, mouse and GUI • da Vinci Master Console: • Most similar to NASA interface • Stereo visualization • Visualization on multiple monitors • Two 3D input devices (da Vinci MTMs) • 3D monitor or HMD possible Direct teleoperation Motion primitives Model-mediated teleoperation

Interactive Planning and Supervised Execution (IPSE)

Experimental Tasks: Cutting multi-layer insulation (MLI) Visualization Augmented

RRM Task Board

• Virtual objects on real • Real camera images on • Virtual models OSAM-1 (formerly Restore-L) camera images virtual models Landsat7 MLI hat Scene Modeling and Registration

• Manual construction Physical setup Cutting tool • Virtual fixture plane for satellite surface • Virtual fixture line for cut path Model-Mediated Teleoperation • Adjusted by operator or by sensor feedback

• Image survey • Manual feature identification on 2D images • Register to known geometry • Reconstruct unknown geometry

• User-defined model with virtual fixtures on master and hybrid motion/force control on slave • Registration update (Update Transform) based on actual motion profile • Task monitor based on friction model of MLI cutting task

Interactive Planning and Supervised Execution (IPSE) Publications • Define a path as a series of waypoints, connected with collision-aware path planning 1. “Augmented Reality Environment with Virtual Fixtures for Robotic Telemanipulation in Space”, IROS 2012. • Path can be previewed, modified, and executed when operator is satisfied 2. “Model-Based Telerobotic Control with Virtual Fixtures For Satellite Servicing Tasks”, ICRA 2013. 3. “Strategies and models for cutting satellite insulation in telerobotic servicing missions”, Haptics 2014. • Support multiple visualization devices, such as 2D and 3D monitors, da Vinci stereo console, 4. “Experimental Evaluation of Force Control for Virtual-Fixture-Assisted Teleoperation for On-Orbit and HoloLens HMD Manipulation of Satellite Thermal Blanket Insulation”, ICRA 2015. • Waypoints can be placed from multiple user input devices, including keyboard/mouse GUI 5. “Registration of planar virtual fixtures by using augmented reality with dynamic textures”, ICRA 2015. and da Vinci master manipulators 6. “Preliminary Study of Virtual Nonholonomic Constraints for Time-Delayed Teleoperation”, IROS 2015. 7. “Parameter Estimation and Anomaly Detection while Cutting Insulation during Telerobotic Satellite Servicing”, IROS 2015. 8. “Task Frame Estimation during Model-Based Teleoperation for Satellite Servicing”, ICRA 2016. 9. “Augmented virtuality for model-based teleoperation”, IROS 2017. 10. “Scene modeling and augmented virtuality interface for telerobotic satellite servicing,” IROS 2018 (RA-L). 11. “Experimental Evaluation of Teleoperation Interfaces for Cutting of Satellite Insulation”, ICRA 2019. 12. “Interactive Planning and Supervised Execution for High-Risk, High-Latency Teleoperation”, IROS 2020. 13. “Visual Monitoring and Servoing of a Cutting Blade during Telerobotic Satellite Servicing”, IROS 2020. Acknowledgments • NASA NNG15CR66C

The GUI (left) offers one interface for defining a path plan, while the 2D Rviz visualization (right) offers one visualization interface. The • Brian Roberts and the Exploration and In-Space Services (ExIS) Division at NASA Goddard operator can switch seamlessly between editing the plan with GUI controls and with an Rviz interface using Interactive Markers. Here Space Flight Center the operator is previewing a path (the semitransparent UR10 model) made up of two waypoints (shown as green end effector models). • The da Vinci Research Kit (dVRK) is supported by NSF NRI-1637789