Reactive Manipulation with Contact Models and Tactile Feedback by Francois R. Hogan Submitted to the Department of Mechanical Engineering in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Mechanical Engineering at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY February 2020 @Massachusetts Institute of Technology 2020. All rights reserved. Author ....................... Signatureredacted Departme4t f Mechanical Engineering December 16, 2019 Signature redacted Certified by....................... ....... .... Alberto Rodriguez Associate Professor Chia . upervisor Signature redacted A ccep ted by ....................................... .. ................, * Accepted by Nicolas Hadjiconstantinou O N Department Graduate Officer FEB_052020 LIBRARIES Reactive Manipulation with Contact Models and Tactile Feedback by Francois R. Hogan Submitted to the Department of Mechanical Engineering on December 16, 2019, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Mechanical Engineering Abstract This thesis focuses on closing the loop in robotic manipulation, moving towards robots that can better perceive their environment and react to unforeseen situations. Hu- mans effectively process and react to information from visual and tactile sensing, however robots often remain programmed in an open-loop fashion, and struggle to correct their motion based on detected errors. We begin our work by developing full-state feedback controllers for dynamical sys- tems involving frictional contact interactions. Hybridness and underactuation are key characteristics of these systems that complicate the design of feedback controllers. We design and experimentally validate the controllers on a planar manipulation system where the purpose is to control the motion of a sliding object on a flat surface using a point robotic pusher. The pusher-slider is a simple dynamical system that retains many of the challenges that are typical of robotic manipulation tasks. We extend this work to partially observable systems, by developing closed-loop tactile controllers for dexterous manipulation with dual-arm robotic palms. We in- troduce Tactile Dexterity, an approach to dexterous manipulation that plans for robot/object interactions that render interpretable tactile information for control. Key to this formulation is the decomposition of manipulation plans into sequences of manipulation primitives with simple mechanics and efficient planners. Thesis Supervisor: Alberto Rodriguez Title: Associate Professor 3 I I I U U Acknowledgments First and foremost, I thank my advisor Alberto Rodriguez. His guidance along with the freedom to pursue my own ideas have been invaluable and been key to helping me become an independent researcher. In the years to come, I will always keep Alberto as a role model that I will draw inspiration from, in particular his care for his students, his attention to detail, and his kindness. I thank James Forbes, my advisor during my time as a Master's graduate student at McGill University, who is the reason I pursue academic research. My time under his guidance has taught me invaluable lessons: the importance of academic rigor, the process of scientific writing, and the value of creating knowledge. I am grateful to my collegues from the MCube lab for creating a stimulating environment that was enriching both personally and academically. I thank Peter Yu for sharing his passion of robotic systems and for patiently answering the questions of a motivated new robotic researcher. I thank Nikhil Chavan-Dafle for the many insightful discussions on the subtleties of the mechanics of manipulation. I am grateful to Nima Fazeli for his academic and career advice as well as being a valued friend during my time at MIT. I thank Maria Bauza, who's drive and work ethic have been conducive to an exciting research collaborative. Finally, I thank my collaborators, Jose Ballester, Siyuan Dong, Oleguer Canal, Eudald Romo Grau, and the members of the MCube lab, who have been instrumental in many of the results presented in this thesis. 5 6 Contents 1 Introduction 19 1.1 Contributions ... .... ..... ..... .... ..... ..... 20 1.2 Outline. .... ...... ...... ....... ...... ..... 21 2 Related Work 23 2.1 Nonprehensile Manipulation . .... .... .... .... .... .. 23 2.2 Contact-Constrainted Motion Planning ... ..... ..... .... 25 2.3 Hybrid Controller Design .. .... ..... ..... ..... .... 26 2.4 Grasp Planning .. ......... ........ ........ ... 27 2.5 Tactile Sensing ......... ............. ........ 28 3 Feedback Control with Contact Models 31 3.1 Contribution .... ........ ......... ........ ... 31 3.2 Introduction ...... ...... ...... ...... ...... .. 32 3.3 Challenges ..... ........ ......... ........ ... 33 3.3.1 Hybridness .. ........ ........ ....... ... 33 3.3.2 Underactuation ....... ........... ........ 35 3.4 Planar Manipulation ...................... ..... 35 3.4.1 Nomenclature ....................... .... 37 3.4.2 Modeling ..... ............ ........... 38 3.4.3 Frictional Contact Constraints .................. 43 3.4.4 Linearization .... ....................... 45 3.5 Controller Design ......... .................... 45 7 3.5.1 Hybrid Model Predictive Control ... .... 45 3.5.2 Mixed-Integer Quadratic Program (MPC-MIQP) 48 3.5.3 Family of Modes (MPC-FOM) ..... 50 3.5.4 Learned Mode Scheduling (MPC-LMS) . 53 3.6 Numerical Results ................ 55 3.6.1 Straight-Line Tracking Simulation . .. 57 3.6.2 Sensitivity to initial state errors . .. 61 3.6.3 Sensitivity to contact mode errors . .. 62 3.7 Experimental Results .............. 64 3.7.1 Case Study A: Single Point Pushing .. 64 3.7.2 Case Study B: Pushing with Line Contact 66 3.8 Influence of Control Parameters ........ 68 3.8.1 Controller Frequency ......... 69 3.8.2 Tracking Velocity ............ 70 3.8.3 Planning horizon ............ 70 3.8.4 Coefficient of Friction ...... .... 70 3.8.5 Race Track Radius of Curvature .... 71 3.9 D iscussion .................... 71 4 Tactile Dexterity 73 4.1 Contribution .................... 73 4.2 Introduction ............ ........ 74 4.3 Approach ..................... 76 4.4 Manipulation Primitives ............. 78 4.4.1 G rasp ................... 78 4.4.2 P ush .................... 78 4.4.3 P ull .................... 79 4.4.4 Pivot ................... 79 4.5 M echanics .................... 79 4.6 Tactile Control ................. 81 4.6.1 Contact State Control ............. ......... 82 4.6.2 Object State Control . ............ .......... 85 4.7 Planning .... ......... ........ ........ ..... 86 4.7.1 Low-Level Trajectory Planning ................. 87 4.7.2 High-Level Planning .............. ......... 90 4.8 R esults ............ ............ ........... 95 4.9 Discussion ............ ............. ........ 96 5 Conclusion 97 5.1 Key Findings . ......................... ..... 97 5.2 Limitations ........... ............. ........ 98 5.3 Future Work . ............. ............ ...... 99 5.4 Closing Thoughts ... ............. ............ 100 5.4.1 What is the right tradeoff between model complexity and com- putational efficiency? . 100 5.4.2 Combining Model-Based and Data-Driven Approaches? .... 101 5.4.3 Towards Achieving General Manipulation Capabilities ..... 102 A Data-Driven Control of Planar Manipulation 105 9 10 List of Figures 3-1 The hand manipulates a light bulb into its socket. The dynamics of the light bulb are dictated by rigid body motions under applied external forces arising from contact interactions with the environment. ..... 32 3-2 Depiction of hybridness. Animation of a simple manipulation task that exploits multiple contact modes. First, the hand sticks to the book and drags it backwards exploiting friction. Second, thumb and fingers slide to perform a regrasp maneuver. Finally, the book is retrieved from the shelf using a stable grasp. ....................... 34 3-3 Depiction of underactuation. Where we interact with an object through contact, we can only transfer a limited set of forces. When pushing a coffee mug with a finger, the finger can only push on the object and cannot pull. Underactuation constrains the possible motions of the cup that can be impressed by the finger. .............. .... 35 3-4 Planar manipulation setup. The goal is to control the motion of the object on a flat surface using a velocity-controlled robotic pusher. The pose of the object is tracked using a Vicon camera system. ...... 36 3-5 Free body diagram of a sliding object with C = 2 contact points... 38 3-6 Depiction of the limit surface. The limit surface (ellipsoid in the figure) describes the set of forces and moments that can be transmitted by a patch contact. By the principle of maximal dissipation, the object twist is perpendicular to the limit surface of the applied frictional wrench w. 40 3-7 Friction cone constraint. The applied force must remain within the blue shaded region. ............... ............. 41 11 3-8 Mode dependent constraints following Coulomb's frictional interaction law. (Force constraint). .... ........ ....... ...... 41 3-9 Tree of optimization programs for a MPC program with N prediction steps. Scales exponentially due to contact hybridness. ........ 46 3-10 Hybrid MPC framework. A sequence of control inputs is computed that will drive the predicted states to the reference trajectory while simultaneously finding the schedule of optimal hybrid mode transitions m = {mi, .. , mM}. The control input io + u* is applied to the system. 48 3-11 Block diagram of the hybrid controller design