The Development of Hierarchical Knowledge in Robot Systems
Total Page:16
File Type:pdf, Size:1020Kb
THE DEVELOPMENT OF HIERARCHICAL KNOWLEDGE IN ROBOT SYSTEMS A Dissertation Presented by STEPHEN W. HART Submitted to the Graduate School of the University of Massachusetts Amherst in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY September 2009 Computer Science c Copyright by Stephen W. Hart 2009 All Rights Reserved THE DEVELOPMENT OF HIERARCHICAL KNOWLEDGE IN ROBOT SYSTEMS A Dissertation Presented by STEPHEN W. HART Approved as to style and content by: Roderic Grupen, Chair Andrew Barto, Member David Jensen, Member Rachel Keen, Member Andrew Barto, Department Chair Computer Science To R. Daneel Olivaw. ACKNOWLEDGMENTS This dissertation would not have been possible without the help and support of many people. Most of all, I would like to extend my gratitude to Rod Grupen for many years of inspiring work, our discussions, and his guidance. Without his sup- port and vision, I cannot imagine that the journey would have been as enormously enjoyable and rewarding as it turned out to be. I am very excited about what we discovered during my time at UMass, but there is much more to be done. I look forward to what comes next! In addition to providing professional inspiration, Rod was a great person to work with and for|creating a warm and encouraging labora- tory atmosphere, motivating us to stay in shape for his annual half-marathons, and ensuring a sufficient amount of cake at the weekly lab meetings. Thanks for all your support, Rod! I am very grateful to my thesis committee|Andy Barto, David Jensen, and Rachel Keen|for many encouraging and inspirational discussions. Their comments and feedback significantly contributed to the form of this document. I would especially like to thank Andy for organizing the Intrinsic Motivation seminar I attended a few years ago. This seminar helped focus the direction of my research and led me to pursue many of the topics explored in this dissertation. I would like to thank the many friends and colleagues who helped make my grad- uate years as enjoyable as they were. Thanks especially to Aron Culotta, Sarah Os- entoski, Ashvin Shah, Maryanne Olson, Trek Palmer, Ricky Chang, Heather Perkins, and Shiraj Sen for their friendship and kindness, and for their company during those many, many nights at the Moan & Dove. I would also like to extend my gratitude to v the members of the Laboratory for Perceptual Robotics and the Autonomous Learn- ing Laboratory for our discussions and collaborations|specifically, Emily Horrell, Rob Platt, Shichao Ou, Dirk Ruiken, George Konidaris, and John Sweeney. Priscilla Scott and Leeanne Leclerc deserve special acclimations for making sure all of those nasty administrative details got taken care of and for being patient with all of us flaky academics. My sincere appreciation goes out to my friends at WMUA, especially the members of the weekday morning Jazz Block|Glenn Siegel, Ron Freshley, Ken Irwin, and Michael Ehlers|and to Hank Berry. My years as the Thursday morning DJ of \Soul Station," provided me with a welcome and necessary outlet that helped balance my work and hobbies. The music I discovered during this period provided an additional level of creative inspiration that has, no doubt, profoundly shaped my being at its core. To all those who stood up for their convictions, ran against the grain, and followed their own paths|John Coltrane, Albert Ayler, Bob Dylan, Cecil Taylor, Peter Br¨otzmann,Robert Wyatt, and many others|the passion, creativity, and love that you brought (and bring) to your work is an inspiration to us all. Most of all I would like to thank my fiance´e,Meg, for her love, her kindness, and her patience|especially during these last few stressful months. As I anticipate the journey we are just beginning, I am filled with excitement more than words can express. I would also like to thank my parents, Bill and Jean, and my sister and brother-in-law, Catherine and Adam, for their support and their love as I chose to stay in school for one degree after another. Finally, special gratitude should be extended to Maximus Felinus and Doris Day for their uncanny tolerance. vi ABSTRACT THE DEVELOPMENT OF HIERARCHICAL KNOWLEDGE IN ROBOT SYSTEMS SEPTEMBER 2009 STEPHEN W. HART B.S., UNIVERSITY OF MASSACHUSETTS AMHERST M.S., UNIVERSITY OF MASSACHUSETTS AMHERST Ph.D., UNIVERSITY OF MASSACHUSETTS AMHERST Directed by: Professor Roderic Grupen This dissertation investigates two complementary ideas in the literature on ma- chine learning and robotics|those of embodiment and intrinsic motivation|to ad- dress a unified framework for skill learning and knowledge acquisition.\Embodied" systems make use of structure derived directly from sensory and motor configurations for learning behavior. Intrinsically motivated systems learn by searching for native, hedonic value through interaction with the world. Psychological theories of intrinsic motivation suggest that there exist internal drives favoring open-ended cognitive de- velopment and exploration. I argue that intrinsically motivated, embodied systems can learn generalizable skills, acquire control knowledge, and form an epistemological understanding of the world in terms of behavioral affordances. I propose that the development of behavior results from the assembly of an agent's sensory and motor resources into state and action spaces that can be explored au- vii tonomously. I introduce an intrinsic reward function that can lead to the open-ended learning of hierarchical behavior. This behavior is factored into declarative \recipes" for patterned activity and common sense procedural strategies for implementing them in a variety of run-time contexts. These skills form a categorical basis for the robot to interpret and model its world in terms of the behavior it affords. Experiments conducted on a bimanual robot illustrate a progression of cumulative manipulation behavior addressing manual and visual skills. Such accumulation of skill over the long-term by a single robot is a novel contribution that has yet to be demonstrated in the literature. viii TABLE OF CONTENTS Page ACKNOWLEDGMENTS ............................................. v ABSTRACT ......................................................... vii LIST OF TABLES ...................................................xiii LIST OF FIGURES.................................................. xiv CHAPTER 1. INTRODUCTION ................................................. 1 1.1 Motivation. 1 1.2 Approach . 2 1.3 Contributions . 4 1.4 Document Overview . 5 2. BACKGROUND .................................................. 8 2.1 Dynamical Systems . 8 2.1.1 Feedback Control . 9 2.1.2 Artificial Potential Fields . 10 2.1.3 Potential Fields for Discrete State/Action Spaces . 11 2.2 Dynamical Systems Theories of Development . 13 2.2.1 Epigenetic Development . 13 2.2.2 Dynamic Field Theory . 15 2.2.3 Composable Motion Primitives . 16 2.3 Developmental Robotics . 18 2.3.1 The Role of Embodiment . 19 2.3.2 Affordance Modeling . 21 ix 2.3.3 Intrinsic Motivation . 23 2.4 Discussion . 26 3. A COMBINATORIC BASIS FOR ROBOT CONTROL ........... 27 3.1 Artificial Potential Functions . 28 3.1.1 Quadratic Potential Functions . 30 3.1.2 Harmonic Functions for Collision-Free Motion . 31 3.1.3 Kinematic Conditioning Functions . 32 3.2 Sensory and Motor Signals . 36 3.2.1 Directly Measured Signals . 37 3.2.2 Kinematic Transformations . 39 3.2.3 Internal Models . 41 3.2.4 Typing . 41 3.3 Typed Control Expressions . 43 3.3.1 Primitive Control Actions . 43 3.3.2 Co-Articulation . 44 3.3.3 Examples . 46 3.3.3.1 Kinematically Conditioned Reaching . 46 3.3.3.2 Conditioning for Sensor Acuity. 49 3.4 Discussion . 53 4. SKILL LEARNING ............................................... 58 4.1 A State Representation for Dynamic Processes . 59 4.2 Affordance Discovery . 62 4.2.1 SearchTrack . 65 4.2.2 TactileProbe . 73 4.3 Hierarchical Composition . 77 4.3.1 ReachTouch . 78 4.3.2 VisualInspect . 81 4.3.3 BimanualTouch . 83 4.4 Discussion . 85 x 5. SKILL GENERALIZATION ...................................... 88 5.1 Background . 89 5.1.1 Piagetian Schema . 90 5.1.2 Computational Approaches . 90 5.2 Controller Abstraction . 91 5.2.1 Abstracting SearchTrack . 92 5.2.2 Abstracting TactileProbe . 94 5.3 Control Basis Schema . 98 5.3.1 ReachTouch Procedural Artifacts . 101 5.3.1.1 Approaches for Learning Handedness . 102 5.3.1.2 Comparison of Approaches . 106 5.3.1.3 Learning \Out of Reach" . 111 5.3.2 Hierarchical Generalizations . 112 5.4 Discussion . 114 6. WORLD MODELING ........................................... 115 6.1 Modeling Affordances . 116 6.2 Catalogs . 118 6.2.1 A Probabilistic Method for Exploring Catalogs . 118 6.2.2 Demonstration: Learning Three Catalogs . 121 6.2.2.1 Experimental Setup . 122 6.2.2.2 Results . 124 6.3 Model Exploration Programs . ..