Telemanipulation with Chopsticks: Analyzing Human Factors in User Demonstrations Liyiming Ke1, Ajinkya Kamat1, Jingqiang Wang2, Tapomayukh Bhattacharjee1, Christoforos Mavrogiannis1, Siddhartha S. Srinivasa1 Abstract— Chopsticks constitute a simple yet versatile tool that humans have used for thousands of years to perform a variety of challenging tasks ranging from food manipulation to surgery. Applying such a simple tool in a diverse repertoire of scenarios requires significant adaptability. Towards devel- oping autonomous manipulators with comparable adaptability to humans, we study chopsticks-based manipulation to gain insights into human manipulation strategies. We conduct a within-subjects user study with 25 participants, evaluating three different data-collection methods: normal chopsticks, motion- captured chopsticks, and a novel chopstick telemanipulation interface. We analyze factors governing human performance across a variety of challenging chopstick-based grasping tasks. Although participants rated teleoperation as the least comfort- able and most difficult-to-use method, teleoperation enabled users to achieve the highest success rates on three out of five objects considered. Further, we notice that subjects quickly Fig. 1: Subjects teleoperating a robot holding chopsticks learned and adapted to the teleoperation interface. Finally, using a motion-capture-marker instrumented chopstick to while motion-captured chopsticks could provide a better reflec- tion of how humans use chopsticks, the teleoperation interface pick up different objects. can produce quality on-hardware demonstrations from which the robot can directly learn. size, weight distribution, and deformation. The design of I. INTRODUCTION chopsticks can be easily extended and its versatility may Roboticists have used both complex tools (such as uni- apply to other robots, e.g., a surgical robot with a chopsticks- versal grippers [1], vacuum suction tools [2], and anthropo- shaped end effector may effortlessly switch between grasp- morphic hands [3], [4]) and simple ones (such as parallel-jaw ing, moving and rotating various shapes of tissues with grippers and forks [5]–[13]) for manipulation. Complex tools different deformability without switching hardware [17]. can be customized for specialized manipulation tasks, while Importantly, human familiarity with chopsticks opens up simple tools require adaptive manipulation strategies for the possibility of easily collecting expert demonstrations. different usage scenarios. Studying the adaptability humans Despite the wide range of applications involving chopsticks, demonstrate in using simple tools could help us extract we are not aware of any prior efforts to learn from human insights for developing autonomous manipulators with com- demonstrations for chopstick-based robot manipulation tasks. parable flexibility to humans. To this end, we analyze how different interfaces and user In this paper, we focus on chopsticks, a simple tool that expertise levels affect the quality of demonstrations by com- many humans are already familiar with. Every day, billions paring three data-collection methods (Fig. 2a): normal chop- of people use chopsticks for food-related manipulation, i.e., sticks (“Chop”), motion captured chopsticks (“MoChop”), to pick up sushi, fried rice, or twirl noodles. One wily and a teleoperation interface (“TeleChop”). We conduct a and dexterous human even used chopsticks to pick-pocket within-subjects user study with 25 participants and examine cellphones [14]. Researchers have adopted the general de- how human factors affect the success rate of picking up sign of chopsticks for various applications, such as meal everyday-life objects. assistance [15], [16], surgery [17], micro-manipulation [18], One key finding is that user preferences for input modal- repetitive pick-and-place [19], [20] and articulated-hand de- ities may not necessarily correlate to the quality of the sign [21]–[23]. A chopsticks-wielding robot has a potentially resulting demonstrations. Although users rated teleoperation versatile use case to pick up objects of diverse shape, as unnatural and uncomfortable, they exhibited comparable 1Paul G. Allen School of Computer Science and Engi- performance using TeleChop, MoChop, and Chop. Surpris- neering, University of Washington, Seattle, WA 98105, USA ingly, TeleChop enabled users to score the highest success fkayke,arkamat,tapo,cmavro,[email protected] rates on picking up three of five objects even though it lacks 2Department of Mechanical Engineering, University of Washington, Seattle, WA 98105, USA [email protected] haptic feedback. Furthermore, since teleoperation moves See accompanying video for snippets of our human study. the robot to complete the task, it provides directly usable (a) Hardware (b) Motion capture (c) Teleoperation in action Fig. 2: Overview of our system. demonstrations for data-driven techniques such as imitation updates at 100Hz. From the tracked markers’ positions, we learning. extracted MoChop’s pose. Overall, we make the following contributions: 1) An analysis of human factors underlying user demon- B. TeleChop strations in chopstick-based telemanipulation. We custom built a 6-DOF robot manipulator, assembled 2) A comparative evaluation of three different technolo- from components provided by HEBI Robotics [26]. We gies for chopstick-based manipulation across a series attached the TeleChop to this robot’s end-effector. TeleChop of grasping tasks of varying difficulty: normal chop- has an actuated pair of chopsticks. The two chopsticks sticks, motion-captured chopsticks, and teleoperated operated on the same plane: one was fixed at the actuator’s chopsticks. body, and the other is attached at the actuator’s output shaft. 3) A novel teleoperation interface featuring a custom- We used HEBI’s X-Series actuators [26] since they provide built, inexpensive 6-DOF robot manipulator, equipped built-in controllers for position, velocity, or torque control, with a pair of chopsticks attached to its end-effector running at a loop rate of 1KHz. and the ability to partially filter vibrations arising from users’ grips or motion tracking. C. Teleoperation Interface Humans successfully teleoperated our robot to complete To initiate teleoperation, users held MoChop to match a challenging manipulation task: picking up a slippery glass TeleChop’s orientation. During teleoperation, users moved ball with slippery metal chopsticks, without haptic feedback. MoChop (leader) to teleoperate the robot mounted with Tele- We believe that teleoperation could be the preferred interface Chop (follower). This further constrains chopsticks’ move- to yield on-hardware demonstrations for robots to learn ment because of the motion-retargeting problem induced by from [24]. the difference between the human arm and the robot arm. We II. SYSTEM DESIGN designed a controller that guide TeleChop to smoothly mimic MoChop’s pose. Depending on how the user held MoChop, We designed a pair of chopsticks for motion-capture the two chopsticks may be on different planes. We projected (“MoChop”) and an interface for users to teleoperate a robot the two chopsticks in MoChop onto the same plane to ensure holding chopsticks (“TeleChop”). Both were adapted from TeleChop could follow the projected pose. This became the normal chopsticks (“Chop”). See Fig. 2a. All methods used target pose that our controller tracked. the same consumer-grade titanium chopsticks that differ only Our controller translated MoChop poses to joint com- in colors. mands for the robot. Upon receiving the desired pose, the A. MoChop controller first computed an Inverse Kinematics solution. However, since smooth chopstick trajectories for humans We 3D printed five light-weight, ball-shaped markers, could require jumps in the robot’s joint space, we applied wrapped them in reflective material and mounted them onto a convolution smoother to get the joint position command, MoChop. To ensure users could still hold the chopsticks, we effectively enabling tremor cancelling. We used a position placed the markers near the tips and tails of chopsticks at dif- PID controller to follow the joint position command. To add ferent positions on each stick. Note the markers changed the smoothness to the robot’s movement, we also supplied a weight distribution and added constraints to finger positions, torque command based on gravity compensation and passed e.g. users cannot hold the chopsticks at the tail. it through a torque PID controller. 1 We added both PID To track the motion of MoChop held by users, we used the controllers’ PWM outputs to compose the final command. OptiTrack motion capture system [25] with 11 cameras (Fig. 2b). The system uses optical reflection to track the position of 1The amount of torque commanded was based on the mass of the robot markers. Its tracking error is up to 0:4mm, and the tracking as specified on the manufacturer’s datasheet [26] Name Foam Chip Nut Pencil Ball L × W × H (mm) 50×25×20 65×45×15 25×10×3 86×7×6 14×14×14 Texture rough rough slippery slippery slippery Geometry curved curved flat long spherical Deformation compliant brittle hard hard hard TABLE I: Different objects to pick up. III. EXPERIMENTS three methods for grasping. Participants manipulated Chop We conducted an in-lab user study with human subjects and MoChop to directly pick up items; for TeleChop, they in which participants performed a series of pick-and-place held MoChop to teleoperate the robot, who was holding manipulation tasks using chopsticks on a variety of objects.