NRI Collaborative Research: Human-Centered Modeling and Control of Cooperative Manipulation with Bimanual Robots Ruzena Bajcsy, Human-Assistive Robotic Technologies (HART) Lab, University of California, Berkeley Oussama Khatib, Stanford Lab, Computer Science Department,

Abstract Berkeley/Stanford Project Collaboration Overview Stanford UC Berkeley In physical human-robot collaboration, the robot can use knowledge of its human partner’s phys- iological limitations and ergonomic preferenc- es to plan safer, more comfortable actions that more easily accomodate the unpredictability of hu- man motion. Our work addresses two crucial steps in endowing co-robots with this knowledge: MRI Ultrasound Intent Estimation Action Primitives Hybrid Control • Development of personalized musculoskele- tal models that estimate human dynamics in re- al-time Motion Capture • Development of a unified framework that inte- grates human data into a more traditional robot- ic control architecture iber Model Dynamic Model EM Although we focus on cooperative manipulation, the results of this research will be applicable to oth- Human Musculoskeletal Modeling Cooperative Manipulation Non-Invasive Sensing of Human Motion er areas, such human-robot cooperation for man- Major scientific challenges of this project: (1) Building subject-specific musculoskeletal models for real-time use. (2) Predicting human state with non-invasive sensor obser- ufacturing, robotic assistance with exoskeletons, vations. (3) Estimating human intent from the model state. (4) Decomposing the manipulation task into action primitives to simplify programming and control. (5) Modeling and rehabilitation robotics. human-robot cooperative manipulation as a hybrid system with provable safety and stability.

MRI-Based Musculoskeletal Models Objective Model Comparison with Existing Models Develop a data-driven subject-specif- Our MRI-based subject-specific model (A) offers ic musculoskeletal model that predicts an unprecedented level of kinematic detail over human kinematics and dynamics in re- state-of-the-art canonical models (B). al-time. A B Model Requirements: 246 actuators/arm • Capture physiological differences 50 actuators/arm between subjects 1. 2. 3. • Extract parameters solely from med-

20 ical imaging data 0.02

0 • Compute muscle dynamics in re- 10 −0.02

al-time 0 Z (m ) −0.04 y −0.06

• Integrate with traditional robot con- -10 −0.08 0.1

0.05 −0.02 trol methods −0.04 −0.06 -20 0 −0.08 Y (m) X (m) 4. 5. 6. We present a set of computational tools -30 -30 -20 -10 0102030 x to create a family of subject-specific Shoulder models automatically from magnetic Our novel muscle fiber generation algorithm maps volumetric muscles into muscle fiber-group actuators. The final piece-wise resonance imaging (MRI) data. linear actuators closely capture muscle physiology. The fiber generation can be varied parameterically to study how modeling accuracy and detail affect dynamics. Elbow Human-Robot Cooperative Manipulation Future Work Objective Approach In the coming year, we will demonstrate our algo- rithms on cooperative manipulation tasks between Develop a human-centered control a human and bimanual robot. Next steps involve: strategy that optimizes safety and com- fort in cooperative manipulation tasks. • Verification of the musculoskeletal dynamic model Challenges: • Unification of musculoskeletal and robot dy- • Synthesize human motion from task namics to allow us to better control internal trajectories forces during the cooperative manipulation of • Unify musculoskeletal dynamics rigid objects with robot dynamics • Identification and specification of action prim- • Proactively plan trajectories to mini- itives involved in cooperative manipulation mize human strain in real-time such as hand control, gaze control, maintaining • Avoid physiological constraints balance, etc. We present a simulation framework that combines human motion reconstruc- Through marker space motion synthesis techniques, we can reconstruct human motion given reference task trajectories. The tion, musculoskeletal control, and ro- next step is using the dynamic estimates of the musculoskeletal model to optimize task trajectories for the comfort and safety of bot control in real-time. the human in real-time.