Robot Systems Integration

Home , Robot kinematics, Visual servoing

Transformative Research and Robotics

Kazuhiro Kosuge Distinguished Professor Department of Robotics Tohoku University 2020 IEEE Vice President-elect for Technical Activities IEEE Fellow, JSME Fellow, SICE Fellow, RSJ Fellow, JSAE Fellow My brief history • March 1978 Bachelor of Engineering, Department of Control Engineering, Tokyo Institute of Technology • Marcy 1980 Master of Engineering, Department of Control Engineering, Tokyo Institute of Technology • April 1980 Research staff, Department of Production Engineering Nippondenso (Denso) Corporation • October 1982 Research Associate, Tokyo Institute of Technology • July 1988 Dr. of Engineering, Tokyo Institute of Technology • September 1989 - August 1990 Visiting Research Scientist, Department of Mechanical Engineering, Massachusetts Institute of Technology • September 1990 Associate Professor, Faculty of Engineering, Nagoya University • March 1995 Professor, School of Engineering, Tohoku University • April 1997 Professor, Graduate School of Engineering, Tohoku University • December 2018 Tohoku University Distinguished Professor 略歴

• 1978年3月東京工業大学工学部制御工学科卒業 • 1980年3月東京工業大学大学院理工学研究科修士課程修了(制御工学専攻，工学修士) • 1980年４月日本電装株式会社（現株式会社デンソー） • 1982年10月東京工業大学工学部制御工学科助手（工学部） • 1988年7月東京工業大学大学院理工学研究科工学博士(制御工学専攻) • 1989年9月-1990年8月米国マサチューセッツ工科大学機械工学科客員研究員 (Visiting Research Scientist, Department of Mechanical Engineering, Massachusetts Institute of Technology) • 1990年 9月名古屋大学助教授（工学部） • 1995年 3月東北大学教授（工学部） • 1997年 4月東北大学教授（工学研究科）大学院重点化による配置換 • 2018年12月東北大学 Distinguished Professor My another history

• Select Fellow, Center for Research and Development Strategy, Japan Science and Technology Agency, FY2005 - FY2011 • Review Board Member, PE7, ERC Advance Grant, 2008, 2010, 2012, 2014, 2019 • Senior Program Officer JSPA • Senior Program Office, Research Center for Science Systems, Japan Society of Promotion of Science, FY2007 - FY2009 • Science Officer, Research Promotion Bureau, Ministry of Education, Culture, Sports, Science and Technology, FY2010 - FY2013 • President, IEEE Robotics and Automation Society, FY2010 - FY2011 • Director & Delegate, Division X, IEEE Board of Directors, FY2015 - FY 2016 • 2020 IEEE Vice President for Technical Activities，FY2020 もう一つの略歴

• 科学技術振興機構, 研究開発戦略センター特任フェロー，FY2005 - FY2011 • Review Board Member, PE7, FP7，ERC Advance Grant, 2008, 2010, 2012, 2014 • 日本学術振興会学術システム研究センター主任研究員, FY2007 - FY2009 • 文部科学省研究振興局科学官 FY2010 - FY2013 • President, IEEE Robotics and Automation Society, FY2010 - FY2011 • Director & Delegate, Division X, IEEE Board of Directors, FY2015 - FY 2016 – Member, IEEE Public Visibility Committee, FY2015 - FY2016 – Member, IEEE TAB Nominations and Appointments Committee, FY2015 - FY2016 – Member, IEEE Ad Hoc Committee on Strategic Planning, FY2015, FY2016 • 2020 IEEE Vice President for Technical Activities，FY2020 Outline

• Some of my research in robotics • Robot Systems Integration • Physical Human-Robot Interaction – Human robot collaboration through interaction – Co-worker robot • Universal Manipulation – Issues and visual servoing for program-free robot • Conclusions Impedance Controller Design Based on Virtual Internal Model Cooperation of Humans for Handling an Object

Coordination of dualCoordination arms of both arms Coordination of Manipulators

Single-Master Multi-Slaves System (1989) K. Kosuge, J. Ishikawa, K. Furuta, M. Sakai, “Control of Single-Master Multi-Slave Manipulator Using VIM,” Proceedings of the 1990 IEEE International Conference on Robotics and Automation, 1990, 1172-1177. Coordination of Manipulators

K. Kosuge, H. Yoshida, T. Fukuda, Masaru Sakai, K. Kanitani, K. Hariki, ”Unified Control for Dynamic Cooperative Manipulation”, Proceedings of the 1994 IEEE/RSJ International Workshop on Intelligent Robotics and Systems, 1994, 1042-1047. Assembly of Two Parts (1994)

Ordinary bilateral feedback Passivity-based realization of bilateral feedback Segment Assembly System (1996)

Segment Assembly System for Tunnel Shield Machine

K. Kosuge, K. Takeo, D. Taguchi, T. Fukuda, H. Murakami, “Task-Oriented Force Control of Parallel Link Robot for the Assembly of Segments of a Shield Tunnel Excavation System,” IEEE/ASME Transactions on Mechatronics, 1 (3), (1996), 250-258. Parts-mating Theory (2001)

K. Kosuge, M. Shimizu, “Planar Parts-mating Using Structured Compliance,” Proceedings of the 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems, (2001), 1477-1482. M. Shimizu, K. Kosuge, “An Admittance Design Method for General Spatial Parts Mating,” Proceedings of the 2004 IEEE International Conference on Robotics and Automation, (2004), 3571-3576. Robot System for Dish Washing Machine (2009.3.)

K. Kosuge, Y. Hirata, J. Lee, A. Kawamura, K. Hashimoto, S. Kagami, Y. Hayashi, N. Yokoshima, H. Miyazawa, R. Teranaka, Y. Natsuizaka, K. Sakai, “Development of an Automatic Dishwashing Robot System,” Proceedings of the 2009 International Conference on Mechatronics and Automation, (2009), 43-48. Cooperation of Humans for Handling an Object

Coordination of multiple humans Multiple Mobile Manipulator Coordination (2001)

Y. Kume, Y. Hirata, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Handling a Single Object in Coordination”, Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2002, 2758-2763. Y. Hirata, Y. Kume, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Based on Virtual 3-D Caster Motion for Handling an Object in Cooperation with a Human”, Proceedings of the 2003 IEEE International Conference on Robotics and Automation, 2003, 938-943. Cooperation of Mobile Dual Manipulators (2003)

Y. Hirata, Y. Kume, T. Sawada, Z. D. Wang, K. Kosuge, ”Handling of an Object by Multiple Mobile Manipulators in Coordination based on Caster-like Dynamics”, Proceedings of the 2004 IEEE International Conference on Robotics and Automation, 2004, 807-812. Mechanical Parking Systems

Elevator Parking Systems Convey Parking Systems Shuttle Parking Systems Mechanical Parking Systems

Users are required to position their cars in a narrow space. Mechanical Parking Systems

• A parking system is required to have a caretaker. • Each driver is required to park his/her car in a narrow space precisely, which is not easy for a novice driver. iCART Concept

Intelligent Cooperative Autonomous Robot Transporters iCART (intelligent Cooperative Autonomous Robot Transporters)

M. Endo, K. Hirose, Y. Hirata, K. Kosuge, T. Kanbayashi, M. Oomoto, K. Akune, H. Arai, H. Shinoduka, K. Suzuki, “A Car Transportation System by Multiple Mobile Robots -iCART-”, Proceedings of 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008, 2795-2801. Demonstration of iCARTII Concept

Koshi Kashiwazaki, Kazuhiro Kosuge, Yasuhisa Hirata, Yusuke Sugahara, Takashi Kanbayashi, Koki Suzuki, Kazunori Murakami and Kenichi Nakamura, “Cooperative Transportation Control in Consideration of not only Internal Force but also External Force Applied to “MRWheel,” Proceedings of the 2012 IEEE International Conference on Robotics and Biomimetics, (2012), 1867-1873. in Germany

https://www.youtube.com/watch?v=Gnypt72F20Q Outline

• Societal Level Societal Values

• Service Level – Service enablers Services

• Fundamental Technologies Level Foundations

CRDS, JST, 2009, Modified by Kosuge, August, 2011 Societal Values

• For Individuals ★Quality of Life • For Communities ★Industrial Competitiveness – For Families – For Industries – For Local Government – For Nations • For the Globe ★Global Issues

CRDS, JST, 2009, Modified by Kosuge, August, 2011 Challenges and Opportunities in Robotics

Social Value Global Level Community Level Quality of Life

•Government orientedRobotics Service/Application •Environmental Monitoring •Utilities •Medicine •Security Service •Natural Resources Exploration •Retailer/Wholesaler •Therapy •Mobility •Agriculture •Transportation •Daily Life Assist •Shopping and Development •Forestry •Communication •Healthcare •Hobby •Space Exploration •Fishery Services •Service Industries •Rehabilitation •Entertainment •Mining •Deep Undersea and Underground •Medicine •Mental care •Sports •Manufacturing Exploration •Education •Learning •Comfort Life •Construction •Anti-terrorism ･Rescue Operation •Research and •Child care •Watch •Wastes Treatment/ Development •Housekeeping •Communication •Prevention of Infectious Diseases Management

•Cyborg (Cybernetic organism) •Software framework - •Stochasticity in Robotics •Social Concerns Emerging •Performance evaluation and Benchmarking •Functional Safety •Ambient intelligence •Nano-micro Robotics Technology •Autonomous Robots •Human Modeling Foundations Robotics •Teleoperation •Wearable Technology •Robotic Emotion (artificial emotion) •Service Contents Design

•Robot Systems Integration •Robot Kinematics and Dynamics •Human Robot Interaction •Manipulation Fundamental •Real-world Real-time Intelligence •Mobility •Spatio-temporal System Design •Actuation •Sensing and Machine Cognition •Physics-based Control

CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Unit Technologies Technical Issues

Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New ・・・ Services - Foundations Robotics

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Design/IdentifyRobot Systems a service/services Integration Unit Technologies necessary for the application as a Technical Issues sustainable business. Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New ・・・ Services - Foundations Robotics

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Design a robotUnit Technologies system architecture for the serviveTechnical/services Issues with necessary unit technologies Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New ・・・ Services - Foundations Robotics

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Unit Technologies Technical Issues

Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application

New New Enhance unit technologies to meet ・・・

Services the requirements for the service/services. - Foundations Robotics

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Unit Technologies Technical Issues

Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New Develop new fundamentals ・・・ necessary forServices the service/services. - Foundations Robotics

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Unit Technologies Technical Issues

Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New ・・・ Services -

Integrate the unit technologies and Foundations Robotics create the robot.

Robotics Foundations

Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 Robot Systems Integration Unit Technologies Technical Issues

Required Services

Domain ① Domain ② Domain ③ Applications/Services Elderly Care Agriculture Medicine orientedRobotics Service/Application New New ・・・ Services - Foundations Robotics

Robotics FoundationsEnrich robotics foundations through application-oriented research Current Robot Function CRDS, JST, 2009, Modified by Kosuge, August, 2011 System robotics is a new field of robotics dealing with robot-related issues in real environments. Systems Robotics Several prototypes of real world robots have been designed and developed based on robot technologies developed in our laboratory.

Walking Helper Power Assisted Chair Cycle

Assistive Robotics

Intention Recognition/Transfer

Intelligent Car Transportation Robot iCARTiCARTand ConceptiCART II Intelligent Car Autonomous-Robot- Transporters

Universal Robot Hand uGRIPP with Two-degrees of Freedom Robot Co-worker “PaDY” (in-time Parts/tools Delivery robot) Assembly and Manipulauion by Dual Manipulators

Stable Power Human Robot Integration of Visual and Impedance Servo Augmentation Coordination Mobile Manipulators Human-Robot Interaction Universal Manipulation Multiple Robots Coordination Outline

Human Power Augmentation [1] K. Kosuge, Y. Fujisawa, T. Fukuda, ”Mechanical System Control with Man-Machine-Environment Interactions”, Proceedings of the 1993 IEEE International Conference on Robotics and Automation, 1993, 239-244. [2] 小菅一弘, 藤沢佳生, 福田敏男, ”環境との相互作用が生じるマン・マシン系の制御”, 日本機械学会論文集(C編), 59 (562), 1993, 1751-1756. Human Power Augmentation

Q 1

M v Fe Dv Kv

Operator Tool Environment

Use of a Tool [1] K. Kosuge, Y. Fujisawa, T. Fukuda, ”Mechanical System Control with Man-Machine-Environment Interactions”, Proceedings of the 1993 IEEE International Conference on Robotics and Automation, 1993, 239-244. [2] 小菅一弘, 藤沢佳生, 福田敏男, ”環境との相互作用が生じるマン・マシン系の制御”, 日本機械学会論文集(C編), 59 (562), 1993, 1751-1756. Human Power Augmentation

Q 1

M v Fe Dv Kv

Mv !x!+ Dv x! + Kv x = QFh - Fe Virtual Tool Dynamics

[1] K. Kosuge, Y. Fujisawa, T. Fukuda, ”Mechanical System Control with Man-Machine-Environment Interactions”, Proceedings of the 1993 IEEE International Conference on Robotics and Automation, 1993, 239-244. [2] 小菅一弘, 藤沢佳生, 福田敏男, ”環境との相互作用が生じるマン・マシン系の制御”, 日本機械学会論文集(C編), 59 (562), 1993, 1751-1756. Human Power Augmentation(1993)

Human-Robot Cooperation (Kosuge, 1993~) Robot Helpers

Robot j

Robot i

Object Human l

Human m Robot k

Passive Dynamics

Stability Issues Robot Helpers

MR Helper (Mobile Robot Helper, 1997~) DR Helpers (Distributed Robot Helpers, 2000)

K. Kosuge, M. Sato, ”Mobile Robot Helper”, [Proceedings of the 2000 IEEE International Y. Hirata, K. Kosuge, ”Distributed Robot Helpers Handling a Single Object in Cooperation with a Human”, Conference on Robotics and Automation (2000) 583-588]. [Proceedings of the 2000 IEEE International Conference on Robotics and Automations (2000) 458-463]. 小菅一弘, 須田理央, 風村典秀, 佐藤学, 角谷啓, ”人と双腕型移動ロボット“MR Helper”による物平田泰久, 初雁卓郎, 小菅一弘, 淺間一, 嘉悦早人, 川端邦明, ”人間と複数の分散型ロボットヘル体の協調搬送”, [日本機械学会論文集(C編) 69 (685) (2003) 84-90]. パ-との協調による単一物体の搬送”, [日本機械学会論文集(C編) 68 (668) (2002) 181-188]. Robot Helpers

DR Helpers (Distributed Robot Helpers)

Y. Hirata, Y. Kume, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Based on Virtual 3-D Caster Motion for Handling an Object in Cooperation with a Human”, [Proceedings of the 2003 IEEE International Conference on Robotics and Automation (2003) 938-943]. Lessons Learned

DR Helper

MR Helper Lessons Learned from Robot Helpers

• Some simple tasks, which could not be done by a human/humans, could be done with a robot helper(s).

• General tasks could not be done easily even with the assistive robot system(s), because the robot does not know how to collaborate with the human. Lessons Learned from Robot Helpers

• In order to collaborate with the user, the robot has to know – the task, – its user’s intention, – how the user wants to be assisted – … Dance Partner Robot

To develop a mechanism for closer human-robot coordination/interaction Dance Partner Robot “PBDR”

PBDR as a Research Platform for Human-robot interaction (2005) Sensory Data Used for Estimation

Reference Data Transition

Dance Figure “A” Dance Figure “B”

Teffective Force/Moment

Time Time series data include uncertainty such as time-lag and variation because a dancer cannot always apply the same force/moment for each figure transition.

T. Takeda, K. Kosuge, Y. Hirata, ”HMM-based Dance Step Estimation for Dance Partner Robot -MS DanceR-”, [Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (2005) 1602-1607]. Sensory Data Used for Estimation

HMM-based “Figure Estimator”

Reference Data Transition

Dance Figure “A” Dance Figure “B”

Teffective Force/Moment

Time

PBDR as a Research Platform for Human-robot interaction(2005) Aichi Expo (March 24 ~ September 25, 2005)

PBDR as a Research Platform for Physical Human-Robot Interaction (2005) Outline

• A sequence of the tasks, necessary parts/tools for each task, and when and where each task is performed are scheduled a priori for each type of the car produced. • During the work, the worker needs to return to a work bench with parts and tools several times to pick up necessary parts/tools. Automobile Assembly Line

• If a robot could provide the worker with necessary parts and tools when he/she needs them, the worker could concentrate on the assembly tasks.

[1] 衣川潤，川合雄太，菅原雄介，小菅一弘，“組立作業支援パートナロボットPaDY（第1報，コンセプトモデルの開発とその制御）”，日本機械学会論文集，C 編, 77(783), (2011), 4204-4217 [2] J. Kinugawa, Y. Kawaai, Y. Sugahara and K. Kosuge, “PaDY : Human-Friendly/Cooperative Working Support Robot for Production Site”, The 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems Proceedings, (2010), 5472-5479. Co-worker Robot “PaDY”

• PaDY is a robot which delivers necessary parts and tools to a worker when he/she needs them. – to reduce the worker’s load – to improve efficiency of the work PaDY – to prevent mistakes of the work – etc.

[1] 衣川潤，川合雄太，菅原雄介，小菅一弘，“組立作業支援パートナロボットPaDY （第1報，コンセプトモデルの開発とその制御）”，[日本機械学会論文集，C 編,77(783),(2011),4204-4217]

[2] J. Kinugawa, Y. Kawaai, Y. Sugahara and K. Kosuge, “PaDY : Human- Friendly/Cooperative Working Support Robot for Production Site”, [2010 IEEE/RSJ International Conference on Intelligent Robots and Systems Proceedings,(2010),5472- 5479]. “in-time Parts/tools Delivery to You” robot Co-worker Robot “PaDY”

• In order to deliver parts/tools to a place, where the worker needs them, when the worker needs them without disturbing the worker’s work, the robot needs to know

– the task, – its user’s intention, and – how the user want to be assisted “in-time Parts/tools Delivery to You” robot The First Prototype of PaDY (P1)

Size：（W）1370×（D)590×（H）1035[mm] Link Mechanism ：Horizontal Articulated Robot Maximum Reach：2.0 [m]（1st Link Length：1168[mm], 2nd Link Length：982[mm]） Weight of Working Parts：11.5[kg] Maximum Load：3 [kg] Range of Movement： 1st Joint： 200[deg], 2nd Joint： 360[deg] Actuator：1st Joint & 2nd Joint：DC Servo Motor 80[W], 3rd Joint： DC Servo Motor 15[W] Evaluation Experiment

＋：LRF1 ×：LRF2

Measured data

Effect of PaDY The worker’s motion necessary for picking parts/tools has been reduced. The worker could finish his tasks earlier than the work schedule. Estimated result Architecture of the Adaptive Motion Planner Support

Desired trajectory Online Trajectory Generator Sensor Target position Robot

Target position Offset position Process determiner sheet

Initial model

Estimated Update current task Worker Motion Predictor Worker’s position Predicted moving • Estimates working trajectory position • Predicts moving trajectory Worker Motion Predictor[6] ü Worker’s movement is modeled by Gaussian mixture distribution using incremental learning algorithm. ü Worker motion predictor estimates worker’s current task and predicts worker’s motion trajectory.

Order Task ② ② Generate 1 Task1 ③ Initialization ③ 2 Task2 ① 3 Task3 ④ ④ ① 4 Task4 Process Chart Initial Worker Model

Sample Data

Prediction&Update Worker

Motion Prediction Model Update

[6] J. Kinugawa, A. Kanazawa, S. Arai, and K. Kosuge, “Adaptive task scheduling for an assembly task coworker robot based on incremental learning of human motion patterns,” IEEE Robot. Autom. Lett., vol. 2, no. 2, pp. 856–863, Apr. 2017. Online Trajectory Generator

Target position Target time 1st Term: Minimizing delivery time delay Predicted The endpoint of the manipulator arrives at the scheduled worker trajectory target position at the target time. Robot trajectory Time scale 2nd Term: Collision avoidance The manipulator avoids colliding with the predicted worker position at each time step. Worker Robot 3rd Term: Velocity / Acceleration limitation The manipulator moves under the preset velocity and acceleration limitations.

The trajectory that satisfies the above three requirements is calculated by minimizing the following cost function. �: Current time Target time () () () () �: � = � � + � � + � � , � �: Robot state 1st Term 2nd Term 3rd Term Cost Function for Uncertainty-based Collision Avoidance

Mean of The variance of the predicted trajectory tends Trajectory to Increase during irregular movement. Variance of Worker Trajectory We consider predicted variance (uncertainty) in the cost function of the collision avoidance.

Mahalanobis distance Predicted trajectory in a regular case Predicted trajectory in an irregular case � �, �, � = � − � � (� − �)

() � � () 1 ! � � = � () () () � � � , � , �

() �: Constant coefficient �! � �: Total number of joints Probabilistic distribution of � �() : Position of the � th joint at step � predicted worker’s position () () � � , � Outline of Control System

Model learning (Update the model in every work cycle)

Work Worker schedule Process Data Sheet

Worker Offset position model Planning layer Control layer Current worker Current Target delivery Optimal Command position task Target Position position trajectory Robot velocity Sensor Worker Robot Determiner Online Trajectory Controller Motion Generator Predictor Worker’s predicted trajectory

Iterating at 30ms Iterating at 1ms Experimental setup 2.0 m

Task1

② ① Vehicle Body LRF 1.2 m 1.2 X Task2 LRF Start Y Robot ③ Task3 Goal First joint X

Y Robot End point

Second joint

• Five participants perform the experiments of attaching three types of parts to three different places of a vehicle body. • The worker attaches the parts supplied by the robot to the overhead vehicle body. • One experiment is completed when the worker returns to the starting point after all tasks are finished. • The prediction model is updated after each experiment is completed. • The cost function with the same parameters is used for all experiments. Experiments

Vehicle Body Task1 ① 2.0 m ② LRF X

1.2 m 1.2 PaDY Task2 Start Y ③ Task3 Goal X

Y PaDY LRF

• A subject is requested to do three tasks. • After the three tasks were finished, the probabilistic model is updated.

Akira Kanazawa, Jun Kinugawa, and Kazuhiro Kosuge, “Adaptive Motion Planning for a Collaborative Robot Based on Prediction Uncertainty to Enhance Human Safety and Work Efficiency”, IEEE Transactions on Robotics, Vol. 35, No. 4, pp.817-832, 2019 Experiments

Ordinary Motion (10th trial by Subject A) Variance of Estimated Trajectory Task1 Estimated Trajetory

Task2 Observed Workers Position

Task3 Start and Goal Position PaDY

trial 1 （without estimation） trial 10 （with estimation）

Start ⇒ Task1

Task1 ⇒ Task2

Comparison of the work by subject B Akira Kanazawa, Jun Kinugawa, and Kazuhiro Kosuge, “Adaptive Motion Planning for a Collaborative Robot Based on Prediction Uncertainty to Enhance Human Safety and Work Efficiency”, IEEE Transactions on Robotics, Vol. 35, No. 4, pp.817-832, 2019 Experiments

Subject B Irregular Behavior

Subject C Irregular Behavior

Subject D Irregular Behavior

Subject E Irregular Behavior

2020/10/31 76 D-PaDY

2020/10/31 Outline

• An industrial robot is utilized as a system together with peripheral systems, such as

an endeffector(s), a fixture/jig(s) and a parts fieder(s), designed for each task.

• The peripheral systems used for a robot often exceeds the cost of the robot. http://www.sigma-fa.co.jp/rob_APPRI_5.htm Today’s Industrial Robots

• An industrial robot is utilized as a system together with peripheral systems, such as

an endeffector(s), a fixture/jig(s) • Today’sand industrial robot is not universal and could not be used withouta parts customization fieder(s), and programing of the robot, which designed for each task. require a lot of man-hours. • The peripheral systems used for a robot often exceeds the cost of the robot. http://www.sigma-fa.co.jp/rob_APPRI_5.htm Integrated Visual and Impedance Servo for Program-fee Assembly

Forward Kinematics

Motion Inverse Kinematics + + Controller Manipulator + + Impedance External Force Force Sensor Model Estimation

Force Control Coordinate System Estimation

Estimation of Image Motion deviation Visual Tracking Servo Sensor Goal Image Pose Estimation Generated by a CAD model 80 Integrated Visual and Impedance Servo for Program-fee Assembly

• Trajectory Generation Forward Kinematics

Motion Inverse Kinematics + + Controller Manipulator + + Impedance External Force Force Sensor Model Estimation

Force Control Coordinate System Estimation

Estimation of Image Motion deviation Visual Tracking Servo Sensor Goal Image Pose Estimation Generated by Visual Servo a CAD model 81 Integrated Visual and Impedance Servo for Program-fee Assembly • CAD model-based Impedance control Forward Kinematics

Motion Inverse Kinematics + + Controller Manipulator + + Impedance External Force Force Sensor Model Estimation Impedance Control Force Control Coordinate System Estimation

Estimation of Image Motion deviation Visual Tracking Servo Sensor Goal Image Pose Estimation Generated by a CAD model 82 Integrated Visual and Impedance Servo for Program-fee Assembly

• Trajectory • CAD model-based Impedance Generation control Forward Kinematics

Motion Inverse Kinematics + + Controller Manipulator + + Impedance External Force Force Sensor Model Estimation Impedance Control Force Control Coordinate System Estimation

Estimation of Image Motion deviation Visual Tracking Servo Sensor Goal Image Pose Estimation Generated by Visual Servo a CAD model 83 Program-free Bolt Fastening by Integrated Visual and Impedance Servo 1.5 minuets • The bolt fastening task is shown as a sequence of goal images. • The goal images could be generated without programing. • by demonstration • using CAD model • … Program-free Assembly by Demonstration 2 minuets

Assembly of Two Parts by Demonstration (Tohoku University)

Takashi Nammoto, Kazuhiro Kosuge, Koichi Hashimoto, “Model-Based Compliant Motion Control Scheme for Assembly Tasks Using Vision and Force Information,” Proceedings of 2013 IEEE International Conference on Automation Science and Engineering, (2013), 948-953. Customization Free Bin Picking

Universal Bin Picking System (Tohoku University) Universal Bin Picking and Kitting System

Systems Robotics Lab. Tohoku University Visual Servoing

• Position based visual servoing[1] (PBVS) – Positioning by estimating the deviation of the target pose and current pose from the camera image. • Require precise robot-camera calibration • Speeding up the control cycle is difficult due to the time of pose estimation.

• Image based visual servoing[2] (IBVS) – Positioning is performed by matching the image feature extracted from the camera image with the target image feature. • Robot-camera calibration is unnecessary • Relatively easy to speed up the control cycle

[1] 橋本浩一, ``ビジュアルサーボ-Ⅴ: 位置ベースビジュアルサーボ,” 一般社団法人システム制御情報学会システム/制御/情報, 2010, vol. 54, no. 3, pp. 117-123. [2] 橋本浩一, ``ビジュアルサーボ-Ⅳ: 特徴ベースビジュアルサーボ,” 一般社団法人システム制御情報学会システム/制御/情報, 2010, vol. 54, no. 5, pp. 206-213. Image Based Visual Servoing

• Image based visual servoing (IBVS) – Positioning by minimizing the difference between the feature �∗ and � extracted from the target image �∗ and the current image �, respectively.

Image Jacobian • Control law of IBVS �� ̇ ∗ � = � = −�� − � �� • Design of image feature and estimation of image Jacobean – It is necessary to design an image feature manually and estimate an image Jacobian corresponding to the designed image feature. Background

• Image based visual servoing (IBVS) – Positioning by minimizing the difference between the feature �∗ and � extracted from the target image �∗ and the current image �, respectively.

• Control law of IBVS ̇ ∗ � = −�� − � • We need to design an image feature manually and estimate an image Jacobian corresponding to the designed image feature. Image Jacobian – Direct visual servoing approach �� = – Neural network based approach �� Proposed method (IJL-VS)

• Calculate angular velocity command by CNN instead of using image Jacobian

• Input of CNN : Current image � and desired image �∗ • Output of CNN: Difference of the current angle and target angle • Control law of the proposed method �̇ = −� � � �∗ , � � • Control law of IBVS ̇ ∗ � = −� � � � − � Positioning experiment in a real environment • The network is trained only using Object A under constant lighting condition.

Train object:

Object A

Object B Object D Test objects:

Object C Object E Experimental results • Positioning results under unseen lighting condition using seen object (Object A). Experimental results

• Positioning results under seen lighting condition using unseen objects. Experimental results • Positioning results under seen lighting condition using seen objects (Object A) with occlusions.

Occlusion is generated by image processing. Experimental results Outline

• How I started my research in robotics • Robot Systems Integration • Physical Human-Robot Interaction – Human robot collaboration through interaction – Co-worker robot • Universal Manipulation – Issues and visual servoing for program-free robot • Conclusions http://www.nsf.gov/about/transformative_research/index.jsp (October 31, 2020) Transformative research involves ideas, discoveries, or tools that radically change our understanding of an important existing scientific or engineering concept or educational practice or leads to the creation of a new paradigm or field of science, engineering, or education. Such research challenges current understanding or provides pathways to new frontiers. https://www.nsf.gov/about/transformative_research/definition.jsp (October 31, 2020) https://www.nsf.gov/about/transformative_research/characteristics.jsp (October 31, 2020) https://www.nsf.gov/about/transformative_research/merit_review_criteria.jsp (October 31, 2020) https://www.darpa.mil (October 31, 2020) https://www.darpa.mil/about-us/about-darpa (October 31, 2020) Defense Advanced Research Projects Agency

• “The genesis of that mission and of DARPA itself dates to the launch of Sputnik in 1957, and a commitment by the United States that, from that time forward, it would be the initiator and not the victim of strategic technological surprises.”

• “Working with innovators inside and outside of government, DARPA has repeatedly delivered on that mission, transforming revolutionary concepts and even seeming impossibilities into practical capabilities.”

https://www.darpa.mil/about-us/about-darpa (October 31, 2020) Defense Advanced Research Projects Agency

• “The ultimate results have included not only game-changing military capabilities such as precision weapons and stealth technology, but also such icons of modern civilian society such as the Internet, automated voice recognition and language translation, and Global Positioning System receivers small enough to embed in myriad consumer devices.”

https://www.darpa.mil/about-us/about-darpa (October 31, 2020) By design, programs are finite in duration while creating lasting revolutionary change.

https://www.darpa.mil/our-research (October 31, 2020) Defense Advanced Research Project Agency

• DARPA’s scientific investigations span the gamut from laboratory efforts to the creation of full-scale technology demonstrations in the fields of physics, engineering, biology, medicine, computer science, chemistry, mathematics, material sciences, social sciences, neurotechnology and more. • As the Department of Defense’s primary innovation engine, DARPA undertakes projects that are finite in duration but that seek to create lasting revolutionary change.

http://www.darpa.mil/work-with-us/opportunities/more Defense Advanced Research Project Agency

• Exoskeletons for Human Performance Augmentation(2000 ~ ) – “The primary goal of the Exoskeletons program is to develop human performance augmentation capabilities that increase human speed, strength, and endurance in combat environments.” – Computer system interacts with a human to enhance human performance. Human Power Augmentation

• HULC (Human Universal Load Carrier, Unveiled in 2012, after licensed to Rokheed Martin by Prof. H. Kazerooni, UC Berkeley)

– “It takes up to 200 pounds without impeding the wearer (Strength Augmentation)” – “It decreases its wearer’s metabolic cost (Endurance Augmentation).”

https://news.lockheedmartin.com/2018-11-29-Lockheed-Martin-Secures-U-S-Army-Exoskeleton-Development- Agreement#assets_117:19597 (October 31, 2020) Exo Bionics

Eksohealth Eksoworks “Ekso Bionics is the only exoskeleton company to offer technologies that range from helping those with paralysis to stand up and walk, to enhancing human capabilities on job sites across the globe, to providing research for the advancement of R&D projects intended to benefit U.S. defense capabilities.” https://eksobionics.com (October 31, 2020) Industrial Exoskeleton

SuitX by Prof. Homayoon Kazerooni (2016)

https://www.suitx.com (October 31, 2020) BigDog and SpotMini SPOT

https://www.bostondynamics.com https://www.bostondynamics.com SPOT

https://www.youtube.com/watch?v=wlkCQXHEgjA&feature=emb_logo

https://www.bostondynamics.com (October 31, 2020) Global Issues

• Population aging • Urbanization – “Globally, more people live in urban areas than in rural areas, with 55 % of the world’s population residing in urban areas in 2018. In 1950, 30 % of the world’s population was urban, and by 2050, 68 % of the world’s population is projected to be urban.” from “World Urbanization Prospects: The 2018 Revision,” https://population.un.org/wup/Publications/Files/WUP2018-KeyFacts.pdf • Global warming • Pandemic https://population.un.org/wpp/Maps/ https://population.un.org/wpp/Maps/ https://population.un.org/wpp/Maps/ Global Issues

• Population aging • Urbanization – “Globally, more people live in urban areas than in rural areas, with 55 % of the world’s population residing in urban areas in 2018. In 1950, 30 % of the world’s population was urban, and by 2050, 68 % of the world’s population is projected to be urban.” from “World Urbanization Prospects: The 2018 Revision,” https://population.un.org/wup/Publications/Files/WUP2018-KeyFacts.pdf • Global warming • Pandemic • Others Robotics technology is one of the key technologies for us to overcome the negative effects of the global issues. Challenges and Opportunities in Robotics

Social Value Global Level Community Level Quality of Life

CRDS, JST, 2009, Modified by Kosuge, August, 2011 1966

1992

日本の人口構成を考えると，今が新しいことにチャレンジする最後のチャンス。 http://www.mext.go.jp/component/a_menu/education/detail/__icsFiles/afieldfile/2018/02/16/1401001_4.pdf