Design Industrial Applications with MATLAB and Simulink

Presenter Name Here Trends in Industrial Robotics

Cobots Grow fastest in shipment terms with CAGR of 20% from 2017 - 2023

Growing trend toward compact Increasing share of units shipped in 2023 will be payload <10kg 40% of 80% of 82% of Articulated Robots SCARA Robots Cobots Source: Interact Analysis 2 Evolution of Industrial Robotics Technologies

Full Autonomy AI-enabled robots High Autonomy

Conditional Autonomy

Partial Autonomy Cobot UR5 1st Cobot at Linatex, 2008 1961

Human Assistant Traditional Industrial Robots -

Unimate at GM, 1961

No Automation 1961 2008 2019-

3 Autonomous Industrial Robotics Systems Workflow Operates independently, without explicit instructions from a human

Connect/Deploy Code Generation ROS

Autonomous Algorithms

Sense (Observe) Perceive (Orient) Plan (Decide) Control (Act)

Encoder Camera Environment Localization Feedback Controls Understanding Force/Torque Contact Path Planning Decision Logic Sensor Switches Object Gain Scheduling Proximity Sensors Detection/Tracking Obstacle Avoidance Trajectory Control

S1 S3 S2

Platform Prototype HW Physical Model Actuation Model Environ Model Production HW

4 What we’ll discuss today

01 Challenges in Industry System Design 02 Model-Based Design for Autonomous System Case Study: Pick-and- 03 Place Manipulation Application 04 Concluding Remarks

5 What we’ll discuss today

01 Challenges in Industry Robot System Design 02 Model-Based Design for Autonomous System Case Study: Pick-and- 03 Place Manipulation Application 04 Concluding Remarks

6 Traditional Software Development Cycle

Only 6% Of Design/Development Design Concept a Time is Spent on Simulation* * AspenCore - EETimes, “2019 embedded markets study,” EETimes, Tech. Rep., 2019

b Handwritten Code

Embedded System

7 Common Challenges of Industrial Robotics Systems Development

Multidomain End-to-End Expertise workflows

Technical Depth Complexity of and System Algorithms Stability

8 Key Takeaways

In this talk, you will learn

Reference workflow for development

Multi-domain functional areas of Platform, Sensing, Perception, Planning and Control

MATLAB and Simulink capabilities to develop new robot algorithms

» Kinematic and dynamic models of robots » Perception algorithm design using deep learning » Gazebo co-simulation for sensor models and environment simulation » Path planning with obstacle avoidance » Supervisory logic and control using Stateflow / RL » C/C++ code / ROS nodes generation

9 What we’ll discuss today

01 Challenges in Industry Robot System Design 02 Model-Based Design for Autonomous System Case Study: Pick-and- 03 Place Manipulation Application 04 Concluding Remarks Key to developing robust autonomous system

Complete Model-Based Design Workflow

Modeling & Test & Simulation Verification Need: An end-to-end development solution that includes modeling & simulation, code generation and test & verification.

Code Generation Simulate First and Simulate Often!

11 Full Model-Based Design Workflow Connect / Deploy Connect /Deploy

Autonomous Algorithms for Manipulators Plan & Perceive Decide

Sense Control

Platform Platform

12 What we’ll discuss today

01 Challenges in Industry Robot System Design 02 Model-Based Design for Autonomous System Case Study: Pick-and- 03 Place Manipulation Application 04 Concluding Remarks

13 Pick-and-Place Manipulators Robot Arm Demo Model-Based Design Deep Learning for detecting an object Path planning with collision avoidance

14 Mechanical Modeling Automatic import from CAD Tools

CAD Model Platform

Sense

Perceive

Plan & Connect & Deploy & Connect Decide Simscape Multibody Control Model

15 Evaluating motor requirements Actuators Connect & Deploy Perceive Control Plan & Sense Platform Decide – actuator sizing Robotics Simscape Toolbox System 16 Robotics System Environment Modeling Toolbox

Connect to an external robotics simulator ROS Toolbox

Robot arm simulation with Gazebo

Gazebo: Physics-based simulator with sensors and noise

17 Sensing Computer Vision Point cloud processing for pose estimation Toolbox

Platform

Sense

Colorized point cloud Detect table Intel® RealSense™ Perceive RGB-D camera

Plan & Connect & Deploy & Connect Decide

Control Point clouds of objects Remove noise and cluster

18 Computer Vision Sensing Toolbox Common sensors and sensing functionalities for autonomous systems Image Processing Toolbox

Platform • Support for Common Sensors

Sense • Image analysis

• Image enhancement Perceive • Visualizing Point Clouds

Plan & • Apps Connect & Deploy & Connect Decide

Control

19 Perception Deep learning for object classification Deep Learning Toolbox

Platform Object detector using Deep Learning (YOLO v2) Sense

• Perceive • •

Plan & Connect & Deploy & Connect Decide

Control

20 Perception Object Classification

Platform

Sense

Perceive

Plan & Connect & Deploy & Connect Decide

Control

21 Motion Planning Initial Pose X(t0) Final Pose X(tf) Motion Joint Trajectories Joint Limits Planner Q(t) Obstacles

Platform Joint positions

Follow & track trajectory Sense Reach waypoints

Interact with environment Perceive X(t0) q(t0) Manage gripper actions

Plan & Connect & Deploy & Connect Decide q(t1) q(t3) X(tf) Control q(t2) q(tf)

22 Robotics System Motion Planning Toolbox Model Predictive Path Planning + Trajectory Gen + Trajectory Following Control Toolbox

Platform

Sense

Perceive

Plan & Connect & Deploy & Connect Decide

Control

23 Motion Control

Decision Logic Stateflow

Platform

Sense Supervisory Control

Perceive

Plan & Connect & Deploy & Connect Decide

Control

24 Advanced Control: Reinforcement Learning Grasping an object with image inputs Reinforcement Learning Toolbox

AGENT ACTION STATE Policy

Policy update Reinforceme nt Learning Algorithm

REWARD

ENVIRONMENT

25 Hardware Connectivity

Code Generation Support ROS Toolbox

Platform Application Sense

Sense1 Sense2 Simulator Perceive

Plan &

Connect & Deploy & Connect Middleware Decide

Control Act1 HMI1 HMI2

26 Hardware Connectivity Code Generation Support Stateflow

Use publisher/subscriber Platform capabilities in Simulink to connect to ROS topics

Sense Jetson Xavier (Ubuntu 18.04)

Perceive (Bouncy)

Node2 Node1 User Application Ethernet Kinova

Plan & System Connect & Deploy & Connect Node3 Object Decide Kortex API Simulink to ROS2 User Application stand-alone node deployment Allows multi-thread, Control multi-core and pseudo real-time

27 Use the same reference workflow For warehouse pick-and-place (storage shelf)

This workflow example highlights the use of Robotics System Toolbox collision-checking algorithms, nonlinear MPC, and Stateflow for MATLAB

28 Use the same reference workflow For warehouse pick-and-place (storage shelf)

With obstacle avoidance ON (obstacle shown in blue)

With obstacle avoidance OFF (obstacle shown in black for reference)

29 Use the same reference workflow For Delta robot for automated parts sorting Connect

Sense

Perceive

Plan & Control Platform Decide What we’ll discuss today

01 Challenges in Industry Robot System Design 02 Model-Based Design for Autonomous System Case Study: Pick-and- 03 Place Manipulation Application 04 Concluding Remarks

31 Full Model-Based Design Workflow Connect / Deploy Connect /Deploy

ROS Toolbox Autonomous Algorithms

Sensor Fusion and Tracking Tbx Plan & Perceive Navigation Deep Learning Decide Toolbox Toolbox

Robotics System Toolbox

Stateflow

Computer Vision Reinforcement Toolbox Sense Control Learning Toolbox Model Predictive Control Toolbox

Platform Platform MATLAB / Simulink Simscape

32 Resources to get started with

33 Concluding Remarks

Challenges Develop Software End-to-end workflow in industrial robot with Model-Based for industry robot application Design applications development development

Multi-domain Fast Iterations Platform Expertise

Complexity of Sense Algorithms Perceive End-to-End Strong Focus workflows on Simulation Plan & Decide Technical Depth and System Stability Control

34 % Thank you!

mathworks.com/robotics

35