Automated Driving System Design and Simulation Using MATLAB And

Automated Driving with MATLAB and Simulink

GianCarlo Pacitti Senior Application Engineer, MathWorks

2 Capabilities of an Autonomous Vehicle

3 Capabilities of an Autonomous Vehicle

4 Capabilities of an Autonomous Vehicle

5 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

How can I How can I How can I synthesize scenarios discover and design integrate to test my designs? in multiple domains? with other environments?

6 How can I design with virtual driving scenarios?

Scenes Cuboid

Testing Controls, sensor fusion, planning

Authoring Driving Scenario Designer App Programmatic API (drivingScenario) Sensing Probabilistic radar (detection list) Probabilistic vision (detection list) Probabilistic lane (detection list)

7 How can I design with virtual driving scenarios?

Scenes Cuboid 3D Simulation

Testing Controls, sensor fusion, planning Controls, sensor fusion, planning, perception

Authoring Driving Scenario Designer App Unreal Engine Editor Programmatic API (drivingScenario) Sensing Probabilistic radar (detection list) Probabilistic radar (detection list) Probabilistic vision (detection list) Monocular camera (image, labels, depth) Probabilistic lane (detection list) Fisheye camera (image) Lidar (point cloud) 8 Simulate controls with perception

Lane-Following Control with Monocular Camera Perception ▪ Author target vehicle trajectories ▪ Synthesize monocular camera and probabilistic radar sensors ▪ Model lane following and spacing control in Simulink ▪ Model lane boundary and vehicle detectors in MATLAB code

Model Predictive Control ToolboxTM Automated Driving ToolboxTM Vehicle Dynamics BlocksetTM

Updated Visit the Demo Station to see more…

9 Visualize logged simulation detection and camera data

Model Predictive Control ToolboxTM Automated Driving ToolboxTM Vehicle Dynamics BlocksetTM Updated

10 How can I design with virtual driving scenarios?

Scenes Cuboid 3D Simulation

Testing Controls, sensor fusion, planning Controls, sensor fusion, planning, perception

Sensor Fusion Using Synthetic Radar and Vision Data ▪ Create scenario ▪ Add probabilistic radar and vision sensors ▪ Create tracker ▪ Visualize coverage area, detections, and tracks

Automated Driving ToolboxTM

12 Graphically author driving scenarios

Driving Scenario Designer ▪ Create roads and lane markings ▪ Add actors and trajectories ▪ Specify actor size and radar cross-section (RCS) ▪ Explore pre-built scenarios ▪ Import OpenDRIVE roads

Automated Driving ToolboxTM

13 Programmatically author driving scenarios

14 Synthesize driving scenarios from recorded data

Scenario Generation from Recorded Vehicle Data ▪ Visualize video ▪ Import OpenDRIVE roads ▪ Import GPS ▪ Import object lists

Automated Driving ToolboxTM

15 Enhancements to driving scenarios

Create Driving Scenario Variations Programmatically

▪ Export the scenario code to MATLAB® and generate scenario variations programmatically ▪ Export the scenario and sensors to Simulink® and use them to test your driving algorithms.

Automated Driving ToolboxTM

16 Integrate driving scenario into closed loop simulation

Lane Following Control with Sensor Fusion ▪ Integrate scenario into system ▪ Design lateral (lane keeping) and longitudinal (lane spacing) model predictive controllers ▪ Visualize sensors and tracks ▪ Generate C/C++ code ▪ Test with software in the loop (SIL) simulation

Model Predictive Control ToolboxTM Automated Driving ToolboxTM Embedded Coder®

17 Design lateral and longitudinal controls

Model Predictive Control ToolboxTM Automated Driving ToolboxTM Embedded Coder®

18 Automate testing against driving scenarios

Testing a Lane Following Controller with Simulink Test Requirements link ▪ Specify driving scenario

Simulink Model

Scenarios

TM Define scenario ID Simulink Test and data initialization Automated Driving ToolboxTM Model Predictive Control ToolboxTM Plot the results

19 How can I design with virtual driving scenarios?

Scenes Cuboid 3D Simulation

Testing Controls, sensor fusion, planning Controls, sensor fusion, planning, perception

3D Simulation for Automated Driving ▪ Straight road ▪ Curved road ▪ Parking lot ▪ Double lane change ▪ Open surface ▪ US city block ▪ US highway ▪ Virtual Mcity

Automated Driving ToolboxTM

21 Customize 3D simulation scenes

Support Package for Customizing Scenes ▪ Install Unreal Engine ▪ Set up environment and open Unreal Editor ▪ Configure configuration Block for Unreal Editor co-simulation ▪ Use Unreal Editor to customize scenes

Vehicle Dynamics BlocksetTM

22 Model sensors in 3D simulation environment

3D Simulation for Automated Driving ▪ Monocular camera ▪ Fisheye camera ▪ Lidar ▪ Probabilistic radar

Automated Driving ToolboxTM

23 Synthesize monocular camera sensor data

Visualize Depth and Semantic Segmentation Data in 3D Environment ▪ Synthesize RGB image ▪ Synthesize depth map ▪ Synthesize sematic segmentation

Automated Driving ToolboxTM

24 Synthesize fisheye camera sensor data

Simulate a Simple Driving Scenario and Sensor in 3D Environment ▪ Scaramuzza camera model – parameters for distortion center, image size and mapping coefficients

Automated Driving ToolboxTM

25 Synthesize lidar sensor data

Simulate Lidar Sensor Perception Algorithm ▪ Record and visualize ▪ Develop algorithm ▪ Build a 3D map ▪ Use algorithm within simulation environment

Automated Driving ToolboxTM

26 Synthesize radar sensor data

Simulate Radar Sensors in 3D Environment ▪ Extract the center locations ▪ Use center location for road creation using driving scenario ▪ Define multiple moving vehicles ▪ Export trajectories from app ▪ Configure multiple probabilistic radar models ▪ Calculate confirmed track

Automated Driving ToolboxTM

27 Communicate with the 3D simulation environment

Send and Receive Double-Lane Change Scene Data ▪ Simulation 3D Message Set – Send data to Unreal Engine – Traffic light color

▪ Simulation 3D Message Get – Retrieve data from Unreal Engine – Number of cones hit

Vehicle Dynamics BlocksetTM

28 New Examples for 3D Simulation in Automated Driving Toolbox

29 Simulating automated driving systems with MATLAB and Simulink

Simulation Environment

Perception

Planning

Control

30 Simulating automated driving systems with MATLAB and Simulink

Simulation Environment

Scenarios Perception Scenery

Actions & Actors Events Planning

Goals & Sensors Metrics Control

31 Integrate components and model scenarios

Monocular Actions camera lane & Events detector

Actors

Scenery

Sensors Lane following controller Goals & Metrics 32 Specify equivalent 3D Simulation scenery Supported Scenery: Scenery: ✓ Straight road ✓ Curved road segment - Equivalent straight and curved roads in Simulation 3D ✓ Curved road (not exposed in Scene Configuration and Scenario Reader example, but available)

33 Specify 3D Simulation actor trajectories

Action & Events: - Scenario Reader describes trajectories of target vehicles - Target trajectories are converted to world coordinates

34 Specify 3D Simulation vehicles Actors: - Target vehicle positions specified from Scenario Reader block

Actors: - Ego vehicle position specified based on vehicle dynamics

35 Synthesize scenarios to test your design

Lane Following Control with Design of Lane Marker Lane-Following Control with Sensor Fusion Detector in 3D Simulation Monocular Camera Perception Model Predictive Control ToolboxTM Environment Model Predictive Control ToolboxTM Automated Driving ToolboxTM Automated Driving ToolboxTM ® TM Embedded Coder Automated Driving ToolboxTM Vehicle Dynamics Blockset Updated

36 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

How can I How can I How can I synthesize scenarios discover and design integrate to test my designs? in multiple domains? with other environments?

37 Design camera, lidar, and radar perception algorithms

Detect vehicle with Detect ground with Detect pedestrian with camera lidar radar

Object Detection Using Segment Ground Points Introduction to Micro-Doppler YOLO v2 Deep Learning from Organized Lidar Data Effects Computer Vision ToolboxTM Computer Vision ToolboxTM Phased Array System ToolboxTM Deep Learning ToolboxTM

38 Interoperate with neural network frameworks

PyTorch Keras- Tensorflow Caffe2

MXNet ONNX MATLAB

Core ML Caffe CNTK (…)

Open Neural Network Exchange Visit the Demo Stations to see more… 39 Simulate lane detection and lane following system with MATLAB and Simulink

Simulation Environment

Scenarios Monocular Scenery camera lane detector

Actions & Actors Events

Goals & Lane following Sensors Metrics controller

40 Monocular Camera Lane detector example

Monocular camera lane detector - Based on shipping example Visual Perception Using - Lane rejection and tracking added to Monocular Camera improve performance Automated Driving ToolboxTM

41 Design detector for lidar point cloud data

Track Vehicles Using Lidar: From Point Cloud to Track List ▪ Design 3-D bounding box detector ▪ Design tracker (target state and measurement models) ▪ Generate C/C++ code for detector and tracker

Sensor Fusion and Tracking ToolboxTM Computer Vision ToolboxTM

42 Design tracker for lidar point cloud data

Sensor Fusion and Tracking ToolboxTM Computer Vision ToolboxTM

43 Design trackers

Multi-Object Tracker

Association & Tracking Detections Track Tracks Filter Management

From various sensors at various update rates

▪ Multi-object tracker ▪ Linear, extended, and unscented Kalman filters

Automated Driving ToolboxTM

44 Design trackers

Multi-Object Tracker

Association & Tracking Detections Track Tracks Filter Management

From various sensors at various update rates

▪ Multi-object tracker ▪ Linear, extended, and ▪ Global Nearest Neighbor (GNN) tracker unscented Kalman filters ▪ Joint Probabilistic Data Association (JPDA) tracker ▪ Particle, Gaussian-sum, ▪ Track-Oriented Multi-Hypothesis Tracker (TOMHT) IMM filters ▪ Probability Hypothesis Density (PHD) tracker

Automated Driving ToolboxTM Sensor Fusion and Tracking ToolboxTM

45 Evaluate error metrics

Extended Object Tracking ▪ Design multi-object tracker ▪ Design extended object trackers ▪ Evaluate tracking metrics ▪ Evaluate error metrics ▪ Evaluate desktop execution time

Sensor Fusion and Tracking ToolboxTM Automated Driving ToolboxTM Updated

Multi-object tracker Probability Hypothesis Density tracker Extended object (size and orientation) tracker

46 Compare relative execution times of object trackers

Extended Object Tracking ▪ Design multi-object tracker ▪ Design extended object trackers ▪ Evaluate tracking performance ▪ Evaluate error metrics ▪ Evaluate desktop execution time

Sensor Fusion and Tracking ToolboxTM Automated Driving ToolboxTM Updated

Multi-object tracker Probability Hypothesis Density tracker Extended object (size and orientation) tracker

47 Design track level fusion systems

Vehicle 1

Multi-Object Detections Tracker Tracks

Vehicle 2

Multi-Object Detections Tracker Tracks

48 Design track level fusion systems

Vehicle 1

Multi-Object Track Detections Tracker Fusion Tracks

Vehicle 2

Multi-Object Detections Tracker Tracks

49 Design track level fusion systems

Vehicle 1

Multi-Object Track Detections Tracker Fusion Tracks

Vehicle 2

Multi-Object Track Detections Tracker Fusion Tracks

50 Track-level fusion

Track-to-Track Fusion for Automotive Safety Applications

Occluded Parked vehicles vehicle fused observed by from vehicle 1 vehicle 1 Occluded Pedestrian pedestrian observed by fused from vehicle 1 vehicle 1 Pedestrian observed by vehicle 2 Sensor Fusion and Rumor control: the fused TM Tracking Toolbox track is dropped by vehicle 1 Automated Driving ToolboxTM because vehicle 2 is coasting and there is no update by vehicle 1 sensors 51 For more on Sensor Fusion and Tracking…

Visit Marc Willerton’s presentation later this afternoon

52 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

How can I How can I How can I synthesize scenarios discover and design integrate to test my designs? in multiple domains? with other environments?

53 Read road and speed attributes from HERE HD Live Map data

Use HERE HD Live Map Data to Verify Lane Configurations ▪ Load camera and GPS data ▪ Retrieve speed limit ▪ Retrieve lane configurations ▪ Visualize composite data

Automated Driving ToolboxTM

54 Read lane attributes from HERE HD Live Map data

Use HERE HD Live Map Data to Verify Lane Configurations ▪ Load camera and GPS data ▪ Retrieve speed limit ▪ Retrieve lane configurations ▪ Visualize composite data

Automated Driving ToolboxTM

55 Visualize HERE HD Live Map recorded data

Use HERE HD Live Map Data to Verify Lane Configurations ▪ Load camera and GPS data ▪ Retrieve speed limit ▪ Retrieve lane configurations ▪ Visualize composite data

Automated Driving ToolboxTM

56 Design path planner

Automated Parking Valet ▪ Create cost map of environment ▪ Inflate cost map for collision checking ▪ Specify goal poses ▪ Plan path using rapidly exploring random tree (RRT*)

Automated Driving ToolboxTM

57 Design path planner and controller

Automated Parking Valet with Simulink ▪ Integrate path planner ▪ Design lateral controller (based on vehicle kinematics) ▪ Design longitudinal controller (PID) ▪ Simulate closed loop with vehicle dynamics

Visualize Automated Parking Valet Using 3D Simulation

Automated Driving ToolboxTM

58 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

How can I How can I How can I synthesize scenarios discover and design integrate to test my designs? in multiple domains? with other environments?

59 Train reinforcement learning networks for ADAS controllers

Train Deep Deterministic Policy Gradient (DDPG) Agent for Adaptive Cruise Control ▪ Create environment interface ▪ Create agent ▪ Train agent ▪ Simulate trained agent

Reinforcement Learning ToolboxTM

Visit the Demo Stations to see more… 60 Simulate lane detection and lane following system with MATLAB and Simulink

Simulation Environment

Scenarios Monocular Scenery camera lane detector

Actions & Actors Events

Goals & Lane following Sensors Metrics controller

61 Lane Following Controller Algorithm

Lane following controller

62 Components of lane following with spacing control algorithm

Estimate lane Path center following control block

Estimate lead vehicle

63 Goal

▪ Maintain the driver-set velocity and keep a safe distance from lead vehicle. 푉푒푔표, 푎 푉푚푖표 퐷푟푒푙푎푡푖푣푒

▪ Keep the ego vehicle in the middle of the lane.

훿 퐸푦푎푤

휌 퐸푙푎푡푒푟푎푙

▪ Slow down the ego vehicle when road is curvy.

64 Model predictive control (MPC)

Measured outputs

Hidden Connector Optimizer Ego References Plant Manipulated variables Vehicle Model Hidden Connector MPC controller

Measured disturbances

65 MPC for Lane Following Control

Measured outputs 퐦퐢퐧퐢퐦퐢퐳퐞: • Relative distance (퐷푟푒푙푎푡푖푣푒) ퟐ ퟐ • Ego velocity (푉푒푔표) 풘ퟏ 푽풆품풐 − 푽풔풆풕 + 풘ퟐ 푬풍풂풕풆풓풂풍 • Lateral deviation (퐸푙푎푡푒푟푎푙) Hidden Connector • Relative yaw angle (퐸푦푎푤) References Optimizer • Ego velocity set point (푉푠푒푡) Ego • Target lateral deviation (=0) Ego Vehicle Manipulated variables Vehicle Model • Acceleration (푎) Hidden Connector • Steering angle (훿) Measured disturbances MPC controller • MIO velocity (푉푚푖표) • Previewed road curvature (휌)

퐬퐮퐛퐣퐞퐜퐭 퐭퐨: 푫풓풆풍풂풕풊풗풆 ≥ 푫풔풂풇풆 Look ahead Act early! 풂풎풊풏 ≤ 풂 ≤ 풂풎풂풙 휹풎풊풏 ≤ 휹 ≤ 휹풎풂풙 66 Components

▪ Estimate lane center ▪ Estimate MIO (lead vehicle)

Inputs to MPC: For lateral control Inputs to MPC: For longitudinal control

Four cases are considered: 1) Both left and right lanes are detected 2) Left lane is detected 3) Right lane is detected ▪ MPC: Path following controller 4) No lane is detected 67 Path Following Control Block Model Predictive Control ToolboxTM

Driver setting Bicycle model parameters

Measurement

Virtual lane For lane change

Delay or latency in the system Disable distance keeping 68 Path Following Control Block Actuator limits

Tune MPC performance

69 Path Following Control Block

Change MPC design

70 Simulate controls with perception

Model Predictive Control ToolboxTM Automated Driving ToolboxTM Vehicle Dynamics BlocksetTM

Updated

71 Design lateral and longitudinal Model Predictive Controllers

Longitudinal Control Lateral Control Longitudinal + Lateral

Adaptive Cruise Control Lane Keeping Assist with Lane Following Control with with Sensor Fusion Lane Detection Sensor Fusion and Lane Automated Driving ToolboxTM Automated Driving ToolboxTM Detection Model Predictive Control Model Predictive Control Automated Driving ToolboxTM ToolboxTM ToolboxTM Model Predictive Control ToolboxTM Embedded Coder® Embedded Coder® Embedded Coder®

72 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

How can I How can I How can I synthesize scenarios discover and design integrate to test my designs? in new domains? with other environments?

73 ROS Toolbox - NEW!

▪ Communicate with ROS and ROS 2 nodes ▪ Multiplatform support ▪ Connect to live ROS data ▪ Replay logged data ▪ Generate standalone ROS nodes through code generation

74 Call C++, Python, and OpenCV from MATLAB

Call OpenCV & Call C++ Call Python OpenCV GPU

cv::Rect tw = ... cv::KeyPoint py.textwrap.TextWrapper(... cv::Size pyargs(... .mex .hpp .mlx 'initial_indent', '% ',... cv::Mat 'subsequent_indent','% ',... cv::Ptr 'width', int32(30))) ...

Import C++ Library Call Python from MATLAB Install and Use Computer Vision Functionality into MATLAB MATLAB® Toolbox OpenCV Interface MATLAB® Computer Vision System ToolboxTM R2014a OpenCV Interface Support Package Updated

75 Call C code from Simulink

Create buses from C Call C code Test and verify C code structs

Bring Custom Image Filter Import Structure and Custom C Code Verification Algorithms as Reusable Enumerated Types with Simulink Test Blocks in Simulink Simulink® Simulink TestTM Simulink® Simulink CoverageTM

76 Cross-release simulation through code generation

Previous Release Integrate Generated Code by Using Cross-Release Workflow ▪ Generate code from previous C release (R2010a or later) ▪ Import generated code as a block in current release ▪ Tune parameters ▪ Access internal signals Current Release crossReleaseImport Embedded Coder R2016a

77 Connect to third party tools

152 Interfaces to 3rd Party Modeling and Simulation Tools (as of March 2019)

78 Some common questions from automated driving engineers

Perception Simulation Integration

ROS CAN

Planning C/C++ Python

Cross Third Release Party Control

Synthesize scenarios Discover and design Integrate to test my designs in multiple domains with other environments

79 Thank You!