NVIDIA DRIVE December 2019 DRIVE PLATFORM Mass production
INDUSTRY / MARKET SAFETY
TECHNOLOGY REGULATIONS
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 2 THE DRIVE INITIATIVE End-to-End, Open Platform for Building Autonomous Vehicles
DGX Saturn V Constellation Xavier
DRIVE AV DRIVE IX Re-SIM
The NVIDIA DRIVE™ Platform provides all major AI technologies for Driving, Development and Validation - from driving & cockpit computers & software to data collection & DNN training, to simulation tools. The system is an open platform with multiple programming points. Car makers are free to use or develop as much as they like and take advantage of an ecosystem of hundreds of partners.
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 3 NVIDIA DRIVE END-TO-END PLATFORM
COLLECT DATA TRAIN MODELS SIMULATE DRIVE
Cars Pedestrians Path Cars Pedestrians Path
Lanes Signs Lights Lanes Signs Lights
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 4 DRIVE COMPUTER / ARCHITECTURE SCALABLE NOW & INTO THE FUTURE NCAP TO ROBOTAXI
5 DRIVE ROADMAP
ORIN
DRIVE Parker DRIVE PX2 DRIVE Xavier DRIVE Pegasus DRIVE Orin-X
1x Perf 20x Perf 23x Perf ~230x Perf >100x Perf 1x Efficiency 1x Efficiency 8x Efficiency 5x Efficiency ~20x Efficiency Samples – Q4’2015 Samples – Q2’2016 Samples – Q1’2018 Samples – Q3’2018 Samples – Mid 2021 Production – Q1’2017 Production – Q3’2016 Production – Q1’2019 Production – Q3’2019 Production - Q3’2022
Auto-Grade
Super Energy-Efficient 6 ORIN – HIGHEST PERF SOC EVER What is driving the need for increasing performance in AV?
Sensor Complexity - 20+ DNN Models New DNN Models- DNN GOPs/Frame More higher required for most CNN, RCNN, RNN increasing resolution demanding ODD cameras
Integration of Future Proof OTA new features DNN Precision / functions EUNCAP / SW Patch Accuracy Centralized With 360° Computer Perception
More perf equates to faster time to market
7 NVIDIA DRIVE One Architecture
XAVIER (NCAP, L2+/L3) PEGASUS (L4/RoboTaxis) ORIN
ORIN
One AV Architecture for L2 -> Robo Taxi
NVIDIA DRIVE AGX platform is an open platform; allows partners to do their own innovation
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 8 DRIVE ARCHITECTURE AUTOMOTIVE GRADE – INDUSTRIALIZED ECU’S
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 9 NVIDIA DRIVE SENSOR ECOSYSTEM
10 SENSOR ECOSYSTEM
IMAGING SENSOR / CAMERA LIDAR
AR0820, AR0144, AR0231, IMX390, IMX490, IMX424
RADAR GNSS/IMU
ARS430
SC13S, MTi-G-710, SPAN-IGM A1
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. https://developer.nvidia.com/drive/drive-core-sensors 11 NVIDIA DRIVE DRIVE SOFTWARE
12 NVIDIA DRIVE SOFTWARE ARCHITECTURE Open and modular software for AV development
DRIVE AV DRIVE IX
DRIVE Planning
DRIVE Mapping Visualization AI CoPilot AI Assistant
DRIVE Perception
DRIVEWORKS
DRIVE Networks DRIVE Calibration
DRIVE Core
DRIVE OS
DRIVE AGX DEVELOPER KITS DRIVE HYPERION (Xavier/Pegasus) (Reference Architecture)
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 13 NVIDIA DRIVE DRIVE OS
14 DRIVE AGX AV Software Platform
OEM/T1 Application 1
DriveWorks
DRIVE OS OS Drivers, USB, cuDNN TensorRT
Diagnostics File System, Network
Security Services Security
L1 Safety Services Safety L1
L3 L3 Safety Services L2 Safety Services
Data Data Distribution (DDS)
ES ES 3.2
Foundation Services Foundation
Power Power Management
EGL EGL 1.5
Open GL GL Open NvStreams
Linux Linux BSP NvMedia CUDA Communication Services Communication
Software Hypervisor
Hardware MCU Xavier
CAN & RS GMSL 10G/1G/BR Ethernet PMIC NVLINK PCIe NOR eMMC UFS NVME I/O Flexray 232 (via CSI-2)
Radar Lidar GPS IMU Scalability; Cameras TU104 +Xavier, NvME, or Optional Processor Hardware NVIDIA Software OS/3rd Party Software Notes: 1) For development only. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 15 Optional Services DRIVE AGX
AV Software Platform
1 1
1 DRIVE AV
DRIVE OS
1
OS Drivers, USB , 1 1 1 cuDNN TensorRT
Diagnostics File System, Network
Security Services Security
L1 Safety Services Safety L1
L3 L3 Safety Services L2 Safety Services
Data Data Distribution (DDS)
ES ES 3.2
Foundation Services Foundation
Power Power Management
EGL EGL 1.5
NvStreams Open GL Open
QNX BSP NvMedia CUDA Communication Services Communication
Software Hypervisor
Hardware MCU Xavier
CAN & RS GMSL 10G/1G/BR Ethernet PMIC NVLINK PCIe NOR eMMC UFS NVME I/O Flexray 232 (via CSI-2)
Scalability Radar Lidar GPS IMU Scalability; Cameras w/dGPU +Xavier, NvME, or Optional Processor Hardware NVIDIA Software OS/3rd Party Software NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Notes: 1) Not for use in production in a safety context. 16 Optional Services NVIDIA DRIVE AUTONOMOUS DRIVING APPLICATION
17 NVIDIA DRIVEWORKS SDK Data Abstraction | Calibration | Compute Modules
DRIVE AV DRIVE IX DRIVEWORKS
DRIVE Networks DRIVE Calibration Calibration DriveNet ClearSightNet OpenRoadNet Tools Self- PathNet MapNet LaneNet Calibration
WaitNet LightNet SignNet Egomotion
DRIVE Core Sensor Abstraction Image Processing DNN Frameworks Layer + Plugins
Vehicle IO Point Cloud Processing Tools (Recorder)
DRIVE OS DRIVE AGX DEVELOPER KITS DRIVE HYPERION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 18 NVIDIA DRIVEWORKS SDK Foundation for AV SW Development on NVIDIA DRIVE
SOFTWARE MODULES TOOLS
- Open, modularized library of functions - Software dev tools ranging from sensor - Optimized for DRIVE AGX data capture to calibration and - Easy to integrate and build upon visualization
DEVELOPER SUPPORT PRODUCTION GRADE
- Sample apps - Architected for automotive safety - Documentation, tutorials - Compliant with industry standards, (ISO - Active developer forum and community 26262/21448, MISRA)
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 19 DRIVE AV
Actuation
Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization
Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet
DNNs
Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration
MapWorks Sensor Abstraction
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 20 SCALABLE PLATFORM One architecture – Highway and Urban AV Driving
ON RAMP TO OFF RAMP URBAN: ADDRESS TO ADDRESS
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 21 ADVANCED FUNCTIONS Auto High Beam Clear Sight
Parking NCAP
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 DRIVE AV
Actuation
Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization
Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet
DNNs
Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration
MapWorks Sensor Abstraction
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 23 SENSOR SELF-CALIBRATION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 24 EGOMOTION
IMU/Odometry Ego-motion Radar Ego-motion
Camera Ego-motion (SLAM) LIDAR Ego-motion
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 25 DRIVE AV
Actuation
Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization
Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet
DNNs
Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration
MapWorks Sensor Abstraction
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 26 CAMERA OBSTACLE PERCEPTION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 27 OBJECT DETECTION SURROUND OBJECT TRACKING DNN DISTANCE
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 31 FUTURE MOTION PREDICTION (RNN) VELOCITY ESTIMATION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 33 VELOCITY ESTIMATION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 34 LIDAR – OBSTACLE PERCEPTION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 35 LIDAR – OBSTACLE PERCEPTION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 36 LIDAR FREESPACE
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 37 LIDAR MOTION DETECTION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 38 RADAR TRACKING
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 39 PATH REPRESENTATION (THE LANE GRAPH)
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 40 LANENET + TRACKING + FUSION
Tracking and Fusion Lane Assignment (OIPA)
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 41 PATHNET
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 42 FACTORED PILOTNET TURNS
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 43 PATH PERCEPTION ENSEMBLE WAIT CONDITIONS
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 45 INTERSECTION DETECTION
46 SIGN AND TRAFFIC LIGHT CLASSIFICATION
47 CAMERA BLINDNESS DETECTION PARKING SPACE DETECTION LIGHT SOURCE PERCEPTION Detects objects including vehicles, pedestrians, DriveNet DRIVE NETWORKS bicycles, traffic lights & signs Fully trained DNN solutions for DepthNet Dense depth inference at pixel level advanced situational awareness Obstacle Perception OpenRoadNet Detects drivable freespace around the vehicle
PredictionNet RNN to predict dynamic objects
LidarNet Lidar point cloud classification
LaneNet Detects and classifies lanes
MapNet Detects visual landmarks such as lanes & crosswalks Path Perception PilotNet Learn trajectories from human driver
PathNet Predicts full geometry of drivable paths
Classifies traffic signs detected by DriveNet, for US and SignNet EU Wait Condition Classifies traffic lights (color, solid, and arrows) LightNet Perception detected by DriveNet
WaitNet Detects intersections
ClearSightNet Determines if the camera view is blocked Assistance Functions AhbNet Auto high beam detection NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 51 NVIDIA DRIVE MAPPING Drive, Update and Create HD Maps
Map Localization Map Update Map Creation Using DRIVE Localization API Using DRIVE MapStream API Using DRIVE MapStream/MapServices API
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 52 LOCALIZATION STEPS HERE example
Detect Project Match Drive MapNet HERE HD Live Map DLA and CUDA DRIVE Software
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 53 CAMERA LOCALIZATION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 54
LIDAR/RADAR MAPPING+LOCALIZATION
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 56 MAP UPDATE STEPS TomTom example on I-85 in CA
TODO from TomTom
Detect Create MapStreams Convert Update Map MapNet Map update primitives MapStreams to Roadgrams HD map in the Cloud
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 57 MAPSTREAM CREATION IN THE CAR
Pole Trajectory Dashed Road boundary
Solid Traffic sign
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 58 VISUALIZING A CREATED MAPSTREAM
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 59
Route Planner PLANNING Lane Planner
Route Planner
Advanced Behavior Planner Lane Planner
Behavior Planner
Search Based Learned Prediction Planning Planning Basic Behavior Planner Latest Safe Safety Force Safety Arrival (LSA) Field Checks Image NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 61 SAFETY FORCE FIELD High-Level Architecture
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 62 SAFETY FORCE FIELD
Our vehicle is always perceiving other actors and obstacles.
The Safety Force Field is a computational module that monitors for overlap between our safety procedure and that of other actors.
The Safety Force Field provides a repelling constraint that prevents us from contributing to a collision.
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 63 SAFETY FORCE FIELD
Longitudinal
Lateral
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 64 NVIDIA DRIVE DRIVE EXPERIENCE
65 DRIVE IX – CONNECT TO SELF DRIVING Using AR and VR
Confidence View Perception view
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 66 EXAMPLE USE CASES
User Ext Face ID / Trunk opening / Driver/ Passenger Int face ID/ Car start Greeting/ Settings Auto adjust mirrors/seats Emotion detect Health/Wellness detect
PA: Personal Assistant / NLU PA: AV Driving modes Hand Gestures detection Distraction detect Drowsiness detect
Child /Pet detect Side traffic detect Front cross traffic detect Safe back-up Respond to sign commands
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Convenience Cockpit MMI Guardian Angel 67 DRIVE IX AI Toolkit for Intelligent Experience
AI Assistant AI CoPilot AR Viz
IX Server API Manager AV Perceptions Remote Rendering Engine Inference Hub IX (Rich Graphics, Video, Audio) IX Cabin Sensors Surround Sensor Camera Sensor Hub Hub Plugin Manager
IX Core Capabilities IX Speech Engine Partner plugins
Driver Surround Surround Remote Monitoring Monitoring View Experience ASR NLU TTS Face ID Emotions Partner Technology Head Pose, Gaze, Stitching, Pixel Distraction, Risk Assessment Level Drowsiness Segmentation OTA NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 68 NVIDIA DRIVE DRIVE SIM
69 DRIVE CONSTELLATION Virtual Reality AV Simulator
Hardware in the-Loop System Level Simulator Simulate Rare and Difficult Conditions Scalable Platform | Data Center Solution Timing Accurate and Bit Accurate Scenario Based Coverage
70 DRIVE CONSTELLATION ARCHITECTURE
Environment Traffic Vehicle Sensor Scenario Model Model Model Model Model
DRIVE Sim API DRIVE AV DRIVE IX Perception Mapping Planning
DRIVE Sim DRIVE CORE | DRIVE NETWORKS
DRIVE Constellation OS DRIVE OS
DRIVE Constellation Simulator DRIVE Constellation Vehicle
71 CONSTELLATION DATA CENTER WORKFLOW – CONT.
72 DRIVE PLATFORM Mass production
INDUSTRY / MARKET SAFETY
TECHNOLOGY REGULATIONS
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 73