Nvidia Drive Update
Total Page:16
File Type:pdf, Size:1020Kb
NVIDIA DRIVE December 2019 DRIVE PLATFORM Mass production INDUSTRY / MARKET SAFETY TECHNOLOGY REGULATIONS NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 2 THE DRIVE INITIATIVE End-to-End, Open Platform for Building Autonomous Vehicles DGX Saturn V Constellation Xavier DRIVE AV DRIVE IX Re-SIM The NVIDIA DRIVE™ Platform provides all major AI technologies for Driving, Development and Validation - from driving & cockpit computers & software to data collection & DNN training, to simulation tools. The system is an open platform with multiple programming points. Car makers are free to use or develop as much as they like and take advantage of an ecosystem of hundreds of partners. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 3 NVIDIA DRIVE END-TO-END PLATFORM COLLECT DATA TRAIN MODELS SIMULATE DRIVE Cars Pedestrians Path Cars Pedestrians Path Lanes Signs Lights Lanes Signs Lights NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 4 DRIVE COMPUTER / ARCHITECTURE SCALABLE NOW & INTO THE FUTURE NCAP TO ROBOTAXI 5 DRIVE ROADMAP ORIN DRIVE Parker DRIVE PX2 DRIVE Xavier DRIVE Pegasus DRIVE Orin-X 1x Perf 20x Perf 23x Perf ~230x Perf >100x Perf 1x Efficiency 1x Efficiency 8x Efficiency 5x Efficiency ~20x Efficiency Samples – Q4’2015 Samples – Q2’2016 Samples – Q1’2018 Samples – Q3’2018 Samples – Mid 2021 Production – Q1’2017 Production – Q3’2016 Production – Q1’2019 Production – Q3’2019 Production - Q3’2022 Auto-Grade Super Energy-Efficient 6 ORIN – HIGHEST PERF SOC EVER What is driving the need for increasing performance in AV? Sensor Complexity - 20+ DNN Models New DNN Models- DNN GOPs/Frame More higher required for most CNN, RCNN, RNN increasing resolution demanding ODD cameras Integration of Future Proof OTA new features DNN Precision / functions EUNCAP / SW Patch Accuracy Centralized With 360° Computer Perception More perf equates to faster time to market 7 NVIDIA DRIVE One Architecture XAVIER (NCAP, L2+/L3) PEGASUS (L4/RoboTaxis) ORIN ORIN One AV Architecture for L2 -> Robo Taxi NVIDIA DRIVE AGX platform is an open platform; allows partners to do their own innovation NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 8 DRIVE ARCHITECTURE AUTOMOTIVE GRADE – INDUSTRIALIZED ECU’S NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 9 NVIDIA DRIVE SENSOR ECOSYSTEM 10 SENSOR ECOSYSTEM IMAGING SENSOR / CAMERA LIDAR AR0820, AR0144, AR0231, IMX390, IMX490, IMX424 RADAR GNSS/IMU ARS430 SC13S, MTi-G-710, SPAN-IGM A1 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. https://developer.nvidia.com/drive/drive-core-sensors 11 NVIDIA DRIVE DRIVE SOFTWARE 12 NVIDIA DRIVE SOFTWARE ARCHITECTURE Open and modular software for AV development DRIVE AV DRIVE IX DRIVE Planning DRIVE Mapping Visualization AI CoPilot AI Assistant DRIVE Perception DRIVEWORKS DRIVE Networks DRIVE Calibration DRIVE Core DRIVE OS DRIVE AGX DEVELOPER KITS DRIVE HYPERION (Xavier/Pegasus) (Reference Architecture) NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 13 NVIDIA DRIVE DRIVE OS 14 DRIVE AGX AV Software Platform OEM/T1 Application 1 DriveWorks DRIVE OS OS Drivers, USB, cuDNN TensorRT Diagnostics File System, Network Security Services Security L1 Safety Services Safety L1 L3 L3 Safety Services L2 Safety Services Data Data Distribution (DDS) ES 3.2 Foundation Services Foundation Power Management EGL EGL 1.5 Open GL GL Open NvStreams Linux Linux BSP NvMedia CUDA Communication Services Communication Software Hypervisor Hardware MCU Xavier CAN & RS GMSL 10G/1G/BR Ethernet PMIC NVLINK PCIe NOR eMMC UFS NVME I/O Flexray 232 (via CSI-2) Radar Lidar GPS IMU Scalability; Cameras TU104 +Xavier, NvME, or Optional Processor Hardware NVIDIA Software OS/3rd Party Software Notes: 1) For development only. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 15 Optional Services DRIVE AGX AV Software Platform 1 1 1 DRIVE AV DRIVE OS 1 OS Drivers, USB , 1 1 1 cuDNN TensorRT Diagnostics File System, Network Security Services Security L1 Safety Services Safety L1 L3 L3 Safety Services L2 Safety Services Data Data Distribution (DDS) ES 3.2 Foundation Services Foundation Power Management EGL EGL 1.5 NvStreams Open GL Open QNX BSP NvMedia CUDA Communication Services Communication Software Hypervisor Hardware MCU Xavier CAN & RS GMSL 10G/1G/BR Ethernet PMIC NVLINK PCIe NOR eMMC UFS NVME I/O Flexray 232 (via CSI-2) Scalability Radar Lidar GPS IMU Scalability; Cameras w/dGPU +Xavier, NvME, or Optional Processor Hardware NVIDIA Software OS/3rd Party Software NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Notes: 1) Not for use in production in a safety context. 16 Optional Services NVIDIA DRIVE AUTONOMOUS DRIVING APPLICATION 17 NVIDIA DRIVEWORKS SDK Data Abstraction | Calibration | Compute Modules DRIVE AV DRIVE IX DRIVEWORKS DRIVE Networks DRIVE Calibration Calibration DriveNet ClearSightNet OpenRoadNet Tools Self- PathNet MapNet LaneNet Calibration WaitNet LightNet SignNet Egomotion DRIVE Core Sensor Abstraction Image Processing DNN Frameworks Layer + Plugins Vehicle IO Point Cloud Processing Tools (Recorder) DRIVE OS DRIVE AGX DEVELOPER KITS DRIVE HYPERION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 18 NVIDIA DRIVEWORKS SDK Foundation for AV SW Development on NVIDIA DRIVE SOFTWARE MODULES TOOLS - Open, modularized library of functions - Software dev tools ranging from sensor - Optimized for DRIVE AGX data capture to calibration and - Easy to integrate and build upon visualization DEVELOPER SUPPORT PRODUCTION GRADE - Sample apps - Architected for automotive safety - Documentation, tutorials - Compliant with industry standards, (ISO - Active developer forum and community 26262/21448, MISRA) NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 19 DRIVE AV Actuation Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet DNNs Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration MapWorks Sensor Abstraction NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 20 SCALABLE PLATFORM One architecture – Highway and Urban AV Driving ON RAMP TO OFF RAMP URBAN: ADDRESS TO ADDRESS NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 21 ADVANCED FUNCTIONS Auto High Beam Clear Sight Parking NCAP NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 DRIVE AV Actuation Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet DNNs Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration MapWorks Sensor Abstraction NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 23 SENSOR SELF-CALIBRATION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 24 EGOMOTION IMU/Odometry Ego-motion Radar Ego-motion Camera Ego-motion (SLAM) LIDAR Ego-motion NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 25 DRIVE AV Actuation Behavior Planner Control Cloud Mapping Search Based Planning AI Based Prediction Learned Planning Lane Route Planner Safety Force Field Safety Checks LSA Image Planner HD Map Local World Model World Model Localization Map Perception Obstacle Perception Path Perception Wait Perception RoadRunner Frame Packaging and Map Creation DriveNet PilotNet Serialization and updates DriveNet OpenRoadNet LaneNet WaitNet SubModule Motion/Structure PathNet MapNet DNNs Egomotion Estimation Drive AV Low Level Processing DriveWorks Self-Calibration MapWorks Sensor Abstraction NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 26 CAMERA OBSTACLE PERCEPTION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 27 OBJECT DETECTION SURROUND OBJECT TRACKING DNN DISTANCE NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 31 FUTURE MOTION PREDICTION (RNN) VELOCITY ESTIMATION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 33 VELOCITY ESTIMATION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 34 LIDAR – OBSTACLE PERCEPTION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 35 LIDAR – OBSTACLE PERCEPTION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 36 LIDAR FREESPACE NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 37 LIDAR MOTION DETECTION NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 38 RADAR TRACKING NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 39 PATH REPRESENTATION (THE LANE GRAPH) NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 40 LANENET + TRACKING + FUSION Tracking and Fusion Lane Assignment (OIPA) NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 41 PATHNET NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 42 FACTORED PILOTNET TURNS NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 43 PATH PERCEPTION ENSEMBLE WAIT CONDITIONS NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 45 INTERSECTION DETECTION 46 SIGN AND TRAFFIC LIGHT CLASSIFICATION 47 CAMERA BLINDNESS DETECTION PARKING SPACE DETECTION LIGHT SOURCE PERCEPTION Detects objects including vehicles, pedestrians, DriveNet DRIVE NETWORKS bicycles, traffic lights & signs Fully trained DNN solutions for DepthNet Dense depth inference at pixel level advanced situational awareness Obstacle Perception OpenRoadNet Detects drivable freespace around the vehicle PredictionNet RNN to predict dynamic objects LidarNet Lidar point cloud classification LaneNet Detects and classifies lanes MapNet Detects visual landmarks such as lanes &