Automated Driving Data Chain Challenges

Maxime Flament, ERTICO – ITS Europe AUTOMATED DRIVING DATA CHAIN CHALLENGES

Flament | Automated Vehicle Symposium | July 2017 1 Tomorrow’s Situation: Sensors, Maps and Online Data The Vehicle Looks beyond 300m and Around the Corner

3. Extension of limited in-vehicle resources

2. Extended preview 1. Highly accurate map information Close preview: model provided and 10 minutes updated via the Backend

4. Fleet based data collection

Vehicle Sensor range 0-300m

GSV-Forum Automatisiertes Fahren October 2016 Public FF@ Continental AG 2 Automated driving data chain and ecosystem Reference architecture proposed by OADF N NDS A ------INFRASTRUCTURE ------VEHICLE ------S T Backend (one or multiple) operated by Map supplier / OEM / 3rd Party ADAS (Local) Sensors (V2x) provided) OEM-specific Backends Horizon Sensors Provider Live Map Updates N (Dynamic Data) T Live Map N Local Live Map Delivery Maps & Updates T (Dynamic Data) A Environmental Model N

N HD Map (Dyn.) Data Production HD Map Delivery A Analytics/Sensor Fusion Localization

Data Collection Strategy

Infotainment Map Delivery Infotainment Map N A

Systematic Data Community Data

Collection Store S S Data Store (Mobile Mapping) Data Collection (raw or pre-processed) S

Flament | Automated Vehicle Symposium | July 2017 3 INTRODUCTION TO ADASIS

Flament | Automated Vehicle Symposium | July 2017 4 ADASIS horizon along the map data chain

TPEG: Traffic Information

Map database NDS: Incremental Map Update In-vehicle systems Layers HMI

e. g. hazard spots

e. g. traffic information  Determination of location and most ADAS, Energy Management e. g. speed limits probable path (MPP) 30 Map data  Enrichment of MPP with Relevant information Digital map (static) (NDS Format) road information (e.g. for road ahead topography, speed (ADASIS Format) limits)  Conversion into ADASIS format

Flament | Automated Vehicle Symposium | July 2017 5 ADASIS horizon – evolution to the cloud

In-vehicle embedded software Cloud-based backend software

i. function horizon provider map Link to NDS Local

function horizon provider map map Link to NDS ii. Hybrid function horizon provider map map dynamic data Link to NDS & TPEG

Link to Sensoris, NDS & iii. function horizon provider map dynamic data

Cloud TPEG

i. ADASIS Horizon uses data from local map database and provides information for functions in the vehicle ii. ADASIS Horizon uses recent data from updated map database via cloud backend or live data from additional databases and provides information for functions in the vehicle iii. ADASIS Horizon uses cloud-based data and provides information for in-vehicle functions directly from the cloud

Flament | Automated Vehicle Symposium | July 2017 6 ADASIS V3 specification supports highly automated driving New ADASIS V3 specifications supports different aspects of autonomous driving

1. Support of HAD maps (NDS)

• Copyright: Here.com

Flament | Automated Vehicle Symposium | July 2017 7 ADASIS V3 specification supports highly automated driving Lane model & Geometry

Example: Highway Exit “switch from a map-centric view to a vehicle-centric view of the world to create a simpler representation”

Segmentation of road into stretches with Segmentation of road same lane characteristics 3 Lanes into stretches with 1stsame level =lane path characteristics description level detail: 1 Lane number1st level of = lanespath description for each segment level detail: number of lanes for each segment 3+1 Lanes

3 Lanes + shoulder

Lane Model & Connectivity: Logical Description similar to v2 Lane #3: Lane #2: nd 2 level = logical view normal normal Lane #1: level detail: Lane #1: ramp lane details and their connectivity normal

Lane #4: normal Lane #3: normal Lane #2: normal

Lane #3: Lane #2: Lane #1: normal normal normal

© Wikimedia Commons, licensed by CC-by-SA 2.5 Flament | Automated Vehicle Symposium | July 2017 10 ADASIS V3 specification supports highly automated driving Lane Model & Connectivity Geometry for Linear Objects: • Lane Boundaries Geometry for Linear Objects: • Guardrails • Lane Boundaries •3rdGuardrails level = geometry description level detail: 3geometryrd level = geometryof road, lane description and lines level detail: geometry of road, lane and lines lines and boundaries

Lane #3: Lane #2: Lane #1: normal normal normal

Flament | Automated Vehicle Symposium | July 2017 11 ADASIS Forum Membership & Steering Board

From 46 to 52 ADASIS Forum Members (updated 31 may 2017) Vehicle Manufacturers (16) ADAS Suppliers (15) Navigation System Suppliers (13) BMW Autonomos AISIN AW China FAW-RDC ** Continental Automotive * Alpine CRF (FCA) ** CTAG Autoliv Daimler * Denso Elektrobit Automotive Ford * dSPACE Garmin Ford-Otosan Fujitsu Ten (Europe) ** Harman Honda * Hitachi Mappers Co. ** Hyundai Motor Company Ibeo Mitsubishi Electric Europe Jaguar IPG MXNAVI Opel * LG Electronic NNG LLC Nissan Magna Electronic Europe Panasonic Renault Magneti Marelli Robert Bosch GmbH* Toyota Motor Corp. Novero Telenav Volkswagen TRW (ZF) Volvo Car Corp. Valeo Volvo Tech. Dev. Corp.

Map & Data Providers (8) * Steering Board Members AND AutoNavi Holding GeoDigital Automotive** Here * ** New Members since June 2016 Navinfo Co TomTom* Wuhan Kotei Informatics** Zenrin Flament | Automated Vehicle Symposium | July 2017 14 Automated driving data chain and ecosystem Reference architecture proposed by OADF N NDS A ------INFRASTRUCTURE ------VEHICLE ------S T Backend (one or multiple) operated by Map supplier / OEM / 3rd Party ADAS (Local) Sensors (V2x) provided) OEM-specific Backends Horizon Sensors Provider Live Map Updates N (Dynamic Data) T Live Map N Local Live Map Delivery Maps & Updates T (Dynamic Data) A Environmental Model N

N HD Map (Dyn.) Data Production HD Map Delivery A Analytics/Sensor Fusion Localization

Data Collection Strategy

Infotainment Map Delivery Infotainment Map N A

Systematic Data Community Data

Collection Store S S Data Store (Mobile Mapping) Data Collection (raw or pre-processed) S

Flament | Automated Vehicle Symposium | July 2017 15 INTRODUCTION TO SENSORIS

Flament | Automated Vehicle Symposium | July 2017 16 SENSORIS: Sensor Ingestion Interface Specification

Standardization of sensor data content Vehicle-to-Cloud / Cloud-to-Cloud o Sensor manufacturers o Car manufacturers o Location-based services provider o Mobile network operators Part of Open AutoDrive Forum o Focus on Sensor Data up- stream o Alignment with NDS, ADASIS, TISA

Flament | Automated Vehicle Symposium | July 2017 17 SENSORIS architecture: Multi-role model

Flament | Automated Vehicle Symposium | July 2017 18 Use Case: Real-time services

Traffic flow Hazard warnings Environmental Traffic signage Traffic incidents conditions

Flament | Automated Vehicle Symposium | July 2017 19 Use Case: Map maintenance

• Road geometry and attributes • Lane geometry and attributes • POI entries and exits • Road condition

Flament | Automated Vehicle Symposium | July 2017 20 Use Case: Statistical analysis

• Historical and real-time data analysis • Personal preference learning • POI recommendations

Flament | Automated Vehicle Symposium | July 2017 21 SENSORIS Members

ADAS manufacturers Navigation System Suppliers Vehicle manufacturers Elektrobit Automotive GmbH Audi AISIN AW Harman Daimler AG Continental Automotive GmbH Hyundai Mnsoft Jaguar Land Rover Limited DENSO NNG Volvo Car Fujitsu Ten (Europe) GmbH PIONEER Co. Other LG Electronics Robert Bosch Car Multimedia ICCS GmbH Valeo Comfort and Driving Assistance In process to join Sensor & Component Location content & Service providers BMW AG Manufacturers CTAG AutoNavi Software Co. Ltd. Qualcomm Delphi HERE Global B.V. Telecom & Cloud Infrasrtucture Ericsson Nissan INRIX Inc. Providers Tass International IBM NavInfo Co.Ltd. TeleNav Toyota TomTom International B.V. Trafikverket Zenrin VTT Wuhan kotei informatics Flament | Automated Vehicle Symposium | July 2017 22 Conclusions

• ADASIS and SENSORIS are both industry-wide specification initiatives facilitating the use of map data in automated vehicles • ADASIS V3 transforms the map information into relevant data for the Automated Driving functions • SENSORIS gathers vehicle sensor data to make it accessible in the cloud for various real-time, historical and statistical purposes • ADASIS and SENSORIS are part of the Open Auto Drive Forum

Flament | Automated Vehicle Symposium | July 2017 23 Flament | Automated Vehicle Symposium | July 2017 24 HERE Digital Infrastructure Technologies Autonomous Vehicle Symposium July 11, 2017 A history of transforming maps into location technology 2015 3 new Investors 2011 1st pure 2009 location 1st map for Predictive Cruise Control cloud High-precision data collection and map-building technology Use of sensor data for map building 2007 Community 2004 mapping 1st map for ADAS Offline maps 1st map for phone for mobile 1985 1994 1st map for Adaptive Navigation 1st map for in-car nav Cruise Control Technologies 1st map for web founded

Location & POIs Beyond roads Things People

Spatial/aerial

AssistanceAutomation Learning

• Built on the HERE Open Location Platform • Active pilots (Colorado DOT, Iowa DOT) to develop standards for infrastructure data

OEM backend

The OEM The edge backend cloud

OEM backend ERTICO/OADF*: SENSORIS data standard

Incident detection HERE • Smartphones, aftermarket devices or Location Cloud embedded solutions become a collective community – detecting changing road or environmental conditions • Incidents or event information may also be manually input from the community and authorities

Reporting & Location Cloud Analytics HERE Location • Receiving and ingesting incident data Cloud • Running analytics • Determining identification of an impact area

Distribution of Road Hazard Warnings HERE City Traffic • The impact area and incident Location Management Center message type are send to traffic Cloud management centers and back to the drivers across all connected infrastructure in the affected area

Intel® Architecture for Autonomous Driving July 2017 FIVE Things You Should Know about Autonomous VEHICLES

Workloads are Some critical System designs Intel powers Autonomous more than artificial algorithms will need to change hundreds of level 4 Vehicles intelligence (AI). haven’t even been down the road. and level 5 test are an End-to-End developed yet. vehicles. Solution

2 Building the Best Autonomous Brain

Work with Fewer Limits Design with Flexible, Scalable Intel Architecture

Greater agility to adapt to evolving system designs More flexibility to place compute Optimized performance per watt Future programmability with FPGAs Lower TCO with FPGAs over ASICs Faster software development Reduced training time in the data center

5 Intel, the Intel logo, Intel Atom, Intel Nervana, and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.

NVIDIA NVIDIA DGX-1 DRIVE PX 2

AI THAT MIMICS HUMAN BEHAVIOR

SELF-DRIVING TO STARBUCKS

Left Front 1 6 Left Side Camera Corner Camera

2 Center Front Camera

5 Rear Center Camera

Right Front 4 Right Side Camera 3 Corner Camera

MULTIPLE AI’S WORKING TOGETHER

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. ENABLING THE NEXT-GENERATION OF DRIVER ASSISTANCE NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. Thank you! [email protected] Recent progress of dynamic map in Japan

12nd July, 2017 Automated Vehicles Symposium 2017 Hilton San Francisco Union Square, USA

Satoru Nakajo, the University of Tokyo Contents of the presentation

0. Overview of SIP-adus

1. FOT 2017-2018

2. Standardization activities

3. Challenges we are facing (personal view)

Ref. Workshop in Tokyo

1 0. Overview of SIP-adus

SIP: Cross-ministerial Strategic Innovation Promotion Program  Start from FY2013.  The Council for Science, Technology and Innovation selects 11 projects.  Cross-ministerial Initiatives.  Promote focused, end-to-end research and development, from basic research to practical application and commercialization.  Total budget for FY2016 was ¥50 billion (around 500 million dollars).

Cyber-Security for Critical Infrastructures were added from FY2015

adus: Automated Driving for Universal Services ¥2.7 billion (FY2016) (around 27 million dollars) 2 0. Overview of SIP-adus

(I) Development and verification of (III) International automated driving system cooperation Road Transport system (1) Open research Driver facility Recognition Judgment Operation (2) Social acceptance (5) System Security (4) Driver Model (3) Technology Recognition Judgment Operation transfer

(1) Dynamic Map Area of Cooperation

(2) Prediction based (1) Enhanced local on ITS information traffic management Area of Competition Traffic environment Traffic (3) Sensors (2) Next generation transport system Vehicle (IV) Development (1) Traffic fatality reduction effect estimation method & national shared database for next (2) Macro and micro data analysis and simulation technology (3) Local traffic CO emission visualization technology generation 2 urban (II) Basic technologies to reduce transport traffic fatalities and congestion 3 0. Overview of SIP-adus

 Research results by SIP-adus http://en.sip-adus.jp/ (results of FY2014-FY2016 are available)  Specification on map features and attributes for ADS by JAMA http://www.jama.or.jp/safe/automated_driving/pdf/recommended_spec.pdf (Japanese) JAMA: Japan Automobile Manufacturers Association Inc.  A company “Dynamic Map Platform Inc. (DMP)” had established in June 2017

4 1. FOT 2017-2018

 SIP-adus is planning FOTs for 5 technological fields.  Dynamic map  HMI  Information Security  Pedestrian Accidents Reduction  Advanced Urban Transit

 Call for Participants (first call) was completed.  Call for participants (First call) by each technological fields including Dynamic Map  Sample Distribution of the map before the test would be available  Entry (in English) was possible from following HP http://www.nedo.go.jp/english/sip_ai2017.html (you can also access from http://en.sip-adus.jp/)

5 1. FOT 2017-2018

 Overview of Dynamic Map FOT

6 1. FOT 2017-2018

 Overview of Dynamic Map FOT

Outline of FOT

Schedule

7 2. Standardization activities  Promote standardization activities at ISO/TC204  Discussion for the importance of industrial specifications

Lane-level LR

GDF5.1

8 Scopes are ONLY a PERSONAL VIEW Lane-level LR 3. Challenges we are facing (personal view)

 Connection between static and dynamic (inc. semi.) data

 Map updates (before actual changes)

 Map definition for various roads including local streets

 Relationship between Dynamic Map and Navigation data

 Balance between Map and other key elements such as sensing devices, AI algorithm

Try to go forward by FOTs (real experiments) and discussions 9 Ref. Workshop in Tokyo

 Workshop 2017 will be held on 14th to 16th Nov. in Tokyo  Just after the 8th OADF meting (13th Nov.) in Tokyo  (almost of) all presentations will be done by or translated in English  Dynamic Map will be one of the main topic  Last day of the workshop will be only available for invited person (around 20 people for each topic, mainly for discussion)  Further details will be released at the following website http://en.sip-adus.jp/

10 END

[email protected]