Powering AI and Automotive Applications with the MIPI Camera Interface Agenda
Total Page:16
File Type:pdf, Size:1020Kb
Hyoung-Bae Choi Synopsys Powering AI and Automotive Applications with the MIPI Camera Interface Agenda Adoption of MIPI CSI-2℠ Image sensors beyond mobile AI and automotive examples CSI-2 interface overview Meeting reliability requirements of automotive applications Supporting artificial intelligence (AI) applications Summary © 2018 MIPI Alliance, Inc. 2 MIPI Specifications in New Applications Automotive, IoT / Wearables, Virtual / Augmented Reality © 2018 MIPI Alliance, Inc. 3 Industrial & Surveillance Applications Using MIPI CSI-2 Image Sensors © 2018 MIPI Alliance, Inc. 4 Example of MIPI in an Automotive Application MIPI CSI-2 Image Sensors & DSI Display MIPI CSI-2 Image Sensors Front Camera Module Vbat Power Supply Front Camera Left Camera MIPI DSI Right Display Module Camera Display Proprietary, LVDS or Right Camera MPU Left Ethernet Switch Module Camera CAN Interface Rear Camera Module Flash LVDS or Rear Camera DRAM Memory Ethernet Link Other Camera Module Other Camera Module © 2018 MIPI Alliance, Inc. 5 Safety-Critical ADAS Applications Require ISO 26262 Functional Safety Compliance and ASIL Certification Electronics failure can have hazardous impact Emergency braking Pedestrian detection Collision avoidance ≠ © 2018 MIPI Alliance, Inc. 6 MIPI Specs for Automotive Applications Infotainment Vehicle Networks & V2X • Real time video & data network • Gateways • Navigation • Telematics • Audio/Video • V2V • Entertainment • V2I • Security Driver Information Driver Assistance • Parking assist • Instrument clusters • Lane departure warning & • Voice recognition Lane keep aid • Hi-def displays • Pedestrian detection & • Surround view correction • Automatic emergency braking © 2018 MIPI Alliance, Inc. 7 Centralized ADAS Domain Controller SoC Architecture Encompassing Numerous IP • Interfaces: LPDDR4/4x, Ethernet SATA/ CAN, CAN- LPDDR4 PCIe AVB/TSN eSATA FD ARC HS Ethernet AVB/TSN, MIPI, HDMI, Logic Libraries FlexRay PCI Express, SATA, ADC ARC EV Vision Embedded Memories CPU Processor • Embedded vision MOST Data path • Security AMBA Interconnect HDMI • Sensor fusion SD/eMMC Radar DSP Sensor & Control IP Security Subsystem Graphics eDP • Going to advanced process SPI/QSPI Core ARC EM CPU ADCs nodes I2C SPI DACs MIPI GPIO UART • Requires functional safety © 2018 MIPI Alliance, Inc. 8 Key Requirements of Automotive-Grade IP Reduce Risk and Accelerate Qualification for Automotive SoCs Accelerate ISO 26262 functional safety assessments to Functional Safety help ensure designers reach target ASIL levels Reduce risk & development time for Reliability AEC-Q100 qualification of SoCs Quality Meet quality levels required for automotive applications © 2018 MIPI Alliance, Inc. 9 Artificial Intelligence • Artificial Intelligence Artificial Intelligence mimics human Mimics human behavior behavior Machine Learning • Machine learning uses Uses advanced statistical algorithms to improve AI advanced statistical models to find patterns Regression & results • Deep learning is a Bayesian Deep Learning specialized subset of Clustering Training of neural networks machine learning using Decision Trees neural networks data Vector Machines to recognize patterns Convolutional Recurrent Neural Networks © 2018 MIPI Alliance, Inc. 10 Edge Inference Connectivity for Deep Learning Cloud Connectivity • Ethernet Memory Performance • DDR/LPDDR Chip-to-Chip Connectivity • HBM2 • PCI Express • Embedded Memories & Logic Libraries • CCIX Super Image Resolution • USB/DP Vision • HDMI • MIPI CSI-2 & MIPI D-PHY Audio • USB • PDM/I2S • ARC Data Fusion IP Subsystem Wireless Sensor Connectivity Sensor Connectivity • Bluetooth 5 Link Layer & PHY • MIPI I3C Controller • 802.15.4 MAC & PHY • ARC Sensor and Control IP Subsystem © 2018 MIPI Alliance, Inc. 11 Deep Learning SoC Challenges Unique Requirements for Processing, Memory, Connectivity • Heterogeneous processing (scalar, vector, neural network) • Massively parallel, matrix multiplication (neural network) Specialized Processing • Model compression via pruning and quantization – (Increases irregular compute intensity and memory accesses) • Capacity and bandwidth constraints Memory Performance • Cache coherency requirements • Advanced processes maximize on-chip SRAM to reduce data movement • Reliable and configurable connectivity to AI data centers • Real-time interface to sensors, images, audio, cloud, and more Real-Time Connectivity • Reduced energy via power management features and FinFET technologies © 2018 MIPI Alliance, Inc. 12 MIPI CSI-2 Specification © 2018 MIPI Alliance, Inc. 13 MIPI Camera Serial Interface 2 (CSI-2) Key Improvements; From Mobile to Imaging & Vision CSI-2 Specification v1.x v2.x v3.X TBD version Approval Date 29-Nov’05 28-Mar’17 Target: Q4’18 Target: Q4’19 • First release. • D-PHY v2.1 support • D-PHY v3.0 support • GLD • C-PHY v1.2 support • C-PHY v2.0 support • sROI - Phase 2 • RAW 16/20 & DPCM 12-10-12 • RAW24 • AR/VR vX.0 • PPI width extension (up to 32) • EoTp • Security • LRTE • USL • Functional Safety • Scrambling • sROI - Phase 1 • Sync Image Sensor • Extension of Virtual Channel • Unified Packet Header • A-PHY 14-Feb’11 vX.01 • D-PHY℠ v1.0 support 14-Feb’11 09-Apr’18 • D-PHY 1.0 support • I3C SDR & HDR_DRR support vX.1 • RAW24 • EoTp • Interleaved Encryption 10-Sep’14 vX.2 • D-PHY v1.2 support 07-Oct’14 vX.3 • C-PHY v1.0 support © 2018 MIPI Alliance, Inc. 14 MIPI CSI-2 Over D-PHY Packet Builder Data Format Payload byte size Data CRC processing Definition V D WC ECC CRC C T Virtual Channel Identification ECC protecting the header CSI-2 Receiver CSI-2 Transmitter Frame Buffer Packet Lane Clk+ Clk+ Lane Packet Frame Buffer Decoder Merger Distribution Builder Clk- D- D- Clk- L0+ P P L0+ D-PHY HS Burst L0- H H L0- D-PHY L1+ Y Y L1+ HS Burst CSI-2 CSI-2 L1- L1- CSI-2 CSI-2 Packet Packet Packet Packet CSI-2 Device CSI-2 Host CCI Master CCI Slave SCL SCL SDA SDA © 2018 MIPI Alliance, Inc. 15 New Imaging Features for Automotive and Other Applications • RAW-16 and RAW-20 color depth • Latency Reduction and Transport Efficiency (LRTE) feature • Differential Pulse Code Modulation (DPCM) 12-10-12 compression • Scrambling to reduce Power Spectral Density (PSD) emissions • Expanded number of virtual channels from 4 to 32 Image Quality/HDR - Latency - Reliability - Aggregation Source: MIPI Alliance © 2018 MIPI Alliance, Inc. 16 RAW-16 & RAW-20 Color Depths • CSI-2 v1.3 color depths are sufficient for Mobile. Visually, there is almost no change between RAW14 and RAW16/20 • RAW-16 and RAW-20 color depth bring advanced vision capabilities to Automotive and Industrial applications – Improves image capture when the environment changes suddenly and dramatically, for example in a big change in lighting condition Raw 4 Vs Raw 8 © 2018 MIPI Alliance, Inc. 17 Up to 32 Virtual Channels • To accommodate the larger number of image sensors and their multiple data types • To support multi-exposure and multi-range sensor fusion for Advanced Driver Assistance Systems Virtual Channel 0 – Line 0 Virtual Channel 0 – Line 1 Virtual Channel 0 – Line 2 Virtual Channel 0 – Line 3 Virtual Channel 0 – Line 4 Virtual Channel 0 – Line N Virtual Channel 1 – Line 0 Virtual Channel 1 – Line 1 Virtual Channel 1 – Line 2 Virtual Channel 1 – Line 3 Virtual Channel 1 – Line 4 Virtual Channel 1 – Line M © 2018 MIPI Alliance, Inc. 18 Added Latency Reduction & Transport Efficiency (LRTE) • LRTE reduces frame transport latency & leakage power due to frequent "High Speed - Low Power” transitions. • This will enhance image sensor aggregation and multi exposure for real- time perception and decision making applications LPS SoT SP EoT LPS SoT PH Data PF EoT LPS SoT PH Data PF EoT LPS SoT SP EoT LPS Latency reduced Leakage power ALP S ASoT SP EPD PH Data PF EPD PH Data PF EPD SP AEoT ALP S reduced © 2018 MIPI Alliance, Inc. 19 Scrambling and New Compression Scheme • Galois Field Scrambling reduces power spectral density (PSD) emissions – Minimizes PSD emissions from aggressor components, which are particularly beneficial when placed near sensitive receiver New DPCM 12-10-12 compression to further boost • CSI-2 over D-PHY PSD emission reduction image quality – Superior SNR using reduced bandwidth PHY – Removes more compression artifacts when comparing with previous MPI CSI-2 v1.3 compression mode DPCM 12-8-12 vs DPCM 12-10-12 © 2018 MIPI Alliance, Inc. 20 MIPI D-PHY Architecture The Popular Physical Layer for MIPI CSI-2 and DSI Protocols • Synchronous Forwarded DDR clock link architecture • One clock and multiple data lanes configuration • Static/dynamic de-skew supported through Two Data Lane Configuration calibration • Calibration hand-shake not supported • No encoding overhead • Low-power and high-speed modes • Primarily targeting camera and display • Spread spectrum clocking supported for EMI/EMC considerations • Large eco-system, proven in millions of phones, cars © 2018 MIPI Alliance, Inc. 21 Benefits and Evolution of MIPI D-PHY • Higher data rate enables ultra- high-definition cameras and displays • Easier adaption of newer technologies • Backward compatible • Reliable with sufficient margins • New specs augment existing ecosystem • Growing market applications and segments • Longer channel length © 2018 MIPI Alliance, Inc. 22 MIPI I3C ℠ Specification © 2018 MIPI Alliance, Inc. 23 MIPI I3C Overview I/Os reduced to just two!! • Two wire serial Interface up to 12.5 MHz • Supports Legacy I2C Slave Devices Accelerometer Magnetometer SENSOR_CLK • I3C Single Data Rate (SDR) Mode SENSOR_DATA • I3C High Data Rate (HDR) Modes Accelerometer • In-band