Project Tango and Tegra K1

PROJECT TANGO AND TEGRA K1 BRINGING SPATIAL AWARENESS TO MOBILE Neil Trevett (Vice President of Mobile Ecosystems, NVIDIA) Wil Braithwaite (Senior Applied Engineer, NVIDIA) Visual Computing = Graphics AND Vision Processing

Enhanced sensor capability deepens Vision the interaction Processing between real and virtual worlds

Imagery

Data Real-time GPU Compute Research project on CUDA-enabled laptop High-Quality Reflections, Refractions, and Caustics in Augmented Reality and their Contribution to Visual Coherence P. Kán, H. Kaufmann, Institute of Software Technology and Interactive Graphics Systems, Vienna University of Technology, Vienna, Austria Processing https://www.youtube.com/watch?v=i2MEwVZzDaA

Need for advanced sensors and the GPU throughput to process them

Computational Face, Body and 3D Scene/Object Augmented Photography and Gesture Tracking Reconstruction Reality Videography

NVIDIA Tegra K1 processor Google ATAP Project 4GB of RAM to seed and inspire developers 128GB of storage - similar to Google Glass 1080p display Stock Android 4.4 WiFi, Bluetooth LE and 4G LTE 120 degree front camera 4MP Camera and Depth Sensor Motion-tracking Camera $1,024

Tango is intended to inspire and catalyze innovative applications

© Copyright NVIDIA 2014 - Page 4 What is special about Tango? Knows much more about its environment than normal mobile devices Advanced visual sensors – and Tegra K1 350GFlop GPU RGB/IR Sensor Wide-Angle IR + Flash Odometry Projector to process them It can compute its position (‘pose’) It can sense objects and construct 3D models of them

© Copyright NVIDIA 2014 - Page 5 Visual Odometry Accurate Pose Estimation Position (x,y,z) Pointing direction (i, j, k, rotation) Uses feature tracking and SLAM Accuracy: <1% Fuses inertial and visual odometry Significantly more accurate than using inertial sensors alone Wide angle lens, global shutter Tracking through fast movements Avoid rolling shutter artifacts GPU accelerated processing

© Copyright NVIDIA 2014 - Page 6 Depth Sensor Depth information for many points on a 2D picture for: 3D geometry and model (Mesh) Body position and movement Object recognition Mantis Vision Structured Light projector iR image captured by 4MP Camera GPU accelerated processing Analyses structured pattern deformation to extract depth information

© Copyright NVIDIA 2014 - Page 7 Tango Tablet Hardware Architecture No external image Camera CSI processors needed Front 120 Degree Reduces cost, power, latency Camera CSI Odometry RGBiR 4MP 30 FPS Strobes Tegra K1 ~3-5ms GPU processing/frame Camera CSI VGA BW GPU load: ~15% Wide Angle Depth Decoding MEMS Sensor SPI 5 FPS Sensors Hub ~13-15ms GPU IR processing/frame Projector GPU load: ~8%

© Copyright NVIDIA 2014 - Page 8 Tango Software Pipeline GPU processed sensor data made available to applications Java interfaces or native access, together with 3D engines such as Unity NVIDIA VisionWorks can be used for native vision processing Library of GPU accelerated vision primitives Use OpenVX for optimized execution of graphs of primitives

Camera Control Sensor Hub SW Use standard Android APIs GPU Sensor Processing

Time-stamping CUDA Feature Odometry

Tracking Client Camera HAL V3 Laser Control Core CUDA Depth Inertial Sensor Depth Camera Stack Decoding Client

© Copyright NVIDIA 2014 - Page 9 Metaio - IKEA Furniture Catalog Select catalog item, and display in own home 1 million users in Europe Depth camera enhances experience Absolute measurements and ground plane detection with no marker Environmental lighting Occlusion

© Copyright NVIDIA 2014 - Page 10 Matterport - Interior Space Capture Real Estate, Training, Historical Preservation Use dedicated camera now Uses Kinect 5 degree drift after 360 degrees Tango sensors will enable widespread adoption

3D Descriptor Search Database

Manufacturers provide 3D Object Descriptors - much more information than Amazon-style 2D search

Social Loop Object Photos -> Facebook Upload, View, Share Capture Videos -> YouTube and Comment 3D -> ?

3D Printing Print Inspire and (e.g. shapeways.com) Motivate