1

TRANSIENT IMAGING Freshman Imaging Project 2014 2 Outline

❖ What is transient imaging? ❖ Previous work: MIT, UBC ❖ Possible applications of transient imaging ❖ Theory behind transient imaging ❖ Our goals ❖ Technology ❖ Work breakdown structure ❖ Project schedule 3

http://www.ted.com/talks/ramesh_raskar_a_camera_that_takes_one_trillion_frames_per_second?language=en http://www.cs.ubc.ca/labs/imager/tr/2013/TransientPMD/ 4 Previous Research

❖ MIT ➢ School of Architecture and Planning, Program in Media Arts and Sciences ➢ Ramesh Raskar, Ahmed Kirmani

❖ UBC ➢ Department of Computer Science, Imager Lab ➢ Felix Heide, Matthias Hullin 5 UBC’s RESEARCH ❖ used Photonic Device (Camboard Nano) in conjunction with laser diodes

http://www.cs.ubc. ca/labs/imager/tr/2013/Tran sientPMD/ 6

Theory Behind Transient Imaging 7 What is Time-of-Flight?

❖ Time of flight (ToF) measures the time required for pulses of laser light to travel from an emitter and reflect back to an image sensor 8 What is a PMD?

❖ A Photonic Mixer Device ❖ Does not measure specific time, measures the crosscorrelation of a reference signal and a modulated light source ❖ “Mixes” the signals 9 Setup

http://www.cs.ubc.ca/labs/imager/tr/2013/TransientPMD/HeideEtAl-LowBudgetTransientImaging-SIGGRAPH2013.pdf 10 11 Raw Data Needed

❖ System collects intensity and time for each pixel ❖ The intensity of each pixel at a given frequency and at a given time is the impulse response of light ❖ The impulse response is used to predict the passage of light through the scene ❖ ~100 frames per capture (3-5s) will be reconstructed using these predictions 12 Our System Data 13 The Basic Principle of Our System

❖ Our capture is an estimation ❖ MIT used streak cameras and femtosecond lasers ❖ We are using a ToF sensor and modulated light source to construct our own capture via our graphs of intensity over time 14 Components of Light

P = D + S + I D: direct reflection S: subsurface scattering I: interreflections and longer range scattering 15 Software

To break up the interactions and see them intermittently over time, we will be using the code provided by UBC which uses advanced optimization algorithms to solve for each component of our image 16 Our Goals

❖ Replicate UBC’s system ➢ PMDtechnologies Camboard Pico ➢ for Windows v2

❖ Advancements ➢ Microscopy ➢ Multiple sensors 17 Light Source - Kinect System ❖ Internal ➢ 860 nm according to Microsoft ➢ 840-860 nm found with spectrometer ❖ External ➢ 850 nm laser diodes ➢ Signal generator https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 ➢ Oscilloscope 18 19 Light Source - Kinect System ❖ Internal ➢ 860 nm according to Microsoft ➢ 840-870 nm found with spectrometer ❖ External ➢ 850 nm laser diodes ➢ Signal generator https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 ➢ Oscilloscope 20 Sensor - Kinect System

Resolution 524x424 pixels

Modulation Frequency 10-130 MHz (Average 80 MHz)

Field of View 70 (H) 60 (V) degrees

Depth Uncertainty < 0.5% of range

Distance Range 0.8-4.2 meters

Operating Wavelength 860 nm

Frame Rate Max 60 fps (Normally 30 fps) 21 Sensor - Kinect System

http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6756701&tag=1

22 Processing - Kinect Sensor

❖ Software ➢ Kinect for Windows Software Development Kit 2.0 (SDK) ➢ MetriCam Software Development Kit ❖ Processing ➢ Write an app for the Kinect 23 PMD System ❖ Light source ➢ Internal ■ 850nm ❖ Sensor ➢ IR lens ■ Accepts 850 nm ■ Can’t be removed ● Must use different light source than UBC 24 Sensor - PMD System

Resolution 160x120 pixels

Modulation Frequency average of 80 MHz

Field of View 82°x66°

Depth Uncertainty < 3mm @ 0.5meters

Distance Range 0.2 – 1 meters

Operating Wavelength 850nm 25 Sensor - PMD System 26 Processing - PMD System

❖ Software ➢ C/C++ SDK, Matlab SDK ❖ Coding ➢ Code from UBC, however we will need to make sure that we have the data that we will run through the code is in the correct format 27 Preliminary Tests

Running all the code with UBC’s data Testing illumination system with signal generator and oscilloscope and a photodiode 28 Budget

❖ PMD ➢ $700 ❖ Kinect (x 2) ➢ $150 each ❖ Illumination System ➢ $140 ❖ Graphics Card ➢ $50 29 Work Breakdown Structure PROJECT SCHEDULE 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

PRELIM DESIGN REVIEW ILLUMINATION SYSTEM Build Illumination System Verify Illumination Performance

KINECT-BASED SYSTEM SDK – Modulate Internal Illum. SDK – Extract Relevant Data Render Using Internal Illum Data Operate w/ External Illumination Render Using External Illum Data CRITICAL DESIGN REVIEW PMD-BASED SYSTEM Operate w/ External Illumination Render Using External Illum Data ADVANCED PERFORMANCE Conceptual Design Work Implement Advanced Concepts IMAGINERIT EXHIBIT 31 Support/ Resources

CIS PMDtec Microsoft Kinect Support University of British Columbia 32

Questions? 21 Propagation - Kinect System

https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 8 Uses for Transient Imaging

http://web.media.mit.edu/~raskar/cornar/ 34 Citations

[1] F. Heide et al., “Low-budget Transient Imaging using Photonic Mixer Devices,” Vancouver: The University of British Columbia. [2] M. Hullin. (2014, August). Building a Transient Camera. [Online]. Available: http://pulsr.info [3] R. Schwarte et al., “A new electro-optical mixing and correlating sensor: Facilities and Applications of the Photonic Mixer Device,” in Proc. SPIE 3100, 1997 © SPIE. doi: 10.1117/12.287751 31