1
TRANSIENT IMAGING Freshman Imaging Project 2014 2 Outline
❖ What is transient imaging? ❖ Previous work: MIT, UBC ❖ Possible applications of transient imaging ❖ Theory behind transient imaging ❖ Our goals ❖ Technology ❖ Work breakdown structure ❖ Project schedule 3
http://www.ted.com/talks/ramesh_raskar_a_camera_that_takes_one_trillion_frames_per_second?language=en http://www.cs.ubc.ca/labs/imager/tr/2013/TransientPMD/ 4 Previous Research
❖ MIT ➢ School of Architecture and Planning, Program in Media Arts and Sciences ➢ Ramesh Raskar, Ahmed Kirmani
❖ UBC ➢ Department of Computer Science, Imager Lab ➢ Felix Heide, Matthias Hullin 5 UBC’s RESEARCH ❖ used Photonic Mixer Device (Camboard Nano) in conjunction with laser diodes
http://www.cs.ubc. ca/labs/imager/tr/2013/Tran sientPMD/ 6
Theory Behind Transient Imaging 7 What is Time-of-Flight?
❖ Time of flight (ToF) measures the time required for pulses of laser light to travel from an emitter and reflect back to an image sensor 8 What is a PMD?
❖ A Photonic Mixer Device ❖ Does not measure specific time, measures the crosscorrelation of a reference signal and a modulated light source ❖ “Mixes” the signals 9 Setup
http://www.cs.ubc.ca/labs/imager/tr/2013/TransientPMD/HeideEtAl-LowBudgetTransientImaging-SIGGRAPH2013.pdf 10 11 Raw Data Needed
❖ System collects intensity and time for each pixel ❖ The intensity of each pixel at a given frequency and at a given time is the impulse response of light ❖ The impulse response is used to predict the passage of light through the scene ❖ ~100 frames per capture (3-5s) will be reconstructed using these predictions 12 Our System Data 13 The Basic Principle of Our System
❖ Our capture is an estimation ❖ MIT used streak cameras and femtosecond lasers ❖ We are using a ToF sensor and modulated light source to construct our own capture via our graphs of intensity over time 14 Components of Light
P = D + S + I D: direct reflection S: subsurface scattering I: interreflections and longer range scattering 15 Software
To break up the interactions and see them intermittently over time, we will be using the code provided by UBC which uses advanced optimization algorithms to solve for each component of our image 16 Our Goals
❖ Replicate UBC’s system ➢ PMDtechnologies Camboard Pico ➢ Microsoft Kinect for Windows v2
❖ Advancements ➢ Microscopy ➢ Multiple sensors 17 Light Source - Kinect System ❖ Internal ➢ 860 nm according to Microsoft ➢ 840-860 nm found with spectrometer ❖ External ➢ 850 nm laser diodes ➢ Signal generator https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 ➢ Oscilloscope 18 19 Light Source - Kinect System ❖ Internal ➢ 860 nm according to Microsoft ➢ 840-870 nm found with spectrometer ❖ External ➢ 850 nm laser diodes ➢ Signal generator https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 ➢ Oscilloscope 20 Sensor - Kinect System
Resolution 524x424 pixels
Modulation Frequency 10-130 MHz (Average 80 MHz)
Field of View 70 (H) x 60 (V) degrees
Depth Uncertainty < 0.5% of range
Distance Range 0.8-4.2 meters
Operating Wavelength 860 nm
Frame Rate Max 60 fps (Normally 30 fps) 21 Sensor - Kinect System
http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6756701&tag=1
22 Processing - Kinect Sensor
❖ Software ➢ Kinect for Windows Software Development Kit 2.0 (SDK) ➢ MetriCam Software Development Kit ❖ Processing ➢ Write an app for the Kinect 23 PMD System ❖ Light source ➢ Internal ■ 850nm ❖ Sensor ➢ IR lens ■ Accepts 850 nm ■ Can’t be removed ● Must use different light source than UBC 24 Sensor - PMD System
Resolution 160x120 pixels
Modulation Frequency average of 80 MHz
Field of View 82°x66°
Depth Uncertainty < 3mm @ 0.5meters
Distance Range 0.2 – 1 meters
Operating Wavelength 850nm 25 Sensor - PMD System 26 Processing - PMD System
❖ Software ➢ C/C++ SDK, Matlab SDK ❖ Coding ➢ Code from UBC, however we will need to make sure that we have the data that we will run through the code is in the correct format 27 Preliminary Tests
Running all the code with UBC’s data Testing illumination system with signal generator and oscilloscope and a photodiode 28 Budget
❖ PMD ➢ $700 ❖ Kinect (x 2) ➢ $150 each ❖ Illumination System ➢ $140 ❖ Graphics Card ➢ $50 29 Work Breakdown Structure PROJECT SCHEDULE 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
PRELIM DESIGN REVIEW ILLUMINATION SYSTEM Build Illumination System Verify Illumination Performance
KINECT-BASED SYSTEM SDK – Modulate Internal Illum. SDK – Extract Relevant Data Render Using Internal Illum Data Operate w/ External Illumination Render Using External Illum Data CRITICAL DESIGN REVIEW PMD-BASED SYSTEM Operate w/ External Illumination Render Using External Illum Data ADVANCED PERFORMANCE Conceptual Design Work Implement Advanced Concepts IMAGINERIT EXHIBIT 31 Support/ Resources
CIS PMDtec Microsoft Kinect Support University of British Columbia 32
Questions? 21 Propagation - Kinect System
https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725 8 Uses for Transient Imaging
http://web.media.mit.edu/~raskar/cornar/ 34 Citations
[1] F. Heide et al., “Low-budget Transient Imaging using Photonic Mixer Devices,” Vancouver: The University of British Columbia. [2] M. Hullin. (2014, August). Building a Transient Camera. [Online]. Available: http://pulsr.info [3] R. Schwarte et al., “A new electro-optical mixing and correlating sensor: Facilities and Applications of the Photonic Mixer Device,” in Proc. SPIE 3100, 1997 © SPIE. doi: 10.1117/12.287751 31