Lensless Imaging with Compressive Ultrafast Sensing Guy Satat, Matthew Tancik and Ramesh Raskar
Total Page:16
File Type:pdf, Size:1020Kb
1 Lensless Imaging with Compressive Ultrafast Sensing Guy Satat, Matthew Tancik and Ramesh Raskar Abstract—Lensless imaging is an important and challenging time: the physics-based approach is done in one shot (i.e. all problem. One notable solution to lensless imaging is a single the sensing is done in parallel). The single pixel camera and pixel camera which benefits from ideas central to compressive its variants require hundreds of consecutive acquisitions, which sampling. However, traditional single pixel cameras require many illumination patterns which result in a long acquisition process. translates into a substantially longer overall acquisition time. Here we present a method for lensless imaging based on compres- Recently, time-resolved sensors enabled new imaging ca- sive ultrafast sensing. Each sensor acquisition is encoded with a pabilities. Here we consider a time-resolved system with different illumination pattern and produces a time series where pulsed active illumination combined with a sensor with a time is a function of the photon’s origin in the scene. Currently time resolution on the order of picoseconds. Picosecond time available hardware with picosecond time resolution enables time tagging photons as they arrive to an omnidirectional sensor. resolution allows distinguishing between photons that arrive This allows lensless imaging with significantly fewer patterns from different parts of the target with mm resolution. The compared to regular single pixel imaging. To that end, we develop sensor provides more information per acquisition (compared a framework for designing lensless imaging systems that use to regular pixel), and so fewer masks are needed. Moreover, the ultrafast detectors. We provide an algorithm for ideal sensor time-resolved sensor is characterized by a measurement matrix placement and an algorithm for optimized active illumination patterns. We show that efficient lensless imaging is possible that enables us to optimize the active illumination patterns and with ultrafast measurement and compressive sensing. This paves reduce the required number of masks even further. the way for novel imaging architectures and remote sensing in Currently available time-resolved sensors allow a wide extreme situations where imaging with a lens is not possible. range of potential implementations. For example, Streak cam- eras provide picosecond or even sub-picosecond time resolu- tion [4], however they suffer from poor sensitivity. Alterna- I. INTRODUCTION tively, Single Photon Avalanche Photodiode (SPAD) are com- RADITIONAL imaging is based on lenses that map the patible with standard CMOS technology [5] and allow time T scene plane to the sensor plane. In this physics-based tagging with resolutions on the order of tens of picoseconds. approach the imaging quality depends on parameters such as These devices are available as a single pixel or in pixel arrays. lens quality, numerical aperture, density of the sensor array and In this paper we present a method that leverages both time- pixel size. Recently it has been challenged by modern signal resolved sensing and compressive sensing. The method enables processing techniques. Fundamentally the goal is to transfer lensless imaging for reflectance recovery with fewer illumi- most of the burden of imaging from high quality hardware nation patterns compared to traditional single pixel cameras. to computation. This is known as computational imaging, in This relaxed requirement translates to a shorter overall acqui- which the measurement encodes the target features; these are sition time. The presented framework provides guidelines and later computationally decoded to produce the desired image. decision tools for designing time-resolved lensless imaging Furthermore, the end goal is to completely eliminate the need systems. In this framework the traditional single pixel camera for high quality lenses, which are heavy, bulky, and expensive. is one extreme design point which minimizes the cost with One of the key workhorses in computational imaging is simple hardware, but requires many illumination patterns (long compressive sensing (CS) [1], [2]. For example, CS enabled acquisition time). Better hardware reduces the acquisition time the single pixel camera [3], which demonstrated imaging with with fewer illumination patterns at the cost of complexity. a single pixel that captured scene information encoded with We provide sensitivity analysis of reconstruction quality to a spatial light modulator (SLM). The pixel measurement is a changes in various system parameters. Simulations with sys- set of consecutive readings, with different SLM patterns. The tem parameters chosen based on currently available hardware scene is then recovered using compressive deconvolution. indicate potential savings of up to 50× fewer illumination Broadly, the traditional imaging and single pixel camera patterns compared to traditional single pixel cameras. demonstrate two extremes: traditional cameras use a pure hardware approach whereas single pixel cameras minimize A. Contributions the requirement for high quality hardware using modern The contributions presented here can be summarized as: signal processing. There are many trade-offs between the two 1) Computational imaging framework for lensless imaging approaches. One notable difference is the overall acquisition with compressive time-resolved measurement, 2) Analysis of a time-resolved sensor as an imaging pixel, G. Satat, M. Tancik and R. Raskar are with the Media Lab, Mas- sachusetts Institute of Technology, Cambridge, MA, 02139 USA. e-mail: 3) Algorithm for ideal sensor placement in a defined region, [email protected]. 4) Algorithm for optimized illumination patterns. 2 B. Related Works a) Target -f b) Target -f 1) Compressive Sensing for Imaging: Compressive sensing y y Wavefront -G x x Omnidirectional has inspired many novel imaging modalities. Examples in- ultrafast pixel -m(t) clude: ultra-spectral imaging [6], subwavelength imaging [7], H wavefront sensing [8], holography [9], imaging through scat- Pulsed tering media [10], terahertz imaging [11], and ultrafast imag- Source ing [12], [13]. Intensity Sensors area -C t 2) Single Pixel Imaging: One of the most notable appli- t cations of compressive sensing in imaging is the single pixel D z z camera [3]. This was later extended to general imaging with masks [14]. We refer the interested reader to an introduction Fig. 1. Lensless imaging with compressive ultrafast sensing. a) Illumination, on imaging with compressive sampling in [15]. a time pulsed source is wavefront modulated (G) and illuminates a target with Other communities have also discussed the use of indirect reflectance f. b) Measurement, omnidirectional ultrafast sensor (or sensors) measures the time dependent response of the scene m(t). H is a physics- measurements for imaging. In the physics community the based operator that maps scene pixels to the time-resolved measurement. concept of using a single pixel (bucket) detector to perform imaging is known as ghost imaging and was initially thought of as a quantum phenomenon [16]. It was later realized that II. COMPRESSIVE ULTRAFAST SENSING computational techniques can achieve similar results [17]. Our goal is to develop a framework for compressive imaging Ghost imaging was also incorporated with compressive sens- with time-resolved sensing. Fig. 1 shows the system overview. ing [18], [19]. In the computational imaging community this L A target f 2 R with L pixels is illuminated by wavefront is known as dual photography [20]. L G 2 R . G is produced by spatially modulating a time Single pixel imaging extends to multiple sensors. For ex- pulsed source. Light reflected from the scene is measured by ample, multiple sensors were used for 3D reconstruction of a an omnidirectional ultrafast sensor with time resolution T po- scene by using stereo reconstruction [21]. Multiple sensors sitioned on the sensors plane. The time-resolved measurement were also incorporated with optical filters to create color N is denoted by m 2 R , where N is the number of time images [22]. bins in the measurement. Better time resolution (smaller T ) In this work we suggest using a time-resolved sensor instead N×L increases N. H 2 R is the measurement matrix defined of a regular bucket detector for lensless imaging. by the space to time mapping that is enforced by special 3) Time-Resolved Sensing for Imaging: Time-resolved relativity. In the case when the time resolution is very poor, sensing has been mostly used to recover scene geometry. H is just a single row (N = 1), and the process is reduced to This is known as LIDAR [23]. LIDAR was demonstrated the regular single pixel camera case. with a compressive single pixel approach [24], [25]. Time- We consider K sensors (i = 1::K) with N time samples resolved sensing has also been suggested to recover scene and M illumination patterns (j = 1::M) so the time-resolved reflectance [26], [27] for lensless imaging, but without the measurement of the i-th sensor, for a target illuminated by use of structured illumination and compressive sensing. Other the j-th illumination pattern, is defined by: mi;j = HiGjf. examples of time-resolved sensing include non-line of sight Concatenating all measurement vectors results in the total imaging, for example imaging around a corner [28] and measurement vector m~ 2 RNKM , such that the total mea- through scattering [29], [30]. Imaging around corners was also surement process is: demonstrated with low cost time-of-flight sensors, using back propagation [31] and sparsity priors [32]. 2 . 3 2 . 3 . In this paper we use compressive deconvolution with time- 6 7 6 7 m~ = 6 mi;j 7 = 6 HiGj 7 f = Qf (1) resolved sensing for lensless imaging to recover target re- 4 . 5 4 . 5 flectance. where, Q is an NKM × L matrix which defines the total C. Limitations measurement operator. The main limitations of using our suggested approach are: Here we invert the system defined in Eq. 1 using compres- • We assume a linear imaging model (linear modeling in sive sensing approach. To that end, we analyze and physically imaging is common, for example [3], [14]). modify Q to make the inversion robust.