Robust Hough Transform Based 3D Reconstruction from Circular Light Fields Alessandro Vianello, Jens Ackermann Maximilian Diebold, Bernd Jahne¨ Robert Bosch GmbH Heidelberg Collaboratory for Image Processing Robert Bosch Campus 1, Renningen, Germany Berliner Str. 43, Heidelberg, Germany [email protected] [email protected] Abstract Light-field imaging is based on images taken on a regular grid. Thus, high-quality 3D reconstruc- tions are obtainable by analyzing orientations in epipolar plane images (EPIs). Unfortunately, such data only allows to evaluate one side of the object. Moreover, a constant intensity along each orientation is mandatory for Figure 1. The proposed algorithm processes data generated from most of the approaches. This paper presents a novel method a circular camera motion, retrieving the depth from sinusoidal tra- which allows to reconstruct depth information from data jectories of 3D points in the EPIs. The resulting depth maps can acquired with a circular camera motion, termed circular be used to generate a point cloud and a mesh of the target scene. light fields. With this approach it is possible to determine the full 360° view of target objects. Additionally, circular When all images are stacked on top of each other, form- light fields allow retrieving depth from datasets acquired ing an image volume, one slice through this volume is called with telecentric lenses, which is not possible with linear epipolar plane image. EPIs give information about the mo- light fields. The proposed method finds trajectories of 3D tion parallax of the image points, which are moving with points in the EPIs by means of a modified Hough transform. specific trajectories, depending on the camera motion, see For this purpose, binary EPI-edge images are used, which Figure 1 for the circular case. In contrast to light fields, clas- not only allow to obtain reliable depth information, but sic multi-view stereo algorithms do not make use of the re- also overcome the limitation of constant intensity along dundancy contained in a densely sampled image sequence. trajectories. Experimental results on synthetic and real In fact, these algorithms often have a view selection [11] datasets demonstrate the quality of the proposed algorithm. which leads to discarding images due to the small baseline. Furthermore, optical flow algorithms normally use two or just few images to compute the depth [15, 3], hence they are 1. Introduction less robust than light fields. One of the most popular types of light fields are the so called linear light fields, which are Three dimensional geometry reconstruction is one of the a collection of images captured along a linear path. With most important tasks in computer vision and image process- this type of data, scene points trace straight lines on the ing. Depth data plays a crucial role in industrial applications EPIs, whose slopes are inversely proportional to the dis- (e.g., automatic optical inspection), the game and movie tance of the points. The main disadvantage of linear light industry, as well as common consumer products. Active fields is that they are restricted to linear camera movements. systems, such as structured light, laser scanners, or time- In this way only one side of the scene can be reconstructed. of-flight cameras are often costly and/or time consuming. To have the complete 3D shape, the target object has to be Differently, passive systems like multi-view stereo [21, 10] recorded from four different sides, and then the results have or structure from motion are more attractive, considering to be merged. This constraint makes the acquisition proce- the simple hardware required and the possibility to achieve dure long and tedious. Moreover, most of the light field al- high-quality 3D reconstructions. gorithms strongly rely on the Lambertian hypothesis, which Passive systems based on light fields have been widely means that an EPI-line should have constant intensity. developed. A light field is a densely sampled image se- Linear light field algorithms are generally developed for quence, where an object is acquired from different views. data acquired with standard perspective lenses. However, 7327 for certain applications, e.g., precise measurement tasks in Bolles [2], where salient lines were derived by finding zero optical inspection, telecentric lenses are better suited. This crossings and then by merging collinear segments. Crimin- particular type of lens allows to obtain an orthographic pro- isi et al.[5] proposed to extract EPI-regions by using photo- jection. Therefore, two identical objects will look the same consistency. More recently, Wanner [24, 25], and later on even if one is closer to the camera than the other. Thus, a Diebold [7], used the structure tensor to estimate the lo- linear light field acquired with a telecentric lens would lead cal slope of each pixel in the EPI, obtaining a coarse depth to EPIs where all the lines have the same slope, making it map which is then refined by means of a global optimiza- impossible retrieving any depth information. tion. Unfortunately, structure tensor methods provide only To overcome all these issues, we propose a new approach a local evaluation of EPIs’ orientations. This can be a prob- to extract 3D information from circular light fields. A cir- lem especially in noisy datasets, where using all the avail- cular light field acquires the scene by rotating the object in able information, i.e., the full EPI-line, helps to increase the front of the camera (or vice versa). In this way it is possi- quality of the final reconstruction. Additionally, global opti- ble to reconstruct the full 360° shape with just one continu- mization tends to smooth depth discontinuities by averaging ous acquisition. With this setup, every captured scene point between foreground and background disparities. corresponds to a curved trajectory in the EPI. Variations of An approach which takes advantage of the whole trajec- the depth lead to sine shaped curves with different ampli- tory was proposed by Feldmann et al.[8, 9], who used the tudes and phase offsets, as will be explained in Section 3. intensity constancy as a measure to determine valid paths. It will be shown that circular light fields can be used to re- In their work the 3D space is discretized into voxels and trieve depth information even from datasets acquired with then, for each hypothetical 3D point, the algorithm seeks a telecentric lens. The proposed algorithm uses a coarse in the image volume if the corresponding path exists. This EPI-slope map, generated with the local structure tensor, method was also adapted to the case of a camera which ro- together with a binary edge map of the EPI, to extract trajec- tates around the target scene. Crispell et al.[6], and later on tories by using an adapted version of the Hough transform. Lanman et al.[18], retrieve EPI-trajectories in circular light The result, is a set of highly accurate depth maps of the tar- fields only on depth discontinuities instead of texture edges. get scene from all sides. Since the Hough transform uses They use a contour tracking approach which is not robust to binarized EPIs to retrieve trajectories, it is possible to get specularities and cannot deal with EPIs having many close rid of the Lambertian hypothesis and process datasets with trajectories. Similarly to [8], Kim et al.[17] proposed a strong intensity changes along the EPI-curves. In fact, even method for linear light fields which computes depth esti- if a trajectory is only partially visible or its intensity satu- mates around the scene’s highly textured areas by testing rates because of a specular reflection, the Hough transform all the possible disparity hypotheses and choosing the one can still recover the full curve. In order to apply circular that leads to the best color constancy along the EPI-line. light fields to both perspective and telecentric lenses, two This approach was later applied to circular camera motion slightly different versions of the algorithm are proposed. by Yucer¨ et al.[27], to segment the target object and com- Our method is based on [22, 23], were we introduced pute the visual hull. Then, they extended their approach to a new approach for linear light fields which retrieves EPI- estimate the depth also in concave areas by analyzing the lines with a combined structure tensor and Hough trans- local gradient directions in the image volume [26]. Unfor- form. tunately, the main limitation of these methods is the restric- tion to Lambertian surfaces. 2. Related Work Although the term light field was already introduced in 3. Circular Light Fields 1936 by Gershun [12] as the multi-dimensional function EPI analysis was extended to the case of circular camera (later called plenoptic function [1]) describing all the in- movements by Feldmann et al.[8]. The acquisition setup is formation available from the scene’s reflected light, light composed of a fixed camera and an object rotating around fields were introduced into computer graphics only in 1996 a point M aligned with the camera’s optical center C, as by both Gortler et al.[14] and Levoy et al.[19] for an im- shown in the left side of Figure 2. In this section, the im- age base rendering application. In order to acquire light age formation of circular light fields is explained, for both fields, the plenoptic function is simplified to a 4D subspace, orthographic and perspective camera projection models. termed the Lumigraph [14]. With this representation, the ray space of the light field can be parametrized by the two 3.1. Orthographic Camera points of intersection of a ray with two parallel planes, so that the light field can be considered as a collection of views, The simplest camera projection is the orthographic pro- where the focal points of the cameras lie in a 2D plane.