Multiview Mm-Wave Imaging with Augmented Depth Camera Information
Total Page:16
File Type:pdf, Size:1020Kb
Received November 13, 2017, accepted March 4, 2018, date of publication March 15, 2018, date of current version April 18, 2018. Digital Object Identifier 10.1109/ACCESS.2018.2816466 Multiview mm-Wave Imaging With Augmented Depth Camera Information JAIME LAVIADA 1, MIGUEL LÓPEZ-PORTUGUÉS1, ANA ARBOLEYA-ARBOLEYA2, AND FERNANDO LAS-HERAS1, (Senior Member, IEEE) 1Department Ingeniería Eléctrica, Universidad de Oviedo, 33203 Gijón, Spain 2Universidad Rey Juan Carlos, 28942 Madrid, Spain Corresponding author: Jaime Laviada ([email protected]) This work was supported in part by Ayudas Fundación BBVA a Investigadores y Creadores Culturales 2016, in part by the Ministerio de Ciencia e Innovación of Spain /FEDER under Project TEC2014-55290-JIN and Project TEC2014-54005-P, and in part by the Gobierno del Principado de Asturias (PCTI)/FEDER-FSE under Project GRUPIN14-114. ABSTRACT This paper introduces a hybrid method that combines two 3-D imaging methods of very different nature but complementary. On the one hand, a depth camera, also known as RGB-D camera, is used in order to obtain a 3-D scan of the target under test. On the other hand, a synthetic aperture radar at millimeter-wave frequency band is used to exploit the penetration capabilities of these waves, allowing electromagnetic imaging. As a result, the system can provide two complementary and aligned 3-D models. In contrast to the previous work, in which the position estimation and the external 3-D model relied on conventional optical cameras, this setup can construct the model even in the case of flat textures as it does not resort to color information at any stage. Several examples, in which the millimeter-wave scanner is validated by raster scan, are presented for people screening revealing the validity of the approach. INDEX TERMS Millimeter-wave imaging, portable camera, multiview imaging, depth camera, structured light, synthetic aperture radar. I. INTRODUCTION diverse expenses such as the use of ionizing radiation Object and people scanning is a well-known tool that enables (X-rays) or the lost of resolution with respect to conventional to build a computational three-dimensional (3D) model of a optical cameras (micro- and mm-waves). given target. On the one hand, the most widespread approach In the last years, the use of mm-wave technology has is based on scanning an object so that a model of its visible widespread for different reasons. First, this frequency band layer, which is the most external layer, is calculated. This provides very interesting balance between penetration capa- goal can be achieved by means of multiple technologies such bilities and resolution. Consequently, it is appealing for appli- as lasers [1] or a collection of images taken from optical cations such as people screening [7], [9], [10] or automotive cameras [2], [3]. Also, depth cameras, which will be later radars [11]. Second, this frequency band can also provide introduced, can be used for this purpose. very high absolute bandwidth when compared to standard Depending on the technology and investment, different microwave communications, which has motivated its use for degrees of accuracy and scanning speed can be achieved. It is high data-rate wireless communications [12]–[14]. also relevant to observe that these approaches usually entail In addition, micro- and mm-wave camera and portable sys- a relatively low computational burden. Furthermore, it has tems are progressively being developed [6], [15], [16]. This been demonstrated that currently smartphones are capable fact has motivated the use of mm-wave cameras as portable to perform real-time 3D external reconstruction from the 3D electromagnetic scanners [17] in a similar fashion to information captured by their built-in cameras [4]. the aforementioned smartphones scanners with capability to On the other hand, the use of other technologies such build 3D models on-the-fly [4]. In this framework, [17] pro- as X-ray computed tomography, microwaves [5], [6] and posed the use of a Synthetic Aperture Radar (SAR) to scan millimeter-waves (mm-waves) [7], [8] can provide infor- a certain object or body at mm-wave frequencies by com- mation of not only the visible (i.e., external) surface but bining multiview information. The data were collected from also about the inner layers. However, this advantage entails multiple angles in order to achieve a complete model scan. 2169-3536 2018 IEEE. Translations and content mining are permitted for academic research only. VOLUME 6, 2018 Personal use is also permitted, but republication/redistribution requires IEEE permission. 16869 See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. J. Laviada et al.: Multiview mm-Wave Imaging With Augmented Depth Camera Information The position and attitude of the scanner at each view acqui- sition were estimated by using an optical camera. Thus, by taking pictures at the same positions as the mm-wave acqui- sitions are performed and processing them with Structure from Motion techniques [18], the scanner rotation angles and positions can be found to apply the appropriate multiview merge algorithms. The possibility of reducing the number of transmitting and receiving elements resorting to multistatic Fig. 1. Diagram of the steps of the proposed scanning involving techniques was later studied in [19]. electromagnetic and depth camera information. Nevertheless, as pointed out in [17], this positioning scheme requires to find a number of common key points from different views so that the position can be calculated by Fig.1 depicts a diagram with the different steps of the triangulation. In particular, the scale-invariant feature trans- scanning process. As shown, a mm-wave acquisition by form (SIFT) was used to compute the key points in [17]. means of multiple transmitters and receivers is performed in Consequently, the technique is expected to fail when consid- parallel to the acquisition of a depth camera. On one hand, ered objects lack of textures. the acquired scattered field is processed by means of standard In order to bypass the previous limitation, the use of depth range-migration techniques [25] to obtain the reflectivity cameras, also known as RGB-D cameras [20], [21], is pro- evaluated on a regular grid of points. On the other hand, posed in this paper. These cameras are based on two different the PC measured by the depth camera is processed by means approaches. The first one is usually known as structured of the iterative closest point (ICP) algorithm [26] to find light and it relies on projecting a known set of (infrared) out the movement with respect to the previously acquired patterns into the object so that its shape can be estimated from PC, i.e., to know the relative movement between two depth the deformed pattern [22]. The second technique is usually camera acquisitions. Consequently, the global camera attitude known as time-of-flight and it is based on measuring how and position can be estimated by taking into account the long it takes for light to travel the round trip from the sensor cumulative movements and rotations. to a given scene point [23]. It is worthy to note that these Finally, the reflectivity and PC data are merged with the tools have also been hybridized with mm-wave imaging for data from previous acquisitions yielding a double 3D model people screening with booth-size flat-panels to help reducing that contains two sets of information. One related to the exter- the observation volume [24]. nal layer of the object (global point cloud) and the second one The main contribution of this paper is the extension of related to inner layers (global reflectivity). the mm-wave portable scanner concept presented in [17] by replacing the conventional optical camera with a depth B. mm-WAVE IMAGING camera. Thus, positioning is not accomplished by finding key In this work, a regular grid of monostatic transceivers points but by comparing point clouds (PCs) acquired by the is considered as in [7] and [17]. Consequently, standard depth camera. This change does not affect the ability of the range-migration imaging algorithms are used. In particular, system to produce a double model including the external and the approach described in [25], which relies on fast Fourier inner layers. The accuracy achieved by this combination is transforms, is employed. Thus, if a system of coordinates, assessed for several cases in the people screening context. local to the scanner, is defined as x0; y0; z0, where the scan- This paper is structured as follows. First, an overview of the ner aperture is assumed to be along the plane y0 D 0 , then approach combining mm-wave and depth camera information the reflectivity ρ x0; y0; z0 can be computed as: is presented. Next, the details of both imaging systems are detailed. After that, the merging approach to create the double ρ x0; y0; z0 D F −1 F s x0; z0;! model strategy is presented. In the Results section, the accu- k xz racy and time to estimate the position with the depth camera q 2 2 2 are evaluated and, next, several examples considering one × exp −j 4k − kx − kz D (1) dimensional as well as two-dimensional synthetic apertures are shown. Finally, the conclusions are drawn. where s(x0; z0;!) is the acquired complex amplitude for the transceiver placed at the aperture position x0; z0 when it is transmitting a tone at the angular frequency !; D is the II. mm-WAVE SCANNER ENHANCED BY distance along the y axis from the center of the scanner DEPTH CAMERA INFORMATION aperture to the center of the observation grid; Fxz f·g denotes A. GLOBAL OVERVIEW a bidimensional Fourier transform from the (x0; z0) space to The scanner proposed in [17] is based on a relatively small the .kx ; kz/ space; k D !=c is the wavenumber, being c the aperture (e.g., a maximum size of 15 cm was considered speed of light; Fk f·g denotes a 3D Fourier transform from in [17]) so that it can be moved around the object under test the k-domain defined by the coordinates (ky; ky; kz), which 2 2 2 2 to perform a multiview scan.