Efficient Implementation of Gaussian and Laplacian Kernels for Feature

Journal of Imaging

Article Efﬁcient Implementation of Gaussian and Laplacian Kernels for Feature Extraction from IP Fisheye Cameras

Konstantinos K. Delibasis

Department of Computer Science and Biomedical Informatics, University of Thessaly, Lamia 35131, Greece; [email protected]; Tel.: +30-22310-66900

Received: 1 May 2018; Accepted: 17 May 2018; Published: 24 May 2018

Abstract: The Gaussian kernel, its partial derivatives and the Laplacian kernel, applied at different image scales, play a very important role in image processing and in feature extraction from images. Although they have been extensively studied in the case of images acquired by projective cameras, this is not the case for cameras with fisheye lenses. This type of cameras is becoming very popular, since it exhibits a Field of View of 180 degrees. The model of fisheye image formation differs substantially from the simple projective transformation, causing straight lines to be imaged as curves. Thus the traditional kernels used for processing images acquired by projective cameras, are not optimal for fisheye images. This work uses the calibration of the acquiring fisheye camera to define a geodesic metric for distance between pixels in fisheye images and subsequently redefines the Gaussian kernel, its partial derivatives, as well as the Laplacian kernel. Finally, algorithms for applying in the spatial domain these kernels, as well as the Harris corner detector, are proposed, using efficient computational implementations. Comparative results are shown, in terms of correctness of image processing, efficiency of application for multi scale processing, as well as salient point extraction. Thus we conclude that the proposed algorithms allow the efficient application of standard processing and analysis techniques of fisheye images, in the spatial domain, once the calibration of the specific camera is available.

Keywords: Gaussian; Laplacian kernels; multi-resolution image processing; Harris corner detection; spatial domain; ﬁsheye image; camera calibration

1. Introduction A number of types of very wide field cameras acquire images with 180◦ field of view (FoV). These cameras use dioptric systems, spherical lens, often called fisheye, or a combination of catadioptric (parabolic or spherical mirror) and dioptric (normal projection lens), often referred to as omnidirectional. The model of formation of all these types of images differs significantly from the traditional central projection model and introduces rapid deterioration of spatial resolution towards the periphery of the FoV, as well as strong deformations that map straight lines as curves. The model of image formation with 180◦ FoV can be treated in a unified mathematic way, utilizing a spherical optic lens and central projection [1,2]. Thus we will use the terms interchangeably for the rest of the paper. The use of very wide field cameras is becoming very wide in domains like security [3], robotics [4,5], public and private area monitoring, since they allow constant imaging of all directions with a single camera. Spherical fisheye panoramas are used to emulate or to register a PTZ camera and to integrating live webcam feeds within 2D and 3D GIS applications (such as Google Earth and Google Maps) [6,7]. Fisheye cameras with special processing operators are proposed for large space monitoring and crowd

J. Imaging 2018, 4, 73; doi:10.3390/jimaging4060073 www.mdpi.com/journal/jimaging J. Imaging 2018, 4, 73 2 of 21 counting in [8]. In [9], a network of fisheye cameras is used to efficiently track persons in a space with obstacles that hinder visibility. Traffic management at road intersection has been proposed in [10] using network fisheye cameras. Pair of fisheye cameras has been used for pose estimation and motion capture in [11], employing a number of 3D Gaussians to generate a probabilistic body model. Gaussian kernels defined in 3D have also been used for scene modeling, in order to make discrete phenomena, like occlusion, continuous and thus, more easily optimizable [12]. The very different model of image formation for fisheye images has invoked a number of attempts to redefine image processing operators for this type of images. The basic morphology operators have been redefined in [8], although the computational cost has not been discussed. The well-known Scale-Invariant Feature Transform (SIFT) image descriptors that were introduced in [13], are redefined for omnidirectional images in [14] in the frequency domain. Fisheye images are defined over spherical coordinates (azimuth θ, elevation φ). Therefore, Spherical Fourier Transform m (SFT) is used, that decomposes the image as a weighted sum of spherical harmonic functions Yl (θ, ϕ) of degree l and order m, with |m| ≤ l. In [15], the Gaussian kernel has been defined in the (θ, φ) image domain using spherical harmonics of the 0th order. The work of [16] also employs the use of SFT to detect points of interest using the well-known SIFT. In [17] differential operators defined on Riemannian manifolds are used to compute scale space and perform smoothing in fisheye images. That work was furthr improved in [18], where the already developed differential operator was extended to perform SHIFT-like salient point detection in omnidirectional images. Others have reported exploiting the image space, rather than the frequency domain, for specific kernels. In [19], the authors also used a simplified version of SIFT feature extraction method (e.g., no multiresolution was used) for robot navigation by fisheye camera, obtaining good results. In [20], features are extracted from 360 FoV omnidirectional images in the spatial domain, but after the image has been mapped to a hexagonal grid. In [21], 4-neighbours and 8-neighbours Laplacian operators have been proposed for omnidirectional panoramic images. In [22], an approach for fisheye image processing in the spatial domain is proposed, using geodesic distance between pixels. The approach proposed in this work, is also based on the spatial image domain and utilizes the geodesic distance between pixels. The main difference from [22] is that the proposed method uses the geodesic distance metric between pixels to redefine the basic filter kernels, whereas in [22] the image neighborhood is being resampled, while keeping the filter kernels unchanged. In order to avoid computationally expensive image interpolation, [22] uses the nearest neighbor interpolation, which although not a good performing technique, still produced better results than the classic image processing operators. As it will be discussed later, there is no proposed efficient implementation for the method in [22], since it requires the geodesic neighborhood definition for every pixel in each new image, in contrast to the method proposed in this work that requires the construction of a small Gaussian kernel, only once for a calibrated camera. The rest of the paper is structured as following: In the methodology section, the geodesic distance metric is defined using the calibration of the fisheye camera. The Gaussian kernel, its partial derivatives and the Laplacian of Gaussian kernel is redefined, using the geodesic distance. All these kernels vary with their spatial location in the image, therefor, their direct convolution with the image is inefficient. Thus, an efficient computational implementation is proposed, which involves the initial pre-calculation of a single Gaussian kernel with small standard deviation σ0 and the approximation of the Image response of the Gaussian, its derivatives, as well as the Laplacian of Gaussian at larger values of σ.

2. Materials and Methods The methodology that is proposed in this work is tested with images acquired by a Mobotix Q24 ﬁsheye camera, that is installed on the roof, imaging an indoor space. A typical frame of the camera is shown in Figure1, with pixilation of 480 × 640. The camera has been calibrated using [23]. The result of the calibration is shown in Figure2, where the azimuthal angle Θ and the elevation Φ is provided for each pixel of the ﬁsheye image. J. Imaging 2018, 4, 73 3 of 21 J. ImagingJ. Imaging 2018 2018, 4,, 4x, FORx FOR PEER PEER REVIEW REVIEW 3 of3 21 of 21

Figure 1. A typical image from a roof-based fisheye camera. Figure 1. A typical image from a roof-based ﬁsheyefisheye camera.camera.

(a) (b)

Figure 2. The resulting(a calibration) of the fisheye camera, for each image pixel( b(a)) the azimuthal angle θ and (b) the elevation angle φ is provided. Figure 2.2. The resulting calibrationcalibrationof ofthe the ﬁsheye fisheye camera, camera, for for each each image image pixel pixel (a ()a the) the azimuthal azimuthal angle angleθ 2.1.andθ Geodesic and (b )(b the) theDistance elevation elevation Metric angle angle betweenφ isφ provided.is provided. Pixels of the Calibrated Fish-Eye Image

2.1. GeodesicThe formation Distance ofMetric omni-directional between Pixels image of the using Calibrated a spherical Fish-Eye lens, Image as presented in detail in [23] can 2.1.be Geodesic summarized Distance as following: Metric between the intersection Pixels of the of Calibrated the line Fish-Eyeconnecting Image the real world point with the The formation of omni-directional image using a spherical lens, as presented in detail in [23] can centerThe of formation the optical of element omni-directional is calculated image with using the optical a spherical element. lens, This as intersection presented in is detailthen projected in [23] can be summarized as following: the intersection of the line connecting the real world point with the becentrally summarized on the as image following: sensor the plane. intersection It has been of the sh lineown connecting [1] that by thechoosing real world the center point of with projection, the center center of the optical element is calculated with the optical element. This intersection is then projected of theone opticalcan simulate element the is calculated use of any with quadratic the optical shape element. mirror This (spherical, intersection ellipsoid, is then projectedparaboloid centrally and centrally on the image sensor plane. It has been shown [1] that by choosing the center of projection, onhyperboloid). the image sensor This type plane. of Itimage has beenformation shown induce [1] thats non-linear by choosing transformation the center of of distances projection, between one can one can simulate the use of any quadratic shape mirror (spherical, ellipsoid, paraboloid and simulatepixels. the use of any quadratic shape mirror (spherical, ellipsoid, paraboloid and hyperboloid). hyperboloid).These concepts This type are of visualized image formation in Figure 3,induce wheres thnon-lineare semi-spherical transformation optical element of distances of unit radiusbetween This type of image formation induces non-linear transformation of distances between pixels. pixels.and the image plane is displayed. The center of projection is placed at −f on the Y axis, with f set to These concepts are visualized in Figure3, where the semi-spherical optical element of unit radius 0.2.These The image concepts plane are is visualizedtessellated intoin Figure 128 equidistant 3, where th pointse semi-spherical to resemble opticalthe image element pixels. of 21 unit of these radius and the image plane is displayed. The center of projection is placed at −f on the Y axis, with f set and“pixels” the image are backprojected plane is displayed. on the sphericalThe center optical of projection element (bothis placed shown at in−f ondifferent the Y color).axis, with It is self-f set to to 0.2. The image plane is tessellated into 128 equidistant points to resemble the image pixels. 21 of 0.2.evident The image that the plane back-projected is tessellated poin intots 128are equidistantno longer equidistant. points to resemble the image pixels. 21 of these these “pixels” are backprojected on the spherical optical element (both shown in different color). It is “pixels” are backprojected on the spherical optical element (both shown in different color). It is self- self-evident that the back-projected points are no longer equidistant. evident that the back-projected points are no longer equidistant.

J. Imaging 2018, 4, 73 4 of 21 J. Imaging 2018, 4, x FOR PEER REVIEW 4 of 21

Figure 3. The basic concept of ﬁsheye image formation, involving a (semi) spherical lens and central Figure 3. The basic concept of fisheye image formation, involving a (semi) spherical lens and central projection on the image plane. The difference between Euclidean distance between pixels on the image projection on the image plane. The difference between Euclidean distance between pixels on the plane and geodesic distance (on the spherical lens) becomes apparent. image plane and geodesic distance (on the spherical lens) becomes apparent.

If a window of size (2 w + 1, 1) is defined defined on the image plane (red dots in Figure 33,, thenthen thethe definitiondefinition ofof aa Gaussian Gaussian within within this this window, window, using using the Euclideanthe Euclidean distance distance between between pixels pixels on the on image the imageplane isplane visualized is visualized as theblack as the curve black in curve Figure in3 .Figure If this Gaussian3. If this Gaussian is back-projected is back-projected on the spherical on the sphericaloptical element, opticalthe element, kernel the depicted kernel in depicted black (at in the black periphery) (at the periphery) is produced. is produced. As expected, As it expected, is different it isfrom different a Gaussian from a kernel, Gaussian due kernel, to distance due metric.to distance In order metric. to In generate order to a Gaussiangenerate a kernel Gaussian defined kernel on definedthe image on sensor, the image we havesensor, to modifywe have the to distancemodify th metrice distance between metric pixels between on the pixels sensor, on according the sensor, to accordingthe geodesic to the distance geodesic of their distance back-projection of their back-pro on thejection circle. on Figure the circle.3 is a Figure 2D abstraction 3 is a 2D abstraction of the real of3D the camera real 3D setup. camera The setup. asymmetry The asymmetry that is evident that inis evident the 21-point, in the black 21-point, kernel, black defined kernel, on defined the image on theplane image is exaggerated plane is exaggerated since the image since plane the(image image lineplane in this(image 2D abstraction)line in this was 2D tessellatedabstraction) to onlywas 128tessellated pixels. Furthermore,to only 128 pixels. in the caseFurthermore, of a 2D kernel, in th asymmetrye case of a is2D present kernel, only asymmetry along one is of present its two only axes, alongas it will one be of further its two discussedaxes, as it below.will be further discussed below. In [24] [24] the use of the geodesicgeodesic distance has beenbeen proposed for re-definingre-defining the Zernike moment invariant for calibrated calibrated fish-eye fish-eye images. The The proposed proposed approach approach requires the results of the the calibration calibration of the fish-eye fish-eye camera. Several approaches exist for fi fish-eyesh-eye camera calibration, such as [[1,25].1,25]. In this work, we utilize the calibration of the specificspecific fish-eyefish-eye camera, proposed in [[23].23]. The results of the calibration nay be summarized in two look-up tables of size equal to an image frame, which hold the direction ofof eacheach image image pixel pixel (x ,(yx), iny) realin real 3D world.3D world. The directionThe direction is defined is defined by two by angles, two theangles, azimuth the azimuthΘ(x, y) and Θ(x the, y) elevation and the Φelevation(x, y). During Φ(x, y the). During calibration the step,calibration these twostep, angles these are two pre-calculated angles are pre- for calculatedall pixels of for the all video pixels frame of the and video are fr storedame and in two are look-upstored in tables two look-upΘ and Φ tables, as shown Θ and in Φ Figure, as shown2. in FigureIt is2. clear that the distance of two pixels is different when measured on the image sensor and on the projectionIt is clear that of the the image distance points of two on pixels the spherical is different optical when element. measured Therefore, on the image in order sensor to produce and on theaccurate projection results, of imagethe image processing points algorithmson the spherica that usel optical pixel element. distances Therefore, have to be in re-implemented order to produce for accuratefish-eye (omni-directional)results, image processing images. algorithms that use pixel distances have to be re-implemented for fish-eyeIt is (omni-directional) well known that the images. geodesic curve of a sphere is a great circle (a circle that lies on the sphere and hasIt is thewell same known center that withthe geodesic the sphere). curve Thus, of a sphere the distance is a great between circle (a any circle two that points lies onon athe sphere sphere is andthe lengthhas the of same the arc center that with is defined the sphere). by the twoThus, points the distance and belongs between to a any circle two that points passes on through a sphere the is the length of the arc that is defined by the two points and belongs to a circle that passes through the two points and has the same centre with the sphere. The geodesic distance between pixels = () = () p000x , y and p111x , y is defined in [22,24] and can be summarized as following:

J. Imaging 2018, 4, 73 5 of 21

two points and has the same centre with the sphere. The geodesic distance between pixels p0 = (x0, y0) J. Imaging 2018, 4, x FOR PEER REVIEW 5 of 21 and p1 = (x1, y1) is deﬁned in [22,24] and can be summarized as following: Let P0 and P1 be the projections of the two pixels on the spherical optical element and v0, v1 be Let P0 and P1 be the projections of the two pixels on the spherical optical element and vv, be the position vectors pointing to P0 and P1. The direction of the two pixels in real world is given01 by the thepre-calculated position vectors look-up pointing tables: to P0 and P1. The direction of the two pixels in real world is given by the pre-calculated look-up tables: (θ0, ϕ0) = (Θ(x0, y0), Φ(x0, y0)) ()()()θϕ,,,,=Θ()xy Φ xy (1) (θ 00, ϕ ) = (Θ(x 00, y ), Φ(x 00, y )) 1 1 1 1 1 1 (1) ()()()θϕ =Θ Φ 11,,,,()xy 11 xy 11 Thus, the position vectors are calculated: Thus, the position vectors are calculated: v = (cos θ cos ϕ , sin θ cos ϕ , sin ϕ ), i = 0, 1 (2) i ==()iθϕθϕϕi i i i viiiiiicos cos ,sin cos ,sin ,i 0,1 (2) The distance r01 of points P0 and P1 on the unit sphere is easily calculated as the arc-length The distance r01 of points P0 and− 1P1 on the unit sphere is easily calculated as the arc-length between P0 and P1, assuming that cos returns the result in radians: between P0 and P1, assuming that cos-1 returns the result in radians:

−−1 r01 ===⋅d(()v0, v1) = cos1 ()(v0 · v1) (3) rd01vv 0,cos 1 vv 0 1 (3)

FigureFigure 4 shows the geodesic distance component betweenbetween successive pixels along lines (left) and columnscolumns (right), (right), defined deﬁned more more formally formally as: as: ()=+() ( )() DDxydxxx(x,,1,,,y) = d((x + 1, yy), ( xyx, y)) (4)(4) Dy(()x, y) ==+d(( (x, y + 1 )()), (x, y)) Dxydxyy ,,1,,() xy

FigureFigure 4. 4. GraphicGraphic representation representation of Dxx ((leftleft)) and and DDyy (right(right).).

2.2.2.2. Filter Filter Kernels Kernels for for Fish-Eye Fish-Eye Image Image Processing BasedBased on the Geodesic DistanceDistance MetricMetric betweenbetween ImageImage Pixels Pixels The most versatile filter in image processing is the Gaussian. According to its definition for 2D images,The themost Gaussian versatile can filter be writtenin image as processing is the Gaussian. According to its definition for 2D images, the Gaussian can be written as r2 1 − 22 g(x, y; σ) = e 2−σr (5) 2πσ12 σ 2 gxy(),;σ = e2 (5) 2πσ 2 with r2 = x2 + y2 being the distance of pixel (x, y) from the centre of the kernel. The discrete Gaussian 222 withis defined rxy=+ (at least) inbeing the followingthe distance of pixel (x, y) from the centre of the kernel. The discrete Gaussian is defined (at least) in the following (x, y) ∈ [−3dσe, 3dσe] × [−3dσe, 3dσe] (6) ()∈−σσ ×− σσ xy,3,33,3    (6) where x is the operator returning the closest integer larger than x, for x > 0. The standard deviation σ is defined by the user in pixels, which equivalently defines the size of the kernel, according to Equation (6).

J. Imaging 2018, 4, 73 6 of 21 where dxe is the operator returning the closest integer larger than x, for x > 0. The standard deviation σ is defined by the user in pixels, which equivalently defines the size of the kernel, according to Equation (6). The above “classical” definition of the Gaussian needs to be modified for calibrated fish-eye images. Let us suppose that we require the linear convolution of the Gaussian kernel with a fish-eye image, at pixel p0 = (x0, y0). Let Aσ denote the subimage centered at (x0, y0):

Aσ = [x0 − 3dσe, x0 + 3dσe] × [y0 − 3dσe, y0 + 3dσe] (7)

In the case of a calibrated fish-eye image, the definition of Equation (4) is modified as following:

• It is straightforward to use the geodesic distance metric between p0 and any pixel pi = (xi, yi) in Aσ, thus ri = d(pi, p0) • σ is initially deﬁned by the user in pixels. However, since the geodesic distance between pixels is used in Equation (1), the value of σ (in pixels) cannot be used. This becomes clear if one considers that the Gaussian has to approach 0 within Aσ. Since Aσ was deﬁned as a window centered on the current pixel, spanning ±3 σ, we set σ equal to (1/3) of the maximum geodesic distance between the central pixel p0 and any pixel in Aσ:

1 σ = max({r = d(p , p ), p ∈ Aσ}). (8) 3 i i 0 i

• Then σ is replaced in Equation (5).

Thus, for a Gaussian centered at p0 = (x0, y0), the

( )2 1 − d p,p0 g(x, y; σ) = g(p; σ) = e 2σ2 (9) 2πσ2

It becomes obvious that the Gaussian kernel now depends on its location on the fish-eye image. In order to define the Gaussian kernel with a specific size for the whole fisheye image, the following Algorithm 1 is used:

Algorithm 1. Initialization of the Gaussian kernel with small standard deviation. The size of the window kernel (2 w + 1) × (2 w + 1) is deﬁned. The value of the standard deviation σ is calculated as following:

The window A0 is placed near the edge of the FoV, where pixel distances are smaller than other locations with high elevation φ and σ0 is calculated according to Equation (8).

This value of σ0 is used for each location in the ﬁsheye image.

Figure5a shows the Gaussian kernel as deﬁned in Equation (9) at different position on the ﬁsheye image. Radial lines emanating from the CoV (Center of View) have constant azimuthal θ and points on the concentric circles centered on the CoV have constant elevation φ. The same kernels are shown in (b), using color-scale. The CoV is marked on Figure5b. J. Imaging 2018, 4, 73 7 of 21 J. Imaging 2018, 4, x FOR PEER REVIEW 7 of 21

(a)

(b)

Figure 5. The Gaussian defined deﬁned in Equation (9) at diffe differentrent positions on the ﬁsheyefisheye image, using the geodetic distance is shown in (a) 3D and (b) as color-encoded image.image.

It is evident that the shape of the Gaussian kernel is a function of the kernel’s location on the It is evident that the shape of the Gaussian kernel is a function of the kernel’s location on the image. More specifically, the kernel at pixel (x, y) appears rotated by an angle equal to the azimuth image. More specifically, the kernel at pixel (x, y) appears rotated by an angle equal to the azimuth Θ(x, y) at this pixel. Furthermore, the Gaussian kernel exhibits asymmetry, as indicated in the 1D Θ(x, y) at this pixel. Furthermore, the Gaussian kernel exhibits asymmetry, as indicated in the 1D abstraction of Figure 3. It is easy to confirm that consecutive pixels along the direction with constant abstraction of Figure3. It is easy to confirm that consecutive pixels along the direction with constant (iso-) elevation φ have constant geodesic distance. On the other hand, consecutive pixels along the (iso-) elevation φ have constant geodesic distance. On the other hand, consecutive pixels along the direction with constant (iso-) azimuth θ do not have constant geodesic distance. Thus, the geodetic direction with constant (iso-) azimuth θ do not have constant geodesic distance. Thus, the geodetic Gaussian kernel is expected to exhibit asymmetry along the axis with constant azimuth. An example Gaussian kernel is expected to exhibit asymmetry along the axis with constant azimuth. An example depicting this asymmetry is shown in Figure 6. The location that was selected is shown in (c) as a depicting this asymmetry is shown in Figure6. The location that was selected is shown in (c) as a green circle. The iso-azimuth and iso-elevation contours are superimposed in purple and green color green circle. The iso-azimuth and iso-elevation contours are superimposed in purple and green color respectively. Figure 6a shows a 35 × 35 geodetic Gaussian kernel, generated by Equation (9) of the respectively. Figure6a shows a 35 × 35 geodetic Gaussian kernel, generated by Equation (9) of the proposed method. The orientation of the kernel with respect to the directions of constant azimuth proposed method. The orientation of the kernel with respect to the directions of constant azimuth and and elevation, is as described above. Figure 6b displays the asymmetry of the kernel. More elevation, is as described above. Figure6b displays the asymmetry of the kernel. More specifically, specifically, the profile of the kernel is plotted along the directions of constant azimuth θ and of the profile of the kernel is plotted along the directions of constant azimuth θ and of constant elevation constant elevation φ. Both profiles are also plotted after being mirrored, in order to make the φ. Both profiles are also plotted after being mirrored, in order to make the asymmetry apparent. It can asymmetry apparent. It can be observed that the kernel is symmetric along the direction of constant φ and slightly asymmetric along the direction of constant θ (the profile and its mirror do not match). Finally, the resampling of the image to construct the geodetic neighborhood round the selected point,

J. Imaging 2018, 4, 73 8 of 21 be observed that the kernel is symmetric along the direction of constant φ and slightly asymmetric along the direction of constant θ (the proﬁle and its mirror do not match). Finally, the resampling of theJ. Imaging image 2018 to, construct4, x FOR PEER the REVIEW geodetic neighborhood round the selected point, as described8 of 21 in [22], Equation (16), is displayed in Figure6d. These neighborhood pixels should be equidistant using as described in [22], Equation (16), is displayed in Figure 6d. These neighborhood pixels should be geodesic distant metric and they were selected using the nearest neighbor interpolation, as explained equidistant using geodesic distant metric and they were selected using the nearest neighbor in Sectioninterpolation, 3.3 of [22 as]. explained in Section 3.3 of [22]. X

(a) (b)

Figure 6. (a): The shape of the geodetic Gaussian kernel, generated at the location of the frame Figure 6. (a): The shape of the geodetic Gaussian kernel, generated at the location of the frame indicated indicated by a green circle (c). Contours of iso-azimuth and iso-elevation are also superimposed. The by a green circle (c). Contours of iso-azimuth and iso-elevation are also superimposed. The shape shape and symmetry of the kernel is demonstrated in (b), along the direction of constant (iso-) and symmetryazimuth and of theconstant kernel (iso-) is demonstrated elevation. Finally in (b),, alongimage theresampling direction to of constantconstruct (iso-)the geodetic azimuth and constantneighborhood, (iso-) elevation. according Finally, to [22] image is shown resampling in (d). to construct the geodetic neighborhood, according to [22] is shown in (d). The partial derivatives of the Gaussian kernel are easily calculated analytically: The partial derivatives of the Gaussian∂ kernel are easilyx calculated analytically: gxy(),;σσ==− g gxy() ,; (10) ∂x x σ 2 ∂ x g(x, y; σ) = gx = − g(x, y; σ) (10) The x coordinate is calculated∂x as the geodesic σdistance2 of (x, y) from (x0, y), namely = ()() x dxy()ii,,, xy0 i. The same holds for the partial derivative with respect to y. The x coordinate is calculated as the geodesic distance of (x, y) from (x0, y), namely ()2 x = d((xi, yi), (x0, yi)). The same holds for the partial()()() derivative with−d pp respect, 0 to y. ∂ dxyxy,, , 1 2 ()σ ==− 0 2σ gxy,; gx 22 e ∂x σπσ2 2 d(p,p0) d((x,y),(x ,y)) − 2 (11) ∂ 0 1 d2()σpp,2 g(x, y; σ) = gx = −()()()2 2 e − 0 ∂∂x dxyxy,,,σ 0 2πσ1 2 ()σ ==− 2σ 2 (11) gxy,; gy 22 ed(p,p0) ∂∂ d((x,σπσy),(x,y0)) 1 − y g(x y σ) = g = − 2 e 2σ2 ∂y , ; y σ2 2πσ2

J. Imaging 2018, 4, 73 9 of 21

J. Imaging 2018, 4, x FOR PEER REVIEW 9 of 21 Figure7 shows the two partial derivatives Gaussian kernel, as deﬁned in Equation (11) at different J. ImagingFigure 2018 ,7 4 , showsx FOR PEER the REVIEWtwo partial derivatives Gaussian kernel, as defined in Equation (11)9 of at21 position on the ﬁsheye image. Radial lines emanating from the CoV have constant azimuthal θ. Points different position on the fisheye image. Radial lines emanating from the CoV have constant azimuthal φ on theθ. concentricPointsFigure on the7 circlesshows concentric centeredthe two circles partial on cent the eredderivatives CoV on have the CoV constantGaussi havean constant elevationkernel, aselevation defined. φ .in Equation (11) at different position on the fisheye image. Radial lines emanating from the CoV have constant azimuthal θ. Points on the concentric circles centered on the CoV have constant elevation φ.

Figure 7. The kernels of the first partial derivatives gx, gy of Equation (11) at different positions on the Figure 7. The kernels of the ﬁrst partial derivatives g , g of Equation (11) at different positions on the fisheye image. x y ﬁsheyeFigure image. 7. The kernels of the first partial derivatives gx, gy of Equation (11) at different positions on the Similarly,fisheye image the. Laplacian of Gaussian for the calibrated fish-eye image is defined:

Similarly, the Laplacian of Gaussian22 for the calibrated fish-eye image is defined:()2 −−σσ22−d pp, 0 Similarly, the Laplacian ofdd Gaussian()pp,2 for the calibrated() fish-eye pp ,2 image is1 defined:2 LoG() x,; yσσ==00 g()p ; e 2σ (12) 2 4422 2 σσπσ2 22 ()d(p,p20) d(p, p ) −222σ 22d(p, p ) − 2σ 1 −d pp, 0 ()0 −−σσ()0 − 2 LoG(x, y; σ) = ddpp,200g(p; σ) = pp ,21 e σ22σ (12) LoG() x,; yσσ==σ4 g()p ; σ4 2πσ2 e 2 (12) The LoG kernel is shown at differentσσπσ442 locations on the fisheye image in2 Figure 8. It can be easily observed that the LoG defined using the geodetic distance between pixels depends on the location of The LoG kernel is shown at different locations on the fisheye image in Figure8. It can be easily the pixelThe thatLoG it kernel is applied is shown to. at different locations on the fisheye image in Figure 8. It can be easily observedobserved that that the LoGthe LoG defined defined using using the the geodetic geodetic distancedistance between between pixels pixels depends depends on the on thelocation location of of the pixelthe pixel that that it is it applied is applied to. to.

Figure 8. The LoG kernels of Equation (12) at different positions on the fisheye image.

2.3. An EfficientFigureFigure 8. ComputatioThe 8. The LoG LoG kernelsnal kernels Implementation of of Equation Equation of (12) (12)the Ge at di differentodeticallyfferent positions positionsCorrected on g, on the g thex ,fisheye gyﬁsheye and image.LoG image. Kernels for Fisheye Images for Any Value of σ 2.3. An Efficient Computational Implementation of the Geodetically Corrected g, gx, gy and LoG Kernels for The spatial dependence of the proposed geodetically corrected Gaussian kernel and its 2.3. AnFisheye Efﬁcient Images Computational for Any Value Implementationof σ of the Geodetically Corrected g, gx, gy and LoG Kernels for Fisheyederivatives Images for imposes Any Value a very of σhigh computational load, since the kernel itself has to be recomputed at each imageThe spatial pixel, priordependence to the actual of the convolution. proposed Ingeodet this section,ically corrected an approach Gaussian is presented kernel that and uses its Theaderivatives small spatial look-up imposes dependence table aof very the of Gaussianhigh the proposedcomputational kernel geodetically for aload, small since pre-selected corrected the kernel Gaussianvalue itself of has σ. kernel to be andrecomputed its derivatives at imposeseach a image very highpixel, computationalprior to the actual load, convolution. since the In kernel this section, itself hasan approach to be recomputed is presented at that each uses image pixel, a prior small to look-up the actual table convolution. of the Gaussian In kernel this section, for a small an approach pre-selected is presented value of σ.that uses a small look-up table of the Gaussian kernel for a small pre-selected value of σ.

J. Imaging 2018, 4, 73 10 of 21 J. Imaging 2018, 4, x FOR PEER REVIEW 10 of 21

First wewe generategenerate the the Gaussian Gaussian kernel kernel for for an image,an image, using using Equation Equation (8), defined (8), defined over aover 5 × 5a image5 × 5 window.image window. This is aThis demanding is a demanding computation computation that only needsthat only be performed needs be onceperformed after the once calibration after the of thecalibration camera. of The the Gaussian camera. kernelThe Gaus is storedsian kernel for each is pixelstored of for the each fisheye pixel image. of the Thus fisheye a total image. memory Thus for a atotal float memory array of forNL a× floatNc ×array25 is of required. NL × Nc × 25 is required. The method proposed in this work for thethe implementationimplementation of 2D convolutionconvolution using Gaussian kernel with higher σ,, is is successive successive convolutions convolutions with with a a smaller, smaller, initially initially constructed constructed Gaussian Gaussian kernel. kernel. It is wellwell knownknown thatthat convolvingconvolving twotwo GaussiansGaussians withwith standardstandard deviationsdeviations σσ11, σ2 results in another 2 2 2 Gaussian withwithσ σ2 == σσ112 + σ22.. This This fact fact isis testedtested numericallynumerically in in the the case case of of the the definition definition in Equationin Equation (9). More(9). More specifically, specifically, a point a point in the in fisheye the fisheye image isimage selected is selected near the near edge the of theedge FoV of wherethe FoV the where deviation the fromdeviation the simplefrom the projective simple projective image formation image formatio is moren is significant. more significant. The initial The initial Gaussian Gaussian kernel kernelg0 is constructedg0 is constructed according according to Equation to Equation (9) with(9) with a stencil a stencil of 13of ×13 13.× 13. The The standard standard deviation deviationσ σ00 ofof gg00 is calculated using Equation (8). A number of successive linear convolutions (*) is performed

g0 * g0 * g0 * …* g0 (13) g0 * g0 * g0 *...* g0 (13) Thus, the result of the kth convolution should be a Gaussian with σ0·(k + 1)1/2. 1/2 Thus,Figure the 9 shows result ofthe the constructionkth convolution of a 33 should × 33 stencil be a Gaussian Gaussian with kernelσ0·( k(b), + 1) at a .position near the edgeFigure of FoV9 shows(a), using the successive construction convolutions of a 33 × 33 of stencilan initial Gaussian smaller kernel 11 × 11 (b), kernel at a position (centered near at the edgesame ofpixel). FoV (a), using successive convolutions of an initial smaller 11 × 11 kernel (centered at the sameIn pixel). order to facilitate visual comparison, the midline of each kernel is plotted as a curve in Figure 9(d), Inup order to the to facilitatefirst 8 successive visual comparison, convolutions the midline (the first of eachkernel kernel is the is plottedtaller, asnarrower a curve incurve). Figure The9d, upgeodesic to the firstGaussian 8 successive that is convolutions generated using (the first Equation kernel is(9) the directly taller, narrower with the curve). appropriate The geodesic σ, is Gaussiansuperimposed that isusing generated symbol using “□”. Equation It can be (9)observed directly that with the the result appropriate of the 8σ successive, is superimposed convolutions using σ symbolwith the “ small”. It initial can be σ observed0 is almost that identical the result to ofthe the result 8 successive of a single convolutions convolution with with the the small much initial larger0 isand almost computationally identical to the very result demanding of a single kernel. convolution The as withymmetry the much of the larger Gaussian and computationally kernel is shown very in demandingFigure 9c using kernel. the Themidline asymmetry profile and of the the Gaussian mid-colu kernelmn profile is shown of the in kernel, Figure which9c using in thethis midline specific profilelocation and of thethe mid-columnimage, coincide profile with of the the kernel, direction which of inconstant this specific (iso-) location azimuth of and the image,constant coincide (iso-) withelevation. the direction The same of two constant profiles (iso-) are azimuth also plotted and constantafter reflection (iso-) elevation. with respect The to same the middle two profiles point are of alsothe profiles. plotted afterIt can reflection be observed with that respect the mid-column to the middle profile point is of symmetric the profiles. (the It reflected can be observed profile matches that the mid-columnthe original profile), profile is while symmetric the midline (the reflected profile profileshows matchesasymmetry the (the original profile profile), and its while reflection the midline does profilenot match). shows These asymmetry findings (the are profile expected, and its according reflection to does the not discussion match). Thesein the findings previous are subsection expected, according(Figure 6). to the discussion in the previous subsection (Figure6).

(a) (b)

Figure 9. Cont.

J. Imaging 2018, 4, 73 11 of 21 J. Imaging 2018, 4, x FOR PEER REVIEW 11 of 21

10-3 2.5 Midline profile Midline Profile reflected Mid-column profile 2 Mid-column Profile reflected

1.5

0.5

0 0 5 10 15 20 25 30 35 Pixels (c) 10-3 8

0 0 5 10 15 20 25 30 35 pixels (d)

FigureFigure 9. ((aa)) construction construction of of a a 33 33 × 3333 stencil stencil Gaussian Gaussian ke kernelrnel at at a a position position near near the the edge edge of of FoV FoV ( (bb),), usingusing successivesuccessive convolutions convolutions of anof initialan initial smaller smaller 11 × 1111 kernel × 11 (atkernel the same (at the pixel). same (b): pixel). the constructed (b): the constructedkernel and ( ckernel) its line and proﬁles (c) its along line theprofiles iso-azimuth along the (midline) iso-azimuth and iso-elevation (midline) and (mid-column) iso-elevation direction, (mid- column)(d): The midlinedirection, of (d the): The intermediately midline of the generated intermediately kernels generated up to the 8thkernels iteration up to are the shown 8th iteration with dots. are shownThe midline with ofdots. the directlyThe midline constructed of the kerneldirectly is constructed shown with “kernel”. The is resultshown of with successive “□”. The convolutions result of successiveof the initial convolutions Gaussian kernel. of the initial Gaussian kernel.

Thus,Thus, the proposed, efficient efﬁcient algorithm toto calculatecalculate thethe convolutionconvolution ofof aa calibratedcalibrated ﬁsheye fisheye image imageI σ σ I* * g (gx(x, ,y y; ; σ)) with with a a Gaussian Gaussian withwith largelarge valuesvalues ofof standardstandard deviationdeviation σ, using geodetic geodetic pixel pixel distance, distance, isis as as following following Algorithm 2. 2.

AlgorithmAlgorithm 2. 2. TheThe proposed proposed algorithm algorithm for for calculating calculating the the respon responsese of of a geodetically defined deﬁned Gaussian kernel onon a a calibrated calibrated fisheye ﬁsheye image, image, for for large large values of σ.. kk == 0 0 σ InitializeInitialize√ gg00 == gg((xx,,y y; ; σ0)0) while σσ+<σ0 k + 1 < σg while 0 k 1 g k = k + 1 k = k + 1 I = I * g0 end I = I * g0 end

J. Imaging 2018, 4, 73 12 of 21

2.3.1. Approximating the Normalized Geodetic LoG Using DoG The use of Difference of Gaussians (DoG) to approximate a Laplacian of Gaussian is well known in the literature [13], although the relation between the standard deviation of the two Gaussians and the Laplacian is seldom discussed in detail. Many approaches use the concept of octaves, namely deﬁne a number of convolution operations (typically 5) in order to double the value of σ and then resample the image to reduce the number of lines and column by two. The difference of the results of successive convolutions (DoG) approximates the convolution of the original image with the LoG. This multi-resolution approach is not directly applicable in the case of ﬁsheye image processing, where generation of Gaussian kernels with different σ is computationally expensive. Thus the following approach is proposed, which consists of constructing a single geodetically corrected Gaussian kernel with small standard deviation σ0 (whose values depend on its spatial location on the image) and perform successive convolutions with this kernel. At the kth iteration, the standard deviation of the equivalent Gaussian kernel will be equal to √ σ = σ0 k + 1. (14)

The iterations are stopped when the equivalent σ becomes greater than or equal to the required standard deviation. Differences of successive convolutions are assigned to IL variable, since they approximate the response of the original image with the LoG. The factor ak than normalizes the response of LoG with respect to σ is discussed. The steps of the Algorithm 3 for approximating the normalized LoG with DoG for a calibrated ﬁsheye image are given below. A schematic diagram of the proposed algorithm is shown in Figure 10.

Algorithm 3. Approximating normalized LoG with DoG in the case of ﬁsheye image. k = 0 σ Initialize√ g0 = g(x, y; 0), as in Equation (9) while σ0 k + 1 < σg k = k + 1 I1 = I * g0 IL = (I1 − I)ak I = I1 end

Since LoG is used in the concept of different image scales, the response of the LoG needs to be normalized with respect to scale σ. It is well known that σ2·LoG is scale-normalized:

2 2 σ2∑|LoG(x; σ2)| = ∑ σ1|LoG(x; σ1)|, σ2 > σ1 (15) x x

It is easy to conﬁrm that the partial derivative of a Gaussian with respect to σ is

∂g(x; σ) = σ · LoG ∂σ If the derivative is approximated using ﬁnite differences, then

∂g(x; σ) g(x; σ + δσ) − g(x; σ) = ≈ σ · LoG ⇒ DoG = δσ · σ · LoG (16) ∂σ δσ Thus the normalization factor in this case is σ a = (17) k δσ J. Imaging 2018, 4, 73 13 of 21

√ √ During the kth iteration δσ is equal to σ0 k + 1 − σ0 k. Substituting δσ in (16) we obtain: J. Imaging 2018, 4, x FOR PEER REVIEW 13 of 21 √ k + 1 √ ak = √ k +1 (18) a = k + 1 − k (18) k kk+−1

I∗g∗g∗g∗g σ 0 4

∗ a3 - I∗LoG

I∗g∗g∗g

1 σ 3 0.9 0

0.8 0.7 a 0.6 2 0.5 I∗LoG 0.4 ∗ - 0.3

0.2

0.1

0 -4 -3 -2 -1 0 1 2 3 4 ∗ ∗ Gaussian g with σ0 I g g σ 0 2 - I∗LoG ∗

I I∗g DoG approx. of LoG σ 0 ∗

FigureFigure 10. Schematic10. Schematic representation representation ofof thethe image resp responseonse of of the the Gaussian Gaussian kernel kernel and andthe LoG, the LoG, approximatedapproximated as DoG, as DoG, for for increasingly increasingly values valuesof of σσ..

2.3.2.2.3.2. Approximation Approximation of Firstof First Order Order Partial Partial Derivatives Derivatives of of the the Geodesic Geodesic Gaussian Gaussian First order partial derivatives of the Gaussian kernel are fundamental for image processing and First order partial derivatives of the Gaussian kernel are fundamental for image processing and feature extraction. Although these kernels are separable and therefore efficiently computed under feature extraction. Although these kernels are separable and therefore efficiently computed under their their classic definition Equation (10), the geodesic definition of Equation (11) imposes high classiccomputational definition Equation complexity, (10), especially the geodesic for large definition values of of σ. EquationThe proposed (11) method imposes for high approximating computational complexity,geodetically especially corrected for g largex, gy with values large of σvalues. The of proposed σ is the methodfollowing. for According approximating to their geodetically classic correcteddefinition,gx, gy sincewith the large Gaussian values is of a σslowis the varying following. function, According its derivative to their gx may classic be approximated definition, since as the Gaussianfiniteis differences. a slow varying In the function, case of geodetically its derivative definedgx may Gaussian, be approximated neighboring as pixels finite differences.have variable In the case ofdistance geodetically according defined to its locations, Gaussian, given neighboring in Dx, Dy defined pixels in Equation have variable (4). More distance formally: according to its locations, given in Dx, Dy defined in Equation (4).1 More formally: g ()xy,1,1,=−∗[] g() xy x () Dxyx , 1 gx(x, y) = [1, −1] ∗ g(x, y) (19) ()=−∗Dx(1x,y) []T () g y xy,1,1,1 T g xy (19) gy(x, y) = Dxy(), [1, −1] ∗ g(x, y) Dyy(x,y)

2.3.3.2.3.3. Detecting Detecting Corner Corner Pixels Pixels in in Fisheye Fisheye Image Image CornerCorner pixels pixels are salientare salient image image pixels pixels that that play play important important role role in manyin many image image analysis analysis tasks tasks [26 ,27]. The Harris[26,27]. cornerThe Harris detector corner isdetector a well-established is a well-established detector detector that that utilizes utilizes the the local local imageimage structure structure by calculatingby calculating the 2nd the order 2nd order moment moment matrix matrix at each at each image image pixel: pixel: 2 2  I x 2 IIxy  2 IIIxxy! xy,,∈∈ w∑ Ix xy w ∑ Ix Iy M ==wxy(),Ix Ix Iy x,y∈w x,y∈w (20) M = w(x, y) II I2 = I II2  (20) I I xyI2 y xy y 2  x y y xy,,∈∈ w∑ Ix Iy xy w ∑ Iy x,y∈w x,y∈w

J. Imaging 2018, 4, x FOR PEER REVIEW 14 of 21 J. Imaging 2018, 4, 73 14 of 21 where IIxy, are the image partial derivatives and w(x, y) is a user defined window. The image derivatives IIxy, can be approximated using derivative of the Gaussian kernel with appropriate where Ix, Iy are the image partial derivatives and w(x, y) is a user defined window. The image derivativesstandard deviationIx, Iy can σD. be The approximated summations usingover the derivative window ofw are the implemented Gaussian kernel as convolution with appropriate with a σ>σ /0.7 standardGaussian deviationwith standardσD. The deviation summations ID over the [26]. window w are implemented as convolution with a Gaussian with standard deviation σ > σ /0.7 [26]. I D ()2 2 IIIxxy M =∗()()σσg " 2 # DI(Ix) Ix Iy 2 (21) M = (σ )2g(σ ) ∗  (21) D I IIxy () I2 y Ix Iy Iy Corner response is calculated using the standard metric: det(M)-λ·trace(M)2, where λ a constant Corner response is calculated using the standard metric: det(M)-λ·trace(M)2, where λ a constant parameter with value in (0.04, 0.06). parameter with value in (0.04, 0.06). Direct application of the geodetically corrected derivative of the Gaussian, as defined in Direct application of the geodetically corrected derivative of the Gaussian, as defined in Equation (11) is not computationally efficient for arbitrary values of σ. The approximation of Equation (11) is not computationally efficient for arbitrary values of σ. The approximation of Equation (19) is used instead. Thus, the Algorithm 4 that we propose for estimation of matrix Μ in Equation (19) is used instead. Thus, the Algorithm 4 that we propose for estimation of matrix M in the the case of a calibrated fisheye image is described as following: case of a calibrated fisheye image is described as following:

Algorithm 4. The approximation of the Harris matrix M of Equation (21). Algorithm 4. The approximation of the Harris matrix M of Equation (21). Apply Algorithm 2 to calculate I * g Apply Algorithm 2 to calculate I * g Apply Equation (19) to calculate IIxy, Apply Equation (19) to calculateIx, Iy 22 2 ()()2 Apply AlgorithmAlgorithm 2 to2 (toIx ),IIII (xxyIv ),,,IxυIy with only with one only iteration one toiteration calculate to the calculate convolution the withconvolution the geodetic with

theGaussian geodeticg0. Gaussian g0. Construct the the symmetric symmetric matrix matrixM for M eachfor each pixel pixel

3. Results The results ofof AlgorithmAlgorithm 2 2 for for an an image image of of the the calibrated calibrated fisheye fisheye camera camera are are shown shown in Figurein Figure 11 for11 iterationfor iterationk = 1k = (first 1 (first row), row),k = k 10, = 10,k = k 20, = 20,k = k 30 = 30 and andk = k 40= 40 (last (last row). row). The The left left column column shows shows the the result result of Algorithmof Algorithm 2 (using2 (using the the geodetically geodetically defined defined Gaussian Gaussian kernel), kernel), the the middle middle columncolumn thethe responseresponse of the classic Gaussian kernel and the right column column the the ab absolutesolute difference, difference, scaled scaled for for visualization visualization purposes. purposes.

Figure 11. The results of Algorithm 1 for the geodetic Gaussian (left column) for the classic Gaussian Figure 11. The results of Algorithm 1 for the geodetic Gaussian (left column) for the classic Gaussian (middle column and their absolute difference (appropriately scaled for visualization). (middle column and their absolute difference (appropriately scaled for visualization).

J. Imaging 2018, 4, 73 15 of 21 J. Imaging 2018, 4, x FOR PEER REVIEW 15 of 21

ResultsResults ofof the the response response of theof geodesicthe geodesicgx kernel, gx kernel, as approximated as approximated using Algorithm using Algorithm 2 and Equation 2 and (18),Equation for iteration (18), for kiteration= 1 and k k= =1 and 20 are k =shown 20 are shown in Figure in Figure12 (left 12 column). (left column). In the In right the right column column the magnitudethe magnitude of the of the image image gradient gradient is shown,is shown, using usin theg the responses responses of of the the two two partial partial derivativesderivatives ofof thethe geodeticgeodetic GaussianGaussian kernel.kernel. ItIt isis evidentevident thatthat forfor smallsmall valuevalue ofof σσ√ == σσ00,, ﬁnefine detailsdetails appearappear inin thethe σ σσ magnitudemagnitude ofof thethe gradient,gradient, whereas,whereas, inin thethe casecase ofof kk == 20 (thus σ = 00 21),), onlyonly thickthick lineslines appearappear inin thethe magnitudemagnitude ofof thethe gradientgradient image.image.

(a) (b)

Figure 12. (a) The resulting Ix using the proposed algorithm for different scales (k = 1 and k = 20 in Figure 12. (a) The resulting Ix using the proposed algorithm for different scales (k = 1 and k = 20 in EquationEquation (19));(19)); ((bb)) thethe magnitudemagnitude ofof thethe imageimage gradientgradient at at the the same same scales. scales.

The results of the maximum response of the normalized geodetically corrected LoG, The results of the maximum response of the normalized geodetically corrected LoG, approximated approximated by DoG (Algorithm 3) are shown in Figure 13. The initial Gaussian kernel g0 was by DoG (Algorithm 3) are shown in Figure 13. The initial Gaussian kernel g was defined over a defined over a 7 × 7 window and 40 iterations were executed. Yellow indicates0 pixels that exhibited 7 × 7 window and 40 iterations were executed. Yellow indicates pixels that exhibited maximum DoG maximum DoG response within the first 4 iterations, whereas magenta indicates pixels with response within the first 4 iterations, whereas magenta indicates pixels with maximum DoG response maximum DoG response between 5 and 10 iterations. In (a) the whole fisheye frame is displayed, between 5 and 10 iterations. In (a) the whole fisheye frame is displayed, whereas in (b) details are whereas in (b) details are shown for a zoomed portion of the image. shown for a zoomed portion of the image. A comparison between the classic LoG response and the response of the geodetically defined A comparison between the classic LoG response and the response of the geodetically defined LoG, approximated by the DoG, (after normalization in both cases) is given in Figure 14(a). The LoG, approximated by the DoG, (after normalization in both cases) is given in Figure 14a. The curves curves have been produced for the pixel shown in the image portion of Figure 14(b). have been produced for the pixel shown in the image portion of Figure 14b.

J. Imaging 2018, 4, 73 16 of 21

J. Imaging 2018, 4, x FOR PEER REVIEW 16 of 21

(a) (b) Figure 13. (a) The results of the maximum response of the normalized geodetically corrected LoG, Figure 13. (a) The results of the maximum response of the normalized geodetically corrected LoG, approximated by DoG( a(Algorithm) 3), exhibited within the first 4 iterations( b(yellow) pixels) and approximated by DoG (Algorithm 3), exhibited within the ﬁrst 4 iterations (yellow pixels) and between Figurebetween 13. 5 (anda) The 10 iterationsresults of (magenta); the maximum (b) detailed response image of the portion. normalized geodetically corrected LoG, 5 and 10 iterations (magenta); (b) detailed image portion. approximated by DoG (Algorithm 3), exhibited within the first 4 iterations (yellow pixels) and between 5 and 10 iterations (magenta); (b) detailed image portion.

(a) (b)

Figure 14. (a) The response of the normalized DoG for the pixel shown in (b). The small differences between the two curves are( evident.a) (b) Figure 14. (a) The response of the normalized DoG for the pixel shown in (b). The small differences Figure 14. (a) The response of the normalized DoG for the pixel shown in (b). The small differences betweenThe response the two of curv Harrises are corner evident. detection is shown in Figure 15 for the whole frame. Yellow dots betweenindicate the corners two curves detected are by evident. the proposed geodetic Harris detector, whereas magenta squares depict cornerThe pixels response detected of Harris by the corner classic detection Harris isdetec showntor andin Figure blue 15crosses for the are whole corners frame. detected Yellow by dots the method described in [22]. All detectors were applied for a number of different image scales. Common Theindicate response corners of detected Harriscorner by the proposed detection geodetic is shown Harris in detector, Figure 15whereas for the magenta whole squares frame. depict Yellow dots cornerimage scalepixels is detecteddetermined by bythe the classic size ofHarris the kernels. detector In and the resultsblue crosses presented are cornersin this section, detected the by initial the indicate corners detected by the proposed geodetic Harris detector, whereas magenta squares depict methodsmall geodetic described Gaussian in [22]. Allis set detectors to a size were of 5applie × 5 pixels.d for a Thenumber size of of different the equivalent image scales. kernel Common after the corner pixelsimagesubsequent scale detected isconvolutions determined by the classic byis shownthe Harrissize in of Tables the detector kernels. 1 an andd In 2. the blueThe results classic crosses presented Harris-based are corners in this corner section, detected detection the by initial the is method describedsmallapplied in geodetic [22 using]. All Gaussianthis detectors size isfor set were the to aintegration applied size of 5 for ×Gaussian 5 a pixels. number Thekernel of size different(which of the isequivalent image slightly scales. kernellarger Common afterthan thethe image scale issubsequentdifferentiating determined convolutions byGaussian). the size isThe ofshown thegeodetic kernels.in Tables neighborhood In1 an thed results2. methodThe classic presented [22] Harris-based is parameterized in this corner section, by detectionthe the kernel’s initial is small field of view, measured in radians or degrees and the number of pixels of the kernel’s stencil. After geodeticapplied Gaussian using is this set tosize a sizefor ofthe 5integration× 5 pixels. Gaussian The size kernel of the (which equivalent is slightly kernel larger after than the subsequentthe visually studying the image resampling that is required by this method (an example is shown in convolutionsdifferentiating is shown Gaussian). in Tables The1 geo anddetic2. Theneighborhood classic Harris-based method [22] is corner parameterized detection by the is appliedkernel’s using fieldFigure of 6d)view, the measured selected parameterization in radians or degrees was as and follo thewing: number a field of ofpixels view of (FoV) the kernel’s of 1 degree stencil. was After used this sizevisuallyfor for 5 the× 5 studyingand integration 9 × 9 theand image Gaussianan FoV resampling of 2 kernel degrees that (which for is therequired is larger slightly bykernels. this larger method The than numerical (an the example differentiating assessment is shown of the Gaussian).in The geodeticFiguremethods 6d) neighborhood is the two-fold. selected Firstparameterization method the 250 [strongest22] was is parameterized as corner following: responses a field by areof the view considered kernel’s (FoV) of and field 1 degree the of number view,was used measuredof in radiansfor 5 or × 5 degrees and 9 × 9 and and thean FoV number of 2 degrees of pixels for the of larger the kernel’s kernels. The stencil. numerical After assessment visually studyingof the the image resamplingmethods is two-fold. that is requiredFirst the 250 by strongest this method corner (an responses example are isconsidered shown in and Figure the number6d) the of selected parameterization was as following: a field of view (FoV) of 1 degree was used for 5 × 5 and 9 × 9 and an FoV of 2 degrees for the larger kernels. The numerical assessment of the methods is two-fold. First the 250 strongest corner responses are considered and the number of correctly identified corners in the chessboard are enumerated. The chessboard contains 80 corners. The results are presented J. Imaging 2018, 4, 73 17 of 21

J. Imaging 2018, 4, x FOR PEER REVIEW 17 of 21 in Tablecorrectly1. The identified zoomed corners chessboard in the chessboard areas are shownare enumerated. in Figure The 15 chessboard for the ﬁrstcontains four 80 kernel corners. sizes of Table1.The Secondly, results are the presented number in ofTable wrong 1. The corner zoomed responses chessboard is areas measured are shown by visualin Figure inspection 15 for the for all three methods,first four forkernel four sizes (4) of different Table 1. kernelSecondly, sizes the (equivalently,number of wrong image corner scales). responses The is results measured are by measured consideringvisual theinspection 250, as for well all three as the methods, 500 strongest for four (4) corner different responses kernel sizes from (equ eachivalently, method. image The scales). results are The results are measured considering the 250, as well as the 500 strongest corner responses from each presented in Table2. method. The results are presented in Table 2.

(a) (b)

Figure 15. The chessboard area containing 80 corners, with detected corners form the three methods, Figure 15. The chessboard area containing 80 corners, with detected corners form the three methods, considering only the 250 strongest responses. (a) 5 × 5 kernels, (b) 9 × 9 kernels, (c) 13 × 13 kernels and considering(d) 17 × only 17 kernels. the 250 strongest responses. (a) 5 × 5 kernels, (b) 9 × 9 kernels, (c) 13 × 13 kernels and (d) 17 × 17 kernels. Table 1. Quantitative evaluation, in terms of corner detection, of the proposed geodetic kernel Table 1.definition,Quantitative the geodetic evaluation, neighborhood in terms definition of corner in detection,[22] and classic of the Harris-based proposed corner geodetic detection. kernel The definition, number of correctly detected corners in the chessboard (80 corners in total) by each method, the geodetic neighborhood definition in [22] and classic Harris-based corner detection. The number considering the 250 strongest responses are shown. of correctly detected corners in the chessboard (80 corners in total) by each method, considering the 250 strongestKernel responsesSize Classic are shown.Harris Proposed Geodetic Kernels Geodetic Neighborhood [22] 5 × 5 76 80 80 Kernel9 Size × 9 Classic 80 Harris Proposed Geodetic 80 Kernels Geodetic 80Neighborhood [22] 13 × 13 75 80 80 5 ×175 × 17 76 53 80 80 80 80 9 ×219 × 21 80 48 80 80 80 80 13 × 13 75 80 80 17 × 17 53 80 80 21 × 21 48 80 80 J. Imaging 2018, 4, x FOR PEER REVIEW 18 of 21 J. ImagingJ. Imaging2018 2018, 4, ,4 73, x FOR PEER REVIEW 18 of18 21 of 21 Table 2. False corner detection of the proposed geodetic kernel definition, the geodetic neighborhood Tabledefinition 2. False in [22]corner and detection classic Harris-based of the proposed corner geodetic detection. kernel The definition, number ofthe wrongly geodetic detected neighborhood corners Tabledefinitionby each 2. False method, in [22] corner and considering detection classic Harris-based the of the250 proposedand corner500 strongest geodetic detection. responses kernel The number definition, are shown. of wrongly the geodetic detected neighborhood corners definitionby each method, in [22] andconsidering classic Harris-based the 250 and 500 corner strongest detection. responses The are number shown. of wrongly detected corners Kernel by each method, considering250 Strongest theCorner 250 Responses and 500 strongest responses 500are Strongest shown. Corner Responses KernelSize 250 Strongest Corner Responses 500 Strongest Corner Responses Size Proposed Proposed Classic 250 Strongest Corner ResponsesGeodetic Classic 500 Strongest Corner ResponsesGeodetic ProposedGeodetic ProposedGeodetic Kernel Size ClassicHarrisClassic Proposed NeighborhoodGeodeticGeodetic [22] ClassicHarrisClassic Proposed NeighborhoodGeodeticGeodetic [22] Harris GeodeticKernels Kernels Neighborhood [22] Harris GeodeticGeodeticKernels Kernels Neighborhood [22] Harris Neighborhood [22] Harris Neighborhood [22] 55× × 55 9 9Kernels 2 2 21 21 11 11Kernels 4 4 68 68 599 × × 5 99 9 10 10 2 1 1 21 31 31 11 35 35 4 5 5 68 72 72 13 × 13 12 2 17 42 7 69 139 × × 9 13 10 12 1 2 31 17 35 42 5 7 72 69 17 × 17 19 4 28 70 8 72 132117 × × 13 2117 12 19 28 2 4 8 17 28 27 42 70 68 7 8 12 69 72 65 1721 × × 17 21 19 28 4 8 28 27 70 68 8 12 72 65 21 × 21 28 8 27 68 12 65 Two different parts of the fisheye frame are zoomed in Figure 16. Visual inspection of the results Two different parts of the fisheye frame are zoomed in Figure 16. Visual inspection of the results suggests that the proposed geodetic Harris detector has fewer false corners than the classic one. suggestsTwo differentthat the partsproposed of the geodetic fisheye frameHarris are detec zoomedtor has in Figurefewer false16. Visual corner inspections than the of classicthe results one. FiguresuggestsFigure 17 17 showsthat shows the a a fewproposed few sporadic sporadic geodetic false false cornersHarriscorners detec detecteddetectetord has by by the thefewer classic classic false Harris Harriscorner method,s method, than theprobably probably classic due one. due to to theFigurethe texture texture 17 shows of of the the floor.a floor.few Thesporadic The method method false of ofcorners [ 22[22]] exhibitsex detectehibits falsed by thedetections classic Harrismainly mainly method, on on image image probably edges. edges. due to the texture of the floor. The method of [22] exhibits false detections mainly on image edges.

(a) 250 strongest responses (b) 500 strongest responses (a) 250 strongest responses (b) 500 strongest responses Figure 16. Corner pixels detected by the proposed geodetically corrected Harris detector (shown in Figure 16. Corner pixels detected by the proposed geodetically corrected Harris detector (shown in Figureyellow) 16. by Corner the classic pixels detector detected (mag byenta) the proposed and by the geod geodeticetically Harris corrected in [22] Harris (blue detector crosses), (shown with 5 in× 5 yellow) by the classic detector (magenta) and by the geodetic Harris in [22] (blue crosses), with 5 × 5 yellow)kernel size,by the considering: classic detector (a) the (mag 250enta) strongest and bycorner the geodeticresponses Harris and ( bin) the[22] 500 (blue strongest crosses), responses. with 5 × 5 kernel size, considering: (a) the 250 strongest corner responses and (b) the 500 strongest responses. kernel size, considering: (a) the 250 strongest corner responses and (b) the 500 strongest responses.

(a) (b) (a) (b) Figure 17. (a) and (b): Two different parts of the fisheye frame from the result with 500 strongest Figurecorner 17.responses, (a) and using(b): Two kernels different of size parts 17 × of17 the(color fisheye encoding frame is thefrom same the asresult in the with previous 500 strongest Figure). Figure 17. (a) and (b): Two different parts of the ﬁsheye frame from the result with 500 strongest corner corner responses, using kernels of size× 17 × 17 (color encoding is the same as in the previous Figure). responses, using kernels of size 17 17 (color encoding is the same as in the previous Figure).

J. Imaging 2018, 4, 73 19 of 21

The presented results can be summarized as following. The proposed definition and implementation of image processing geodetic kernels exhibited very good corner detection over a number of different kernel sizes: it managed to detect all corners in the chessboard area for kernel size ranging between 5 × 5 to 21 × 21 pixels. The same holds for the method of geodetic image neighborhood resampling [22]. On the other hand, the classic Harris-based corner detection does not perform well on the fisheye images when the size of the kernels (image space) increases. Concerning false corner detection, the proposed algorithm achieved very low rate for the 5 different kernel sizes that it was tested. On the contrary, the classic Harris-based detector exhibited increased false detection rate as the kernel size increases, whereas the method of [22] exhibited high false detection rate for almost all tested kernel sizes. The main algorithms described in this work have been executed on a laptop with Intel core i7-7700 HQ, 16 GB, with Windows 10, using the Matlab programming environment. The algorithms were applied to 480 × 640 fisheye images. The recorded times of various steps of the methods are shown in Table3. It has to be mentioned again that the slowest part of the proposed algorithm, the initial calculation of the geodetic Gaussian kernel g0 with small σ0 (first entry in Table3) is executed only once, after the calibration of the camera and it can be subsequently used for any image acquired by the specific camera. As it can be observed in Table3, Algorithm 1 is the computationally intensive part of the proposed method. Execution times using the classic kernels are also provided for comparison. The geodetic approach for fisheye image processing using geodetic neighborhood [22] has also been timed. The slowest part of the method is the image resampling, which was timed for the following parameterization: FoV of 1 degree, kernel size of 5 × 5. Increasing the FoV and consequently the kernel size imposes a significant overhead to the algorithm [22]. Furthermore, this operation has to be performed for every acquired image.

Table 3. Execution times for the proposed method, corner detection in [22] and classic corner detection.

Method Calculation Execution Time (Second)

Proposed Geodetic kernels geodetic Gaussian g0 (Algorithm 1) 5.5 Geodetic neighborhood [22] Image resampling, 5 × 5, FoV 1◦ 150 Proposed Geodetic kernels Gaussian response (Algorithm 2) 0.15 Geodetic neighborhood [22] Gaussian response 0.017 Classic Harris Gaussian response 0.017 Proposed Geodetic kernels Harris Matrix M 0.5 Classic Harris Harris Matrix M 0.5 Geodetic neighborhood [22] Harris Matrix M 0.5

4. Discussion In this work, the most important image processing kernels, namely, the Gaussian, its partial derivatives with respect to x and y and the Laplacian of Gaussian (LoG) have been redefined for a calibrated fisheye camera, using the geodesic distance between pixels, of the fisheye image. For each kernel, a computationally efficient algorithm that calculates/approximates the response with a fisheye image acquired by a calibrated camera, based on successive convolutions with a small pre-initialized geodetic Gaussian, has been presented. The Harris corner detection has also been implemented for a calibrated fisheye camera, using the image responses with the redefined geodetic kernels. The results show that the proposed definitions and the computational implementations are efficient and present subtle differences from the applications of the classic image processing kernels. Corner detection in calibrated fisheye images appears more robust when using the proposed operators. More specifically, both the proposed method of geodetic definition of image processing kernels and the method of geodetic image neighborhood resampling achieved very high corner detection for all tested kernel sizes. The performance of the classic Harris-based corner detector, deteriorates as the kernel size increases. The number of falsely indicated corners is much lower for the proposed method, consistently J. Imaging 2018, 4, 73 20 of 21 for all tested kernel sizes. The geodetic neighborhood method [22] is producing a large number of false corner detections over all tested kernel sizes, whereas the classic corner detector exhibited false detection rate that deteriorates as the kernel size increases. The execution time of the proposed method was slower than the corresponding time using the classic kernels, but not prohibitive for practical use. The most demanding step was the initial calculation of the initial geodetic Gaussian g0 with small σ0, which needs only be performed once, after camera calibration. The approximation of the Gaussian response which was averaged to 0.017 s per iteration for a 480 × 640 image. Possible parallel of this step may achieve important acceleration. Further work is needed to expand these results into more image analysis/feature extraction methods from fisheye images.

Funding: This research received no external funding Conﬂicts of Interest: The author declares no conﬂict of interest.

References

1. Geyer, C.; Daniilidis, K. Catadioptric projective geometry. Int. J. Comput. Vis. 2001, 45, 223–243. [CrossRef] 2. Bermudez-Cameo, J.; Lopez-Nicolas, G.; Guerrero, J.J. Automatic line extraction in uncalibrated omnidirectional cameras with revolution symmetry. Int. J. Comput. Vis. 2015, 114, 16–37. [CrossRef] 3. Ahmed, M.S.; Gao, Z. Ultra-wide fast fisheye for security and monitoring applications. In Proceedings of the International Symposium on Optoelectronic Technology and Application 2014: Advanced Display Technology; and Nonimaging Optics: Efficient Design for Illumination and Solar Concentration, Beijing, China, 13–15 May 2014. 4. Caruso, D.; Engel, J.; Cremers, D. Large-scale direct slam for omnidirectional cameras. In Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 28 September–2 October 2015; pp. 141–148. 5. Choi, Y.W.; Kwon, K.K.; Lee, S.I.; Choi, J.W.; Lee, S.G. Multi-robot Mapping Using Omnidirectional-Vision SLAM Based on Fisheye Images. ETRI J. 2014, 36, 913–923. [CrossRef] 6. Abrams, A.D.; Pless, R.B. Webcams in context: Web interfaces to create live 3D environments. In Proceedings of the 18th ACM International Conference on Multimedia, Toronto, ON, Canada, 26–30 October 2010; pp. 331–340. 7. Sankaranarayanan, K.; Davis, J.W. A fast linear registration framework for multi-camera GIS coordination. In Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance, Santa Fe, NM, USA, 1–3 September 2008. 8. Hu, X.; Zheng, H.; Chen, Y.; Chen, L. Dense crowd counting based on perspective weight model using a fisheye camera. Optik-Int. J. Light Electron Opt. 2015, 126, 123–130. [CrossRef] 9. Vandewiele, F.; Motamed, C.; Yahiaoui, T. Visibility management for object tracking in the context of a fisheye camera network. In Proceedings of the 2012 Sixth International Conference on Distributed Smart Cameras (ICDSC), Hong Kong, China, 30 October–2 November 2012; pp. 1–6. 10. Wang, W.; Gee, T.; Price, J.; Qi, H. Real time multi-vehicle tracking and counting at intersections from a fisheye camera. In Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 5–9 January 2015; pp. 17–24. 11. Rhodin, H.; Richardt, C.; Casas, D.; Insafutdinov, E.; Shafiei, M.; Seidel, H.P.; Theobalt, C. Egocap: Egocentric marker-less motion capture with two fisheye cameras. ACM Trans. Graph. 2016, 35, 162. [CrossRef] 12. Rhodin, H.; Robertini, N.; Richardt, C.; Seidel, H.P.; Theobalt, C. A versatile scene model with differentiable visibility applied to generative Pose Estimation. In Proceedings of the 2015 International Conference on Computer Vision (ICCV 2015), Tampa, FL, USA, 5–8 December 2015. 13. Lowe, D.G. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; Volume 2, pp. 1150–1157. 14. Hansen, P.; Corke, P.; Boles, W.; Daniilidis, K. Scale Invariant Feature Matching with Wide Angle Images. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, 29 October–2 November 2007; pp. 1689–1694. J. Imaging 2018, 4, 73 21 of 21

15. Bulow, T. Spherical diffusion for 3D surface smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 1650–1654. [CrossRef][PubMed] 16. Cruz-Mota, J.; Bogdanova, I.; Paquier, B.; Bierlaire, M.; Thiran, J.P. Scale invariant feature transform on the sphere: Theory and applications. Int. J. Comput. Vis. 2012, 98, 217–241. [CrossRef] 17. Puig, L.; Guerrero, J.J. Scale space for central catadioptric systems: Towards a generic camera feature extractor. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 1599–1606. 18. Puig, L.; Guerrero, J.J.; Daniilidis, K. Scale space for camera invariant features. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1832–1846. [CrossRef][PubMed] 19. Andreasson, H.; Treptow, A.; Duckett, T. Localization for mobile robots using panoramic vision, local features and particle ﬁlter. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2005; pp. 3348–3353. 20. Zhao, Q.; Feng, W.; Wan, L.; Zhang, J. SPHORB: A fast and robust binary feature on the sphere. Int. J. Comput. Vis. 2015, 113, 143–159. [CrossRef] 21. Hara, K.; Inoue, K.; Urahama, K. Gradient operators for feature extraction from omnidirectional panoramic images. Pattern Recognit. Lett. 2015, 54, 89–96. [CrossRef] 22. Demonceaux, C.; Vasseur, P.; Fougerolle, Y. Central catadioptric image processing with geodesic metric. Image Vis. Comput. 2011, 29, 840–849. [CrossRef] 23. Delibasis, K.K.; Plagianakos, V.P.; Maglogiannis, I. Reﬁnement of human silhouette segmentation in omni-directional indoor videos. Comput. Vis. Image Underst. 2014, 128, 65–83. [CrossRef] 24. Delibasis, K.K.; Georgakopoulos, S.V.; Kottari, K.; Plagianakos, V.P.; Maglogiannis, I. Geodesically-corrected Zernike descriptors for pose recognition in omni-directional images. Integr. Comput.-Aided Eng. 2016, 23, 185–199. [CrossRef] 25. Puig, L.; Bermúdez, J.; Sturm, P.; Guerrero, J.J. Calibration of omnidirectional cameras in practice: A comparison of methods. Comput. Vis. Image Underst. 2012, 116, 120–137. [CrossRef] 26. Harris, C.; Stephens, M. A combined corner and edge detector. In Proceedings of the Alvey Vision Conference, Manchester, UK, 31 August–2 September 1988; Volume 15, pp. 10–5244. 27. Mokhtarian, F.; Mohanna, F. Performance evaluation of corner detectors using consistency and accuracy measures. Comput. Vis. Image Underst. 2006, 102, 81–94. [CrossRef]

© 2018 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).