Cata-Fisheye for Panoramic Imaging

Gurunandan Krishnan Shree K. Nayar Department of Computer Science Columbia University {gkguru,nayar}@cs.columbia.edu

Abstract Current techniques for capturing panoramic videos can We present a novel panoramic imaging system which uses be classified as either dioptric (systems that use only refrac- a curved mirror as a simple optical attachment to a fish- tive optics) or catadioptric (systems that use both reflective eye . When compared to existing panoramic , and refractive optics). Dioptric systems include camera- our “cata-fisheye” camera has a simple, compact and inex- clusters and wide-angle or fisheye lens based systems. Cata- pensive design, and yet yields high optical performance. It dioptric systems include ones that use single or multiple captures the desired panoramic field of view in two parts. cameras with one or more reflective surfaces. The upper part is obtained directly by the fisheye lens and Dioptric camera-clusters [14, 20, 28] compose high reso- the lower part after reflection by the curved mirror. These lution using images captured simultaneously by two parts of the field of view have a small overlap that is multiple cameras looking in different directions. In this ap- used to stitch them into a single seamless . The proach, due to their finite size, the cameras cannot be co- cata-fisheye concept allows us to design cameras with a located. Hence, they do not share a single viewpoint and, wide range of fields of view by simply varying the param- in general, have significant parallax. To address this issue, eters and position of the curved mirror. We provide an au- Nalwa [22] uses a pyramidal (piecewise planar) mirror to tomatic method for the one-time calibration needed to stitch co-locate the viewpoints of multiple cameras, thus eliminat- the two parts of the panoramic field of view. We have done ing the parallax (also see [17]). In all cases, the use of multi- a complete performance evaluation of our concept with re- ple cameras requires careful geometric and photometric cal- spect to (i) the optical quality of the captured images, (ii) ibration of the cameras as well as synchronization with re- the working range of the camera over which the parallax is spect to time. Moreover, the use of multiple cameras can negligible, and (iii) the spatial resolution of the computed make the imaging system bulky and expensive. panorama. Finally, we have built a prototype cata-fisheye A popular dioptric solution is the use of fisheye video camera with a spherical mirror that can capture high [21]. Since they have a hemispherical FOV, fisheye lenses resolution panoramic images (3600x550 ) with a 360 ◦ can only be used to capture hemispherical panoramas [31]. (horizontal) x 55◦ (vertical) field of view. Slater [26] proposed the use of two back-to-back fisheye lens cameras to obtain full spherical panoramas. Similar to 1. Introduction camera-clusters, this approach suffers from large parallax. Wide-angle and panoramic imaging have had significant Also, if we need to capture a reasonable vertical coverage impact on a variety of real-world applications. For exam- about the horizon, say +30◦ to -30◦, then in terms of optics, ple, being able to “see” in all directions provides situational image detectors and bandwidth, it is wasteful to capture the awareness in surveillance and autonomous navigation tasks. entire . Image-based immersive virtual reality [1, 2, 11, 13] enables Catadioptric systems [8, 9, 10, 12, 16, 25, 30] employ realistic exploration of indoor and outdoor environments. one or more reflective surfaces capable of projecting a wide Panoramic video conference [3] and telepresence devices view of the world onto a single sensor. Several of these de- aid in effective virtual collaboration. All these applications signs [18, 22, 23, 24] satisfy the so-called single viewpoint rely on capturing high-quality panoramic videos. In many (SVP) constraint. Baker and Nayar [4] derived the family of of these applications, it is not required to capture the entire mirrors that satisfy the SVP constraint, when used with per- spherical field of view (FOV). It is sufficient to capture 360 ◦ spective lenses. Since in these designs cameras panoramas with a reasonable vertical field of view that has view the world through highly curved reflective surfaces, the more or less equal coverage above and beneath the horizon. vertical resolution (number of pixels per unit elevation an- For instance, in the case of video conferencing, we only gle) of the computed panoramas is not uniform and is, in need to capture the sitting participants and the table. The general, poor near the apex of the mirror [23]. same is true for navigation, where the vehicle only needs to Yagi and Kawato [29] and Lin and Bajscy [18] have used view the roads and the obstacles around it. In this paper, our conical mirrors in their catadioptric designs, with the latter focus is on capturing such panoramic fields of view. satisfying the SVP constraint. In this case, we obtain the

1 best vertical resolution. The conical mirror acts as a planar θfu=θau mirror in the radial direction and hence the vertical resolu- tion is the same as that of the camera. However, this re- θmu stricts the maximum vertical FOV of the panorama to half Mirror Δθ the FOV of the camera. For instance, if we wish to capture a o panorama with a 60 degree vertical field of view, the camera θ Fisheye Lens O fl must have a 120 degree field of view. Even if such a system is used, the resolution of the panorama approaches zero at the apex of the cone. More importantly, conical mirrors are θml ≥ θal known to produce strong optical aberrations due to astigma- tism that fundamentally limit the resolution of the captured panorama [27]. Detector In this work, we present a novel panoramic imaging sys- tem that uses a curved mirror as a simple and compact at- Figure 1. The cata-fisheye system “looks” at the world partly (from θ θ θ θ tachment to a fisheye lens. We refer to this as a cata-fisheye fu to fl) through the fisheye lens and partly (from mu to ml) through the convex mirror. ∆θo is a small overlap between the two system. Fisheye lenses usually have a hemispherical FOV. fields of view. All angles corresponding to the incident rays are In our system (see Figure 1), the curved mirror is placed measured from the vertical optical axis. within the central part of the fisheye FOV. This enables us to capture light rays beyond the hemispherical FOV of the fish- eye lens. A similar system, proposed by Davis et al. [15], θ fl θfu uses a flat mirror to capture panoramic images with vertical θ ◦ ◦ fu FOV below the horizon (less than 0 and greater than -45 ). θ 180o mu Unlike their design, in our system the desired panorama is θfl captured in two parts - the upper part is captured directly θml through the fisheye lens, while the lower part is obtained θ mu o o θml through the mirror. These two parts are seamlessly stitched 0 360 to produce the final panorama. Our approach to panoramic imaging can be viewed as a (a) (b) hybrid of the dioptric and catadioptric approaches. As we Figure 2. (a) Illustration of image formed by a rotationally sym- shall show, it overcomes many of the limitations of previous metric cata-fisheye system. (b) The final panorama is generated by systems. Although our design is in part catadioptric, it uses stitching the mirror-panorama and the fisheye-panorama. a very shallow mirror that produces minimal optical aberra- θfu = θau, (1a) tions. Thus, we are able to capture high quality panoramas. θ ≥ θ , Our system captures its two parts of the panoramic field of ml al (1b) view from two compact viewpoints. In this respect, it too ∆θo > 0. (1c) has an inherent parallax. However, the two viewpoints are very close together and hence the parallax is negligible ex- where, ∆θo =(θfl − θmu) is the overlap between the two cept for very small distances from the system. FOVs. Also, θfl =Ωf /2, where Ωf is the FOV of the fish- 1 We have developed a prototype camera that captures high eye lens . The first two conditions, (1a) and (1b), state that resolution panoramas with a 360◦ x55◦ field of view. Al- the mirror and the lens should not obstruct the panoramic though our prototype has almost equal coverage above and FOV and the condition (1c) ensures that no part of the de- below the horizon, cata-fisheye systems can be designed sired panoramic FOV is missed. All angles are measured with a wide range of vertical fields of view. from the vertical optical axis of the fisheye lens. Figure 2a illustrates the image as formed by a cata- 2. Cata-Fisheye Image Formation fisheye system. Unlike the image obtained by a regular fish- eye lens, the cata-fisheye image is not continuous with re- Figure 1 illustrates the design of a radially symmetric spect to the FOV. The outer orange annulus represents the cata-fisheye system. It uses a mirror in conjunction with upper FOV imaged directly through the fisheye lens. The a fisheye lens in order to capture light rays from beyond the inner blue annulus represents the FOV captured through the FOV of the fisheye lens. If the overall vertical field of view θ θ mirror. The small overlap is depicted as thin gray rings at of the system extends from au to al, then the upper part of the outer boundaries of the two views. The final panorama the FOV (from θfu to θfl) is captured directly by the fish- θ θ is produced by stitching the two individual panoramas corre- eye lens and the remainder (from mu to ml) through the sponding to the FOVs of the mirror (mirror-panorama) and mirror. A small overlap in the two views allows them to be the fisheye lens (fisheye-panorama). We discuss the genera- seamlessly blended into the final panorama. tion of panoramas in detail in Sections 4 and 5. The properties of the cata-fisheye system shown in Figure 1The FOV of a fisheye lens is usually specified as twice its vertical 1, can be concisely expressed as: coverage, i.e. 180◦ fisheye lens covers 90◦ in the vertical direction. Panorama Fisheye Design Solution d θfu m FOV FOV (in mm) P (Required) (Chosen) rm dm hm θmu 170◦ 70.00 47.88 4.6 hm n 180◦ 76.36 44.45 4.5 h O θmu l +30◦ to -30◦ 190◦ 81.49 42.18 4.4 (60◦ to 120◦) 200◦ 96.22 39.73 4.2 210◦ 106.98 37.15 4.1

Figure 4. Given the projection function and field of view of the fish- Figure 3. A cata-fisheye camera design with a spherical mirror with eye lens and the desired panoramic field of view, the radius of cur- radius of curvature rm and width dm is placed at a height of hm vature, width and position of the spherical mirror are determined from the fisheye lens viewpoint O. The point P lies on the rim of using the design equations in Section 3. This table shows a few ∆θ =5◦ the mirror and has a surface normal nˆ. examples of cata-fisheye camera designs. In all cases, o , hl =5mm and dl =55mm. 3. Designing a Cata-Fisheye Camera 1 The cata-fisheye concept allows us to design cameras θnˆ = (π + θfu + θmu). (3) with a wide range of FOVs by simply varying the param- 2 From all of the above constraints, the radius of curvature eters (shape and position) of the curved mirror and the FOV of the fisheye lens. In this section, we discuss determining and the width of the mirror can be easily determined to be: the parameters of the mirror for a desired panoramic FOV. (hm + hl)sinθfu rm = , (4) The mirror used in a cata-fisheye system can have a wide sin(θfu − θnˆ ) − sin θfu range of shapes - spherical, paraboloidal, hyperboloidal, h h θ θ d 2( m + l)sin nˆ sin fu . conical, or even a complex aspherical shape. To minimize m = θ − θ − θ (5) optical aberrations (maximize image resolution) we have sin( fu nˆ ) sin fu chosen to use spherical mirrors. Further, to keep the system Note that the radius and the width are computed as a func- rotationally symmetric, the mirror is assumed to be coaxial tion of the position hm of the mirror. While any arbitrary with the fisheye lens. hm and a mirror with the corresponding r m and dm guaran- Another design parameter that needs to be set is the tee the upper limit (condition (1a)) and the required overlap amount of overlap between the upper and lower views. (condition (1c)), they need not ensure that the lens will not Since the stitching of the two views takes place in the im- obstruct the panoramic field of view(condition (1b)). There- age, it is convenient to specify the overlap in terms of the fore, we perform a linear search to find the smallest h m for number of pixels, po. The corresponding angular overlap in which θml ≥ θal. the two FOVs is given by, Note that there can be several design solutions for a de- −1 sired panoramic field of view, based on the FOV of the fish- ∆θo =Ωf /2 − f [f(Ωf /2) − po]. (2) eye lens and the overlap between the upper and lower views. Here, r = f(θ) is the projection function of the fisheye lens, The table in Figure 4 shows a few examples of cata-fisheye where r is the radial distance of a from the image cen- camera designs. In practice, we select the best optical de- ter and θ is the elevation angle of the incident ray with re- sign by evaluating all the possible designs with respect to spect to the optical axis of the fisheye lens. several performance measures, including the resolution of Figure 3 illustrates the two-dimensional cross-section of the panorama, the camera’s working range with respect to a typical cata-fisheye system with a spherical mirror. The parallax, and the optical quality of the image. We discuss fisheye lens is assumed to have a single viewpoint with its these performance measures in detail in Section 7. position roughly known to be at O. 2 hl is the distance be- tween O and the tip of the lens and dl is the width of the lens 4. Calibration of Cata-fisheye Camera (including its casing). The spherical mirror, whose radius of In order to geometrically map the captured image to the curvature rm and width dm are to be determined, is assumed upper and lower panoramas and then stitch them, the rela- to be positioned at a distance hm from the lens. tionship between the light rays entering the camera and the Let P be a point on the rim of the mirror. According to pixels in the captured image has to be known. This is com- conditions in Equation (1), the position of P and the surface pletely governed by the design parameters. Due to manu- normal nˆ at P (and hence the radius and center of curva- facturing tolerances, however, the actual position and ori- ture of the mirror) should be such that an incoming ray at entation of the mirror need not adhere strictly to the design angle θmu is reflected by P towards O (see Figure 3). To specifications. This, compounded with the possible inaccu- satisfy this condition, the directional angle of the normal at racy in the assumed fisheye lens viewpoint, requires us to P should be: calibrate the system to determine (i) the fisheye projection function r = f(θ), and (ii) the actual center of curvature 2Since fisheye lenses do not exactly have a single viewpoint, its as- (xm,ym,zm,) and orientation (αm,βm) of the mirror. To sumed position may compromise the design and therefore the field of view of the imaging system. One way to solve this problem is to add tolerances calibrate the fisheye lens, either a projection function pro- to the limits of the desired field of view. vided by the manufacturer can be used or any of the existing 2 calibration methods may be used [5, 6, 7]. Here, we focus arg min [(θp,φp) − R(θc,φc)] . (7) (θ ,φ ) on the second part of the calibration. c c We pose the calibration as a process of searching for a so- Subsequently, the required image location is given by 5 (f(θc)cosφc,f(θc)sinφc). Observe that, solving the above lution M˜ =(xm,ym,zm,αm,βm) in an R space (M) for which corresponding local features in the overlapping FOV problem for each pixel in the mirror’s FOV can be compu- of the two views are aligned in the panoramic space. The tationally expensive. Fortunately, we are not required to do camera is first placed in an environment with a large number so. Since the reflecting point on the spherical mirror varies of distant features, particularly in the overlapping region. smoothly with the incoming ray direction, we only need to Nearby features should be avoided as they would have larger solve for pixels at discrete intervals and interpolate the so- parallax and hence tend to introduce errors in the align- lution for in-between pixels. ment process. With the solution initialized to the design Finally, if the pixel lies in the overlapping region, we em- ploy linear feathering to smoothen the blending. The entire values, M˜ =(0, 0,hm + hl, 0, 0), we generate the mirror- mapping between the panorama and the captured image, can panorama and the fisheye-panorama for the overlapping re- be stored as a simple lookup-table. Using the lookup table, gion. Using SIFT[19], we then extract a corresponding set we can map the captured images to high resolution panora- of features from the two panoramas. The extracted fea- mas in real time. Note that the calibration process and the tures are represented in spherical coordinates. Any spurious mapping calculations need to be performed just once for any matches resulting from this procedure can be easily sieved given cata-fisheye camera. out by using the constraint that matched features should have very similar spherical coordinates. In the pruned fea- 6. Camera Prototype i ture set, let the th feature in the mirror-panorama be de- Figure 5b shows our prototype cata-fisheye camera that (i) (i) (i) ◦ ◦ ◦ noted by Fm =[θm ,φm ] and the corresponding feature has a vertical FOV of 55 (+25 to -30 ). It was built using (i) (i) (i) in the fisheye-panorama be denoted by F f =[θf ,φf ]. off-the-shelf components – a 7 frames per second Lumn- 3 Now the problem of finding the solution M˜ can be setup as era Lw570 digital video camera witha5megapixel sensor and a Fujinon FE185C046HA-1 fisheye lens with a FOV of a minimization of an objective function that is the sum of the ◦4 squares of the errors in the alignment of the corresponding 185 . Usually, an overlap of 20 to 30 pixels is sufficient  to perform blending. However, in our prototype, the image features: (i) (i) 2 argmin [Ff − Fm (M˜ )] . (6) formed by the fisheye lens is much larger than the sensor M˜ (see Figure 5c), resulting in the image being chopped off at Note that, during this optimization, it is not required to the top and bottom. Since the missing parts of the FOV are R5 search for the solution in the entire space. We only need near the equator, they could appear as holes in the computed M to search in a small interval ∆ centered at the ideal de- panorama. In order to avoid these holes, we set the overlap sign parameter values. The size of the interval is chosen to be larger than the holes (20◦). Using Equations 4 and 5, based on the manufacturing tolerances and the ambiguity in the radius and width of the mirror were determined to be fisheye lens viewpoint. 70.12mm and 50.26mm, respectively, and its position to be 5. Computing the Panorama 2.6mm above the tip of the lens. Once the system is calibrated, generating the panorama With respect to packaging, our camera design is ex- with the required resolution is straightforward. Each pixel tremely compact. The size of the complete system is just in the panorama represents a spherical coordinate/direction, 10cmx7cmx5.5cm (see Figure 5b). Moreover, the packag- and for each pixel we need to find the corresponding loca- ing is such that the mirror can be easily detached from the tion in the captured image. We consider three possibilities lens, thereby converting the cata-fisheye camera into a reg- ular hemispherical camera. here. A panorama pixel (θp,φp) may lie in (i) the direct FOV of the fisheye lens, (ii) the FOV of the mirror and (iii) Figures 10 and 11 show two examples of panoramas cap- tured using our cata-fisheye camera. The first example cor- both the fisheye lens’ and the mirror’s FOVs. If (θ p,φp) is in the fisheye lens’ FOV, then its corresponding location in responds to an outdoor campus scene. Figure 10a shows an image formed on the detector. Figure 10b shows the corre- the captured image is given by (f(θp)cosφp,f(θp)sinφp) sponding SIFT features extracted from the mirror and fish- where f(θp) is the fisheye projection function. Alternatively, if the pixel is in the mirror’s FOV, then its eye panoramas using our automatic calibration algorithm. corresponding location in the captured image can be deter- Figure 10c shows the alignment of the features after the cal- ibration process. The average error in the alignment was mined by solving for the outgoing ray (θ c,φc) from the lens found to be 0.087◦. The computed panorama, which has which would finally proceed in the pixel’s direction (θ p,φp) after being reflected by the mirror. This is a non-linear prob- a resolution of 3600x550 pixels, is shown in two rows (Fig- lem and can be solved as follows. Let R(θ, φ) be the direc- ures 10d and 10e). The second example, shown in Figure 11, tion in which a ray (θ, φ) from the origin O would proceed corresponds to a typical video conference scenario. Here, in after being reflected by the mirror. Then, the following op- 3Since we purchased this camera, other manufacturers have developed timization solves for the required (θc,φc) in a least squares cameras using the same detector that can output 15-30 fps. sense: 4We found the FOV to be actually 194◦ and used this value instead. hsi eas u ytmol eursteueo very a of use the requires only system our because are mirror is cameras. catadioptric the This most in by found those induced than aberrations smaller much the so, Even mirror. like and aberrations astigmatism optical coma, to due is the blurring to This FOV. corresponding mirror’s the those than to compact corresponding more are PSFs FOV lens’ the and that, green Observe (red, wavelengths blue). different three for in angles incidence different shown to of correspond are spots individual PSFs our The in The 6. for at Figure simulator. PSFs focused optical Zemax and the the F/4 obtain using at We (operated quality. camera PSFs, prototype image the the compact more better the the – of the indication image an captured are the camera of quality a of diagram) spot as known Quality: Optical prototype. our evaluate to used sures .PromneEvaluation Performance 7. per- information. quality two captured high the of show convey that also it from we computed views panorama, spective captured the to addition just with system imaging panoramic a hemispherical A (a) 5. Figure h ne hw h opc irratcmn.()Iaecap- Image (c) attachment. mirror a by compact tured the shows inset The fi hy es b u rttp cata- prototype Our (b) lens. sheye nti eto edsrb oeo h efrac mea- performance the of some describe we section this In fi hy es d mg atrdb h prototype. the by captured Image (d) lens. sheye (a) (c) h on pedfntos(Ss–also – (PSFs functions spread point The fi l uvtr nue ytecurved the by induced curvature eld fi hy aoai camera. panoramic sheye 7cm (d) (b) fi fi sheye nity)

10cm n le hntecmr soeae tF/4. at operated is incidence camera the of when three angles blue) for and various diagrams) for the functions below (indicated spread Point 6. Figure irn.Bsdo hspxlsz n h ieo h image the of size the the and size by pixel 2.5 formed size this of on pixel over Based a energy within microns. PSF concentrated average is the angles, vari- F/4, incident for at all angles instance, For incident con- all F/#s. table over ous of averaged a 30% energy, shows ensquaring row) PSF 7 (second least the pixel Figure at the of ensquare [27]. size the can energy taining pixel PSF the detector of to a considered 30% if is good design optically a instance), be for mosaic, (Bayer saic mirror. curvature) row (low used third shallow be the can size, that pixel detector the of on used. resolution Based atleast corresponding ensquare energy. the to PSF shows required the pixel row the of second of 30% the size F-numbers, minimum different the at shows operating When 7. Figure ag vrwihprla sngiil srfre oa the as to camera. referred the is of negligible range working is parallax The which distance. over working range minimum the than scene farther the are if objects only artifacts, without pro- can panoramas camera seamless the duce words, other In mini- distance). (the dependent, working depth mum scene depth certain a is after negligible have parallax becomes and often However, panoramas stitched artifacts. parallax, ghosting parallax. of inherent presence an the has In it viewpoints, compact, albeit panoramic ent, the of parts two its Range: Working and Parallax of column (third pixel size and 7). pixel resolution Figure and in resolution closest optimal the was to it size as pixel microns, a 2.2 with of detector size off-the-shelf an In used images. we prototype, quality our highest the provide would that resolution hnuigaCDo MSdtco ihaclrmo- a with detector CMOS or CCD a using When sal,prla sde is parallax Usually, / . . . . 8.0 964 4.9 1432 5.6 x 964 3.3 1890 x 4.0 1432 1749 2.5 x 1890 2.8 1227 x 1749 2.7 x 1227 pixels) 2.0 (in 3.85 Res. Detector microns) (in Size Pixel F/#

10μm 10μm fi +25 -5 (b) PSFs for Spherical Mirror’s FOV Mirror’s Spherical for (b) PSFs hy es ecndtrietedetector the determine can we lens, sheye 0 0 a SsfrFseeln’FOV Fisheye for lens’ (a) PSFs fi +15 e nnme fpxl and pixels of number in ned -15 fi 0 0 l fve rmtodiffer- two from view of eld ic u ytmcaptures system our Since aeegh rd green (red, wavelengths -25 +5 0 0 o Detector Res. 512x 1024x 2048x 2148x 10.89 +30 4107 6300 (in pixels) 512 1024 2048 2148 Working Range 0.65 −∞ 1.3 −∞ 2.56 −∞ 2.7 −∞ 6291 (in m) 3698 Figure 8. The working range of the prototype camera for different 3.66 -30o 2877 2750 detector resolutions, assuming the maximum acceptable parallax to be 2 pixels. The last column shows the working range of our (a) (b) Figure 9. (a) Vertical resolution map (in pixels per degree). (b) current configuration. Horizontal resolution map (in pixels per scan line). See Section 7 for details. hence varies with the resolution of the detector. For exam- ple, if we set the maximum acceptable error to be 2 pix- sidered before it is used in any given application. First, it els, then Figure 8 shows the working range of our prototype has an inherent parallax that makes it unsuitable for imag- camera with different sensor resolutions. The working range ing objects that are very close to the camera. Second, it has of our prototype is 2.7m-∞ (fourth column in Figure 8). a spatially varying resolution where the upper part of the panorama is captured at a much higher resolution than the Panorama Resolution: The cata-fisheye camera does not lower part. This makes it unsuitable for applications that capture panoramas with uniform resolution. Here, we sepa- require the panoramas to be captured with more of less uni- rately analyze the spatial variations in the vertical and hori- form resolution. zontal resolution. We know that, if the lens and the mirror are coaxial, then Acknowledgment radial lines in the captured images map to vertical lines in This research was conducted at the Computer Vision the panorama. Hence, the vertical resolution (pixels per unit Laboratory at Columbia University. It was supported by a angle) of the panorama at a given θ is defined as dr/dθ, grant from the Office of Naval Research (No. N00014-05- where r is the radial distance of the corresponding pixel 1-0032). from the center of the captured image. Within the fisheye’s References FOV, the vertical resolution depends directly on the the fish- [1] http://www.apple.com/quicktime/technologies/qtvr/. eye projection function. Our prototype camera has a fisheye r kθ [2] http://maps.google.com/. lens with a equidistant projection function (that is = ). [3] http://www.microsoft.com/uc/products/roundtable.mspx. Therefore, the vertical resolution of the fisheye panorama is [4] S. Baker and S. Nayar. A Theory of Single-Viewpoint Cata- uniform as dr/dθ = k. On the other hand, the vertical res- dioptric Image Formation. IJCV, 35(2):175–196, 1999. olution of the panorama in the mirror’s FOV is determined [5] H. Bakstein and T. Pajdla. Panoramic mosaicing with a 180◦ by the mirror’s geometry and position. We find a numer- field of view lens. In Omnidirectional Vision, pages 60–67, ical approximation to the vertical resolution by computing 2002. the change in r for a small change in θ (∆r/∆θ). Fig- [6] A. Basu and S. Licardie. Alternative models for fish-eye ure 9a shows the vertical resolution map for the panorama lenses. Pattern Recognition Letters, 16(4):433–441, 1995. produced by our prototype camera. While the resolution is [7] S. S. Beauchemin, R. Bajcsy, and G. Givaty. A unified pro- highest and uniform in the fisheye lens’ FOV, it is lower in cedure for calibrating intrinsic parameters of fish-eye lenses. the mirror’s FOV owing to its curvature. In Vision Interface, pages 272–279, 1999. [8] S. Bogner. Introduction to panoramic imaging. In IEEE SMC The horizontal resolution is just the total number of pix- Conference, pages 3100–3106, 1995. els in the captured image that make up each scan line in [9] D. Buchele and W. Buchele. US Patent No. 2,638,033: Uni- the panorama. Figure 9b shows the horizontal resolution tary catadioptric objective lens system, 1953. map. Note that, in contrast to the panoramas captured by [10] J. Chahl and M. Srinivassan. Reflective surfaces for other catadioptric cameras, the resolution of the cata-fisheye panoramic imaging. Applied Optics, 36(31):82758285, 1997. panorama is not smooth. However, in applications such as [11] D. Chapman and A. Deacon. Panoramic imaging and virtual video conferencing, the lower part of the panorama is used reality: Filling the gaps between the lines. ISPRS, 53(6):311– to image closer objects (e.g. the table the camera sits on). 319, 1998. In such cases, the resolution characteristics of our camera [12] J. Charles, R. Reeves, and C. Schur. How to build and use an prove to be advantageous. all-sky camera. Astronomy Magazine, 1987. [13] C. Cruz-Neira, D. Sandin, and T. DeFanti. Virtual reality: 8. Discussion The design and implementation of the cave. In SIGGRAPH, While our current prototype uses a spherical mirror with pages 135–142, 1993. a fisheye lens that has an equidistant projection model, con- [14] R. Cutler, Y. Rui, A. Gupta, J. Cadiz, I. Tashev, L. He, A. Col- burn, Z. Zhang, Z. Liu, and S. Silverberg. Distributed meet- vex mirrors with other profiles may be more suitable in the ings: A meeting capture and broadcasting system. ACM Mul- case of lens with different projection functions. We will ex- timedia, 2002. plore the relationship between the shape of the mirror and [15] J. Davis, K. Castle, M. Todd, T. Stuhlinger, and M. Ruda. US the fisheye projection model in our future work. Patent No. 5,841,589: Panoramic optical assembly having an Our approach has two main limitations that must be con- initial flat reflective element, 1998. 95o

θθ

85o -180o φ +180o (a) (b) (c) +250 550 pixels 550

-300

0o (d) Le Half (3600/2 = 1800 pixels) 180o +250 550 pixels 550

-300

180o (e) Right half ( 3600/2 = 1800 pixels) 360o Figure 10. (a) Image captured using our prototype camera operating at F/5.6. (b) SIFT feature detection and matching applied to the two panoramic strips generated for the overlapping regions in the views of the mirror and fisheye lens before calibration. For clarity, only 20% of the features are depicted in the figure. (c) Alignment of the features after the calibration process with an average alignment error of 0.087◦. The position of the mirror’s center of curvature is at (-0.1492, -0.4463, 77.7345) (in mm) relative to the fisheye lens’ viewpoint. The mirror’s orientation is (161.53◦, 0.822◦). (d,e) These images together represent the computed panorama whose resolution is 3600x550 pixels. [16] J. Hong, X. Tan, B. Pinette, R. Weiss, and E. Riseman. [19] D. Lowe. Distinctive image features from scale-invariant Image-based homing. IEEE Control Systems Magazine, keypoints. IJCV, 20:91–110, 2003. 12(1):38–45, 1992. [20] D. McCutchen. US Patent No. 5,703,604: Immersive dodec- [17] U. Iwerks and S. Oaks. US Patent No. 3,118,340: Panoramic aherdral video viewing system, 1997. motion picture camera arrangement, 1964. [21] K. Miyamoto. Fish eye lens. JOSA, pages 1060–1061, 1964. [18] S. Lin and R. Bajcsy. True single view point cone mir- [22] V. Nalwa. A true omnidirectional viewer. Technical report, ror omni-directional catadioptric system. In ICCV, pages II: Bell Laboratories, Holmdel, NJ 07733, USA, 1996. 102–107, 2001. [23] S. Nayar. Catadioptric omnidirectional camera. In CVPR, (a) (b) (c) +250 550 pixels 550

-300 0o (d) Le Half (3600/2 = 1800 pixels) 180o +250 550 pixels 550

-300 180o (e) Right half ( 3600/2 = 1800 pixels) 360o Figure 11. (a) Captured image. (d, e) These images together represent the computed panorama whose resolution is 3600x550 pixels. (b, c) Two perspective views computed from the spherical panorama. pages 482–488, 1997. [29] Y. Yagi and S. Kawato. Panorama scene analysis with conic [24] S. Nayar and V. Peri. Folded Catadioptric Cameras. In projection. In IEEE International Workshop on Intelligent Panoramic Vision, pages 103–119. 2001. Robots and Systems, volume 1, pages 181–187, 1990. [30] K. Yamazawa, Y. Yagi, and M. Yachida. Omnidirectional [25] D. Rees. US Patent No. 3,505,465: Panoramic television imaging with hyperboloidal projection. In International Con- viewing system, 1970. ference on Intelligent Robots and Systems, 1995. [26] D. Slater. Panoramic with fisheye lenses. In In- [31] S. D. Zimmermann. US Patent No. 5,185,667: Omniview ternational Association of Panoramic Photographers, 1996. motionless camera orientation system, 1993. [27] W. Smith. Modern optical engineering : the design of optical systems. McGraw-Hill, 2000. [28] R. Swaminathan and S. Nayar. Polycameras: Camera Clus- ters for Wide Angle Imaging. Technical report, 1999.