Squaring the Circle in Panoramas

Squaring the Circle in Panoramas Lihi Zelnik-Manor1 Gabriele Peters2 Pietro Perona1 1. Department of Electrical Engineering 2. Informatik VII (Graphische Systeme) California Institute of Technology Universitat Dortmund Pasadena, CA 91125, USA Dortmund, Germany http://www.vision.caltech.edu/lihi/SquarePanorama.html Abstract and conveying the vivid visual impression of large panoramas. Such mosaics are superior to panoramic pictures taken Pictures taken by a rotating camera cover the viewing with conventional fish-eye lenses in many respects: they sphere surrounding the center of rotation. Having a set of may span wider fields of view, they have unlimited reso- images registered and blended on the sphere what is left to lution, they make use of cheaper optics and they are not be done, in order to obtain a flat panorama, is projecting restricted to the projection geometry imposed by the lens. the spherical image onto a picture plane. This step is unfor- The geometry of single view point panoramas has long tunately not obvious – the surface of the sphere may not be been well understood [12, 21]. This has been used for mo- flattened onto a page without some form of distortion. The saicing of video sequences (e.g., [13, 20]) as well as for ob- objective of this paper is discussing the difficulties and op- taining super-resolution images (e.g., [6, 23]). By contrast portunities that are connected to the projection from view- when the point of view changes the mosaic is ‘impossible’ ing sphere to image plane. We first explore a number of al- unless the structure of the scene is very special. When all C ternatives to the commonly used linear perspective projec- pictures share the same center of projection , we can con- tion. These are ‘global’ projections and do not depend on sider the viewing sphere, i.e., the unit sphere centered in C image content. We then show that multiple projections may , and identify each pixel in each picture with the ray con- C coexist successfully in the same mosaic: these projections necting with that pixel and passing through the surface are chosen locally and depend on what is present in the pic- of the viewing sphere, as well as through the physical point tures. We show that such multi-view projections can pro- in the scene that is imaged by that pixel. By detecting and duce more compelling results than the global projections. matching visual features in different images we may regis- ter automatically the images with respect to each other. We may then map every pixel of every image we collected to 1. Introduction the corresponding point of the viewing sphere and obtain a spherical image that summarizes all our information on As we explore a scene we turn our eyes and head and the scene. This spherical image is the most natural repre- capture images in a wide field of view. Recent advances in sentation: we may represent this way a scene of arbitrary storage, computation and display technology have made it angular width and if we place our head in C, the center of possible to develop ‘virtual reality’ environments where the the sphere, we may rotate it around and capture the same user feels ‘immersed’ in a virtual scene and can explore it by images as if we were in the scene. moving within it. However, the humble still picture, painted What is left to be done, in order to obtain our panorama- or printed on a flat surface, is still a popular medium: it on-a-page, is projecting the spherical image onto a picture is inexpensive to reproduce, easy and convenient to carry, plane. This step is unfortunately not obvious – the surface store and display. Even more importantly, it has unrivaled of the sphere may not be flattened onto a page without some size, resolution and contrast. Furthermore, the advent of in- form of distortion. The choice of projection from the sphere expensive digital cameras, their seamless integration with to the plane has been dealt with extensively by painters and computers, and recent progress in detecting and matching cartographers. An excellent review is provided in [9]. informative image features [4] together with the develop- The best known projection is linear perspective (also ment of good blending techniques [7, 5] have made it possi- called ‘gnomonic’ and ‘rectilinear’). It may be obtained by ble for any amateur photographer to produce automatically projecting the relevant points of the viewing sphere onto a mosaics of photographs covering very wide fields of view tangent plane, by means of rays emanating from the cen- 1 ter of the sphere C. Linear perspective became popular amongst painters during the Renaissance. Brunelleschi is credited with being the first to use correct linear perspective. Alberti wrote the first textbook on linear perspective describing the main construction methods [1]. It is believed by many to be the only ‘correct’ projection because it maps lines in 3D space to lines on the 2D image plane and because when the picture is viewed from one special point, the ‘center of projection’ of the picture, the retinal image that is Figure 1. Perspective distortions. Left: Five pho- obtained is the same as when observing the original scene. tographs of the same person taken by a rotating cam- A further, somewhat unexpected, virtue is that perspective era, after rectification (removing spherical lens dis- pictures look ‘correct’ even if the viewer moves away from tortion). Right: An overlay of the five photographs af- the center of projection, a very useful phenomenon called ter blackening everything but the person’s face shows ‘robustness of perspective’ [18, 22]. that spherical objects look distorted under perspec- Unfortunately, linear perspective has a number of draw- tive projection even at mild viewing angles. For ex- backs. First of all: it may only represent scenes that are ample, here, the centers of the faces in the corners ◦ at most 180 wide: as the field of view becomes wider, are at ∼ 20◦ horizontal eccentricity. the area of the tangent plane dedicated to representing one degree of visual angle in the peripheral portion of the picture becomes very large compared to the center, and even- 2 Global Projections tually becomes unbounded. Second, there is an even more stringent limit to the size of the visual field that may be What are the alternatives to linear perspective? represented successfully using linear perspective: beyond ◦ ◦ An important drawback of linear perspective is the ex- widths of 30 -40 architectural structures (parallelepipeds cessive scaling of sizes at high eccentricities. Consider and cylinders) appear to be distorted, despite the fact that a painter taking measurements in the scene by using her their edges are straight [18, 14]. Furthermore, spheres that thumb and using these measurements to scale objects on are not in the center of the viewing field project to el- the canvas. She takes angular measurements in the scene lipses onto the image plane and appear unnatural and dis- and translates them into linear measurements onto the can- torted [18] (see Fig 1). Renaissance painters knew of these vas. This construction is called Postel projection [9]. It shortcomings and adopted a number of corrective mea- avoids the ‘explosion’ of sizes in the periphery of the pic- sures [14], some of which we will discuss later. ture. Along lines radiating from the point where the picture The objective of this paper is discussing the difficulties plane touches the viewing sphere, it actually maps lengths and opportunities that are connected to the projection from on the sphere to equal lengths in the image. Lines that run viewing sphere to image plane, in the context of digital im- orthogonal to those (i.e., concentric circles around the tan- age mosaics. We first argue that linear perspective is not the gent point) will be magnified at higher eccentricities, but only valid projections, and explore a number of alternatives much less than by linear perspective. The Postel projection which were developed by artists and cartographers. These is close to the cartographic stereographic projection. The are ‘global’ projections and do not depend on image con- stereographic projection is obtained by using the pole oppo- tent. We explore experimentally the tradeoffs of these pro- site to the point of tangency as the center of projection. jections: how they distort architecture and people and how Consider now the situation in which we wish to repre- well do they tolerate wide fields of view. We then show that sent a very wide field of view. A viewer contemplating a multiple projections may coexist successfully in the same wide panorama will rotate his head around a vertical axis in mosaic: these projections are chosen locally and depend on order to take in the full view. Suppose now that the view what is seen in the pictures that form the mosaic. We show has been transformed into a flat picture hanging on a wall experimentally that the such a content based approach often and consider a viewer exploring that picture: the viewer will produces more compelling results than the traditional ap- walk in front of the picture with a translatory motion that is proach which utilizes geometry only. We conclude with a parallel to the wall. If we replace rotation around a vertical discussion of the work that lies ahead. axis with sideways translation in front of the picture we ob- In this paper we do not address issues of image reg- tain a family of projections which are popular with cartog- istration and image blending. In all our experiments the raphers. Wrap a sheet of paper around the viewing sphere computation of the transformations of the input images to forming a cylinder that touches the sphere at the equator.

Load more