Multiperspective Images From Video Jiwon Kim Kiera Henning Steve Seitz David Salesin University of Washington Abstract The composed image can further be edited by adjusting the slice and/or the mapping. The user can change the slice itself by chang- We present a prototype of an interactive authoring tool for render- ing the selected columns, and the mapping can be changed by a ing images from a video input. The input video is considered as a stretch-and-compress operation. To stretch or compress a certain space-time cube of pixel data, and the user interacts with the tool to region of the output image, the user first specifies a region of influ- create the desired image by cutting a smoothly varying slice through ence as a range of x coordinate. The operation will affect only the the cube. Such images exhibit interesting temporal and spatial dis- specified region, and the rest of the image will stay fixed. Then the tortions which cannot be seen in normal perspective images, be- user click-and-drags on a column within that region to stretch and cause it samples pixels over varying time and space. This can be compress either side of the column. useful for non-perspective visualization of a scene, or for creating The rest of the paper is organized as follows. After we discuss unusual artistic effects. The tool may also be extended to produce related work in section 2, we describe the implementation details in an animated output by interpolating between a set of rendered im- section 3. Then we demonstrate some example results we produced ages. using the tool. We conclude the paper with a discussion of possible improvements to be made and future directions. 1 Introduction A video sequence is a collection of pixel rays emanating from a time-varying set of points in space captured over a certain lapse of time. Therefore, generating an image by sampling pixels from a 2 Related work video input results in an image composed of pixels from different points in time and space. Let us think of a video sequence as a space-time cube of pixel data as shown in Figure 1. If we cut a [2] provided a major inspiration for our work. It suggests the utility smooth surface through the cube and map it onto a 2D image, it of multiperspective images for storytelling purposes. The desired will result in an image with strange yet continuous temporal and images are generated from a video input filmed by a slit-camera spatial distortions which cannot be observed in normal perspective moving along a carefully planned camera path. They also demon- images, as shown in Figure 2. strated a preliminary implementation which allows the user to in- Such images, if carefully generated to obtain the desired effect, teractively create such images from a 3D model. can be useful for various purposes. For example, a video sequence The issue of rendering a multiperspective image from a 3D that records a static scene with a moving camera can be used to model has been studied by a number of researchers in the past. produce an image that contains multiple viewpoints of different ob- [7] addressed the problem of creating a seamless multiperspective jects in the scene. Such a non-perspective image can be an effective panoramic image corresponding to a user-defined camera path. [6] visualization of the scene in a way that is impossible with a perspec- also produced similar images from a 3D model, called multiple- tive image. This was a common trick used in old paintings such as center-of-projection images (MCOP), but their center of interest egyptian murals or paintings from pre-renaissance period. [2] also was using such images for an image-based 3D reconstruction, suggested that images with multiple viewpoints can be used as a rather than visualization. While the above two used the slit-camera new form of storytelling. Moreover, these images may be useful method to produce images, [1] took an object-based approach for purely artistic purposes. In fact, the underlying philosophy of where a multiperspective image is interactively generated by spec- multiperspective images is quite similar to that of cubism. Other ifying different viewing angles for different objects in the scene. types of input movies, such as a dynamic scene filmed by a static This work is similar in spirit to ours in that the aim was to pro- camera, may provide even more interesting effects. vide an interactive multiperspective rendering tool. However, all of Our aim is to design an interactive authoring tool for rendering the above assumed the existence of a 3D model and the freedom to such images from a video input. To create an image, the user first choose a desired camera path or viewing angles. selects groups of pixels from the input frames that s/he wants to have in the final image and drags them over to the desired locations The concept of considering video as a space-time cube of data of the final image. Then the program automatically calculates a was first introduced by Video Cube ([5]), an interactive tool for tak- smooth slice through the video cube which interpolates these pixels, ing arbitrary planar slices through the video cube. The distinction and also a continuous mapping that maps pixels on the slice to their of our work from this is that our tool is not limited to planar slices, desired positions in the final image. Currently, the tool only sup- but allows arbitrary slices that are vertical to the x-t plane. Also, ports 2D slices across the cube that are made of vertical columns of Video Cube is more of a viewing tool that does not provide a lot of input frames, as illustrated in Figure 2. Therefore, the user can only user controls, whereas our tool is designed for interactive authoring. select a single column or a set of multiple adjacent columns from This concept was utilized by [4] to produce a non-photorealistic input frames. If many consecutive columns are selected from the video with different artistic styles from a video input. In particular, same input frame, the slice will look nearly perspective around the multiperspective video outputs such as those resembling the style region where these columns are interpolated. In general, the num- of cubism or photo collage can be generated. However, it focuses ber of consecutive columns is proportional to the local perspectivity more on simulating different artistic styles rather than creating a of the corresponding input frame in the output image. particular multiperspective image. an input frame y t x Figure 1: A video can be considered as a space-time cube of data, with x-y-t axes. y t x output image Figure 2: A slice cut through the video cube generates a 2D image. 3 Implementation 4 Results 3.1 Slice and mapping computation Using the tool, we produced a variety of images and animations. The types of input video sequences we used varied most signifi- Since the slice is limited to be vertical, i.e., perpendicular to the cantly in the types of motion exhibited by the camera. Below are x-t plane, we can conveniently consider it as a 2D planar curve on selections from our resulting images. the x-t plane instead. When the user specifies the columns to be The pictures in this second group have some clear ties to sev- interpolated, they appear as points or straight lines on the x-t plane. eral art movements from the past century. At the forefront of these We use an interpolating 2D cubic spline to interpolate these points are Surrealism and Cubism. Artists from both of these movements or line segments. Each of the points is considered a control point to often use multiple perspectives to describe some subject matter. be interpolated, as well as the middle point of each line segment. Essentially, both movements sought to more accurately depict the The slope and length of the line segment is used to determine the essence of an object. Cubists usually aimed to accomplish this direction and magnitude of the tangent at its corresponding control specifically by incorporating multiple perspectives, but tended to do point. Since the columns of the same line segment comes from the so in a discontinuous fashion. Cubist pieces tended to look like an same input frame, the line segment is always vertical, and therefore unwrapping of an object geometrically, and because of this, our au- there are two possible directions. The tool allows the user to choose thoring tool is not quite ideal for creating art in a Cubist style. With- one of the directions. The length of the line segment is proportion- out the ability to create cusps in the curve, the transition between ally related to the magnitude of the tangent. The longer the line perspectives is generally too smooth to accomplish this unwrapping segment, the bigger the magnitude of the tangent, thus making the in a stylistically similar way. Surrealists, alternately, did not focus curve more tightly approximate the perspective view of the corre- specifically on unwrapping perspective. Instead, they more ambigu- sponding columns. We use a rather ad-hoc and empirically defined ously defined their goal as trying to show the essence of an object formula for the correlation. As discussed in section 5, its perfor- by showing more aspects than just the visually apparent. This of- mance is not satisfactory and we hope to find a better formula. For ten led to perspective shifts or perspective-related inconsistencies in single points, the tangent is automatically determined by constrain- Surrealist works.