The “Vertigo Effect” on Your Smartphone: Dolly Zoom via Single Shot View Synthesis Yangwen Liang Rohit Ranade Shuangquan Wang Dongwoon Bai Jungwon Lee Samsung Semiconductor Inc {liang.yw, rohit.r7, shuangquan.w, dongwoon.bai, jungwon2.lee}@samsung.com Abstract Dolly zoom is a technique where the camera is moved either forwards or backwards from the subject under focus while simultaneously adjusting the field of view in order to maintain the size of the subject in the frame. This results in perspective effect so that the subject in focus appears sta- tionary while the background field of view changes. The (a) Input image I1 (b) Input image I2 effect is frequently used in films and requires skill, practice and equipment. This paper presents a novel technique to model the effect given a single shot capture from a single camera. The proposed synthesis pipeline based on cam- era geometry simulates the effect by producing a sequence of synthesized views. The technique is also extended to allow simultaneous captures from multiple cameras as in- puts and can be easily extended to video sequence captures. Our pipeline consists of efficient image warping along with depth–aware image inpainting making it suitable for smart- phone applications. The proposed method opens up new avenues for view synthesis applications in modern smart- phones. (c) Dolly zoom synthesized image with SDoF by our method Figure 1: Single camera single shot dolly zoom View Syn- 1. Introduction thesis example. Here (a) is generated from (b) through dig- ital zoom The “dolly zoom” effect was first conceived in Alfred Hitchcock’s 1958 film “Vertigo” and since then, has been frequently used by film makers in numerous other films. Previous attempts at automatically simulating this effect The photographic effect is achieved by zooming in or out required the use of a specialized light field camera [19] in order to adjust the field of view (FoV) while simultane- while the process in [20, 1] involved capturing images while ously moving the camera away or towards the subject. This moving the camera, tracking interest points and then apply- leads to a continuous perspective effect with the most di- ing a calculated scaling factor to the images. Generation rectly noticeable feature being that the background appears of views from different camera positions given a sequence to change size relative to the subject [13]. Execution of the of images through view interpolation is mentioned in [8] effect requires skill and equipment, due to the necessity of while [10] mentions methods to generate images given a 3D simultaneous zooming and camera movement. It is espe- scene. Some the earliest methods for synthesizing images cially difficult to execute on mobile phone cameras, because through view interpolation include [6, 23, 31]. More re- of the requirement of fine control of zoom, object tracking cent methods have applied deep convolutional networks for and movement. producing novel views from a single image [30, 17], from multiple input images [28, 9] or for producing new video P frames in existing videos [15]. A In this paper, we model the effect using camera geometry D1 focus plane and propose a novel synthesis pipeline to simulate the ef- B image plane fect given a single shot of single or multi–camera captures, u1 B at location B where single shot is defined as an image and depth capture D0 f1 collected from each camera at a particular time instant and B image plane C1 f A at location A location. The depth map can be obtained from passive sens- t 1 uA ing methods, cf. e.g. [2, 11], active sensing methods, cf. 1 A e.g. [22, 24] and may also be inferred from a single image C1 through convolutional neural networks, cf. e.g. [5]. The synthesis pipeline handles occlusion areas through depth Figure 2: Single camera system setup under dolly zoom aware image inpainting. Traditional methods for image in- painting include [3, 4, 7, 26] while others like [16] adopt the method in [7] for depth based inpainting. Recent methods the scalar DX are the pixel coordinates on the image plane, P like [21, 14] involve applying convolutional networks for the intrinsic parameters, and the depths of for camera X, R this task. However, since these methods have high complex- X ∈ {A, B}, respectively. The 3 × 3 matrix and the T ity, we implement a simpler algorithm suitable for smart- 3 × 1 vector are the relative rotation and translation of phone applications. Our pipeline also includes the appli- camera B with respect to camera A. In general, the relation- u u cation of the shallow depth of field (SDoF) [27] effect for ship between B and A can be obtained in closed–form as image enhancement. An example result of our method with uB DA −1 uA KBT = KBR(KA) + (1) the camera simulated to move towards the object under fo- 1 DB 1 DB cus while changing focal length simultaneously is shown T T R C C in Figure 1. In this example, Figures 1a, 1b are the input where can also be written as = ( A − B). images and Figure 1c is the dolly zoom synthesized image 2.1. Single Camera System with the SDoF effect applied. Notice that the dog remains in focus and the same size while the background shrinks with Consider the system setup for a single camera under an increase in FoV. dolly zoom as shown in Figure 2. Here, camera 1 is CA A The rest of the paper is organized as follows: System at an initial position 1 with a FoV of θ1 and focal A model and view synthesis with camera geometry are de- length f1 (the relationship between the camera FoV θ and scribed in Section 2 while experiment results are given in its focal length f (in pixel units) may be given as f = Section 3. Conclusions are drawn in Section 4. (W/2)/tan(θ/2) where W is the image width). In order H to achieve the dolly zoom effect, we assume that it under- Notation Scheme In this paper, matrix is denoted as , B T P goes translation by a distance t to position C1 along with and (·) denotes transpose. The projection of point , de- B fined as P T in R3, is denoted as point u, a change in its focal length to f1 and correspondingly, a = (X, Y, Z) B B A A defined as u = (x, y)T in R2. Scalars are denoted as X change of FoV to θ1 (θ1 ≥ θ1 ). D1 is the depth of a 3D I point P and D0 is the depth to the focus plane from the or x. Correspondingly, is used to represent an image. A u camera 1 at the initial position C1 . Our goal is to create I(x, y), or alternately I( ), is the intensity of the image B H a synthetic view at location C1 from a capture at location at location (x, y). Similarly, for a matrix , H(x, y) de- A C1 such thatany objectin the focusplane is projectedat the notes the element at pixel (x, y) in that matrix. Jn and 0n denote the identity matrix and zero vectors. same pixel location in 2D image plane regardless of camera n × n n × 1 A location. For the same 3D point P , u1 is its projection onto diag{x1, x2, ··· , xn} denotes a diagonal matrix with ele- the image plane of camera 1 at its initial position CA while ments 1 2 on the main diagonal. 1 x , x , ··· , xn B u1 is its projection onto the image plane of camera 1 af- CB 2. View Synthesis based on Camera Geometry ter it has moved to position 1 . We make the following assumptions: Consider two pin-hole cameras A and B with camera 1. The translation of the camera center is along the prin- centers at locations CA and CB, respectively. From [12], CA − CB T based on the coordinate system of camera A, the projec- cipal axis Z. Accordingly, 1 1 = 0, 0, −t . P R3 while the depth of P to the camera 1 at position CB is tions of any point ∈ onto the camera image planes are 1 T T 1 T T T T B A u K J3 03 P and u D1 = D1 − t. A, 1 = DA A , 1 B, 1 = T 1 K R T P T for cameras A and B, respec- 2. There is no relative rotation during camera translation. DB B , 1 tively. Here, the 2 × 1 vector uX , the 3 × 3 matrix KX , and Therefore, R is an identity matrix J3. 3. Assuming there is no shear factor, the camera intrin- A A sic matrix K1 of the camera at location C1 can be modeled as [12] A f1 0 u0 A A K1 = 0 f1 v0 , (2) DZ 0 0 1 (a) Input image I1 with FoV (b) Synthesized image I1 with A o B o θ1 = 45 FoV θ1 = 50 T where u0 = (u0, v0) is the principal point in terms of pixel dimensions. Assuming the resulting image Figure 3: Single camera image synthesis under dolly zoom. B resolution did not change, the intrinsic matrix K1 Epipoles (green dots), sample point correspondences (red, B A at position C1 is related to that at C1 through a blue, yellow, magenta dots) along with their epipolar lines B zooming factor k and can be obtained as K1 = are shown. A B A K1 diag{k, k, 1}, where k = f1 /f1 = (D0 − t)/D0. A A 4. The depth of P is D1 > t.