Motion Parallax in Stereo 3D: Model and Applications
Total Page:16
File Type:pdf, Size:1020Kb
Motion Parallax in Stereo 3D: Model and Applications Petr Kellnhofer1 Piotr Didyk1;2 Tobias Ritschel1;2;3 Belen Masia1;4 Karol Myszkowski1 Hans-Peter Seidel1 1MPI Informatik 2Saarland University, MMCI 3University College London 4Universidad de Zaragoza Scene motion Near Far Near Far Original S3D Original disparity Parallax Map Our disparity Our S3D Figure 1: Starting from stereoscopic video content with a static observer in a moving train (Left), our method detects regions where motion parallax acts as an additional depth cue (Center, white color) and uses our model to redistribute the disparity depth budget from such regions (the countryside) to regions where it is more needed (the train interior) (Right). Abstract 1 Introduction Binocular disparity is the main depth cue that makes stereoscopic Perceiving the layout of a scene is one of the main tasks performed images appear 3D. However, in many scenarios, the range of depth by the human visual system. To this end, different depth cues [Cut- that can be reproduced by this cue is greatly limited and typically ting and Vishton 1995] are analyzed and combined into a common fixed due to constraints imposed by displays. For example, due to understanding of the scene. Not all cues, however, provide reliable the low angular resolution of current automultiscopic screens, they information. Pictorial cues, such as shadows, aerial perspective or can only reproduce a shallow depth range. In this work, we study defocus, can often be misleading. Other cues, such as occlusion, the motion parallax cue, which is a relatively strong depth cue, and provide only a depth ordering. There are also very strong cues, such can be freely reproduced even on a 2D screen without any limits. as ocular convergence and binocular disparity, which provide a true We exploit the fact that in many practical scenarios, motion parallax stereoscopic impression. These, however, can only be reproduced in provides sufficiently strong depth information that the presence of a limited fashion due to significant limitations of the current display binocular depth cues can be reduced through aggressive disparity technology. The first problem is the lack of a correct accommoda- compression. To assess the strength of the effect we conduct psycho- tion cue in most of the current stereoscopic displays, which leads to visual experiments that measure the influence of motion parallax on visual discomfort [Hoffman et al. 2008; Lambooij et al. 2009] when depth perception and relate it to the depth resulting from binocular a large disparity range is shown to observers. The limited range disparity. Based on the measurements, we propose a joint disparity- of displayable disparity that does not cause such issues defines the parallax computational model that predicts apparent depth resulting depth budget of the display suitable for practical applications. While from both cues. We demonstrate how this model can be applied in current research tries to address this problem with new light-field the context of stereo and multiscopic image processing, and propose displays [Masia et al. 2013b], these solutions have even stronger new disparity manipulation techniques, which first quantify depth requirements regarding the depth range. This is due to the apparent obtained from motion parallax, and then adjust binocular disparity blur for objects that are located at a significant distance from the information accordingly. This allows us to manipulate the disparity screen plane [Zwicker et al. 2006; Wetzstein et al. 2012]. Because of signal according to the strength of motion parallax to improve the the aforementioned reasons, it is crucial to take any opportunity that overall depth reproduction. This technique is validated in additional allows us to improve depth reproduction without using additional experiments. disparity range. One of the strongest monocular depth cues is motion parallax. It Keywords: stereoscopic 3D, gaze tracking, gaze-contingent dis- arises when the location of features at different depths results in play, eye tracking, retargeting, remapping different retinal velocities. The strength of this cue is relatively high Concepts: •Computing methodologies ! Perception; Image when compared to other monocular cues and also when compared processing; to binocular disparity [Cutting and Vishton 1995]. This fact has been exploited in several applications, such as wiggle stereoscopy Permission to make digital or hard copies of all or part of this work for [Wikipedia 2015b] where motion parallax is used as a metaphor for personal or classroom use is granted without fee provided that copies are not stereoscopic images, or parallax scrolling [Wikipedia 2015a] used made or distributed for profit or commercial advantage and that copies bear in games where, by moving foreground and background at different this notice and the full citation on the first page. Copyrights for components speeds, a depth sensation is evoked. Striking examples of motion of this work owned by others than the author(s) must be honored. Abstracting parallax efficiency are species that introduce subtle head movements with credit is permitted. To copy otherwise, or republish, to post on servers to enable motion parallax [Kral 2003]. This mechanism has been or to redistribute to lists, requires prior specific permission and/or a fee. incorporated into cameras where apparent depth is enhanced by Request permissions from [email protected]. c 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN: 978-1-4503-4514-9/16/12 SA ’16 Technical Papers, December 05 - 08, 2016, Macao DOI: http://dx.doi.org/10.1145/2980179.2980230 dθ ∆f Parallax = dα ≈ f Disparity = (A) (B) |Ddθd ∆−Df d | Parallax =2= θ dα ≈ f dθ ∆f 1 Parallax = DiPasparrallaitxy = dDi(Aspar) itdy(B) dα ≈ f ≈ |Ddα −D | =2θ Disparity = dθ (A∆) f (B) dθ ∆f Parallax = |Dd −Dd | Parallax = 1 dα ≈ f Parallaxdα ≈ fDisparity =2θ α ≈ dα Disparity = 1d(A) d(B) Disparitdyθ= θ ∆d(fA) Pad(Bra)llax |D Dispar−Dity | Parallax = |D −D | =2≈ dθα (1) dα=2d≈αθ f α 1 Disparity = (dAθ1) (B)Parallax Disparity Parallax d θ Dispard ity α≈ dα |D≈ dα−D | (1) =2θ ∆dαf θ (1) 1 f Parallax Didθsparity dαα ≈ dαα d(∆Af) dθ ∆f D dθθ (1) θ Parallax = (1) d(Bf) dα ≈dα f dDα ∆f dθ ∆f α Disparity = (A) Pa(Bra)llax = subtle motion of the sensor [Proffitt and Banton 1999; v3 c Imaging d(AA) dθ ∆d fdθ d dDθParallax = |D f −D | dαθ ≈ ∆ff θ * =2*θ Parallax (1=) 2015]. Interestingly, motion parallax is not limited to observer d(BB) dα ≈ f∆f dα ≈ f ∆f dθ ∆f d(A) Disparity = d(A) d(B) motion, but also provides depth information whenever local motion dα D D1 |D −D | PaDirallasparAx =ity = d(A) f d(BDi) sparity = d(A) d(B) f Padθradllaα∆|Dx≈f f d−DDi(Bspar) it|y =2|Dθ −D | in the scene follows a predictable transformation [Ullman 1983; Padθrallax = =2θ≈ dDα B dα ≈ f d(AA) =21θ Luca et al. 2007] (Fig. 2). These facts suggest that motion parallax ∆fDid(Aspar) ity = d(A) D d(B) D * |D 1 −D* | Parallax Disparity is a very strong source of depth information for the human visual DisparitPay =ralladx(A) Did(sparBd()BBit) y ≈ d1α f d(B) |D=2θ α−DD | Parallax Disparity D ≈ dα ≈ dα system (HVS), but it has never been explored in the context of =2θ 1 A d(A) PaArallax Diθ sparity stereoscopic image manipulations. D 1 B α (1) B ≈ dα d(PaB)rallax αDidsparα ity α In this work, we address this opportunity and propose a compu- D ≈ dα θ (1) A θ dθ tational model for detecting motion parallax and quantifying the α dαθ (1) (1) B dα ∆f amount of apparent depth it induces together with binocular dispar- α θ ddαθ ity. To this end, we conduct a series of psychovisual experiments dθ f (1) θ dα ∆dfθ (1) that measure apparent depth in stereoscopic stimuli in the presence (A) dα ∆fd ∆f of motion parallax. This is done for simple, sinusoidal corrugation dθ D fd(B) dθ ∆f D d(Af) stimuli. Based on the measurements, we propose a computational D ∆f d(A) A (A) model that predicts apparent depth induced by these cues in complex Df Dd (B) Figure 2: Motion(B) parallaxB mechanicsD [Nawrot and Stroyan 2009]. images. Furthermore, we demonstrate how the model can be used f d d(BA) d(DA) D TranslationD of aA pair of points A and B to the right (or equivalently to improve depth reproduction on current display devices. To this d(A) AB observerD translationd(B) to the left) produces motion parallax. Assuming end, we develop a new, motion-aware disparity manipulation tech- (DB) B B nique. The key idea is to reallocate the disparity range from regions thatDAd is theA fixation point, its retinal position does not change, due that exhibit motion parallax to static parts of the scene so that the to the pursuitA B eye rotation α at angular velocity dα=dt, while at overall depth perceived by observers is maximized. To evaluate the the sameB time the retinal position of B changes with respect to A effectiveness of our manipulations, we perform additional validation by the angular distance θ producing retinal motion with velocity experiments which confirm that by taking the motion parallax depth dθ=dt. The relation dθ=dα ≈ ∆f=f = m approximately holds, cue into account, the overall depth impression can be enhanced with- which, given the fixation distance f, enables recovery of depth ∆f out extending the disparity budget (Fig. 1). More precisely, we make from motion parallax. The right side of the figure shows the relation the following contributions: of motion parallax to the angular measure of binocular disparity d = jD (A) − D (B)j [Howard and Rogers 2012, Fig.