Iterative Image Warping

EUROGRAPHICS 2012 / P. Cignoni, T. Ertl Volume 31 (2012), Number 2 (Guest Editors) Iterative Image Warping Huw Bowles1 Kenny Mitchell2 Robert W. Sumner2 Jeremy Moore1 Markus Gross2;3 1Disney Interactive Studios 2Disney Research Zurich 3ETH Zurich Abstract Animated image sequences often exhibit a large amount of inter-frame coherence which standard rendering algorithms and pipelines are ill-equipped to exploit, limiting their efficiency. To address this inefficiency we transfer rendering results across frames using a novel image warping algorithm based on fixed point iteration. We ana- lyze the behavior of the iteration and describe two alternative algorithms designed to suit different performance requirements. Further, to demonstrate the versatility of our approach we apply it to a number of spatio-temporal rendering problems including 30-to-60Hz frame upsampling, stereoscopic 3D conversion, defocus and motion blur. Finally we compare our approach against existing image warping methods and demonstrate a significant performance improvement. Categories and Subject Descriptors (according to ACM CCS): I.3.m [Computer Graphics]: Miscellaneous—Image- based rendering 1. Introduction with minimal performance overhead. Additionally from the mathematical properties of fixed point iteration we derive a In computer graphics significant coherence is exhibited local convergence criterion which informs a robust adaptive across frames of an animation (temporal coherence) and variant of our approach which requires a little more com- across nearby views of a scene (spatial coherence). Current putation time but still a fraction of the time of comparable rendering pipelines recompute each frame from scratch, re- methods. Our approach can be attached to existing render- sulting in a large amount of repeated work in processing ing pipelines with minimal side effects, which we demon- the scene geometry and in performing the light simulation strate by installing our warping kernel into an existing AAA necessary to shade the surfaces. This inefficiency imposes console game to post convert it stereoscopic 3D and to up- severe constraints on the visual fidelity of real-time appli- sample its frame rate from 30 to 60Hz. Finally we compare cations in which rendering budgets are measured in mere our method to existing work and demonstrate a substantial milliseconds. Consequently enormous time and effort are in- performance improvement without loss of quality. vested in optimization of algorithms and assets which in- creases rendering pipeline complexity, complicates content creation workflows and drives up production costs. 2. Background Although previous work includes a number of approaches In this section we review the available options for exploit- to re-use information across frames to exploit this redun- ing frame to frame coherence. Reverse reprojection caching dancy, they are yet to meet the rigorous requirements of (RRC) [NSL∗07,SaLY∗08] rasterizes the scene from the tar- production scenarios and have therefore eluded widespread get viewpoint and computes at each pixel the inverse map- adoption. These methods incur a heavy geometry processing ping from target to source image frame. Although the au- cost or require significant changes to rendering pipelines and thors report reduced fragment shader cost, the scene geom- workflows. etry needs to be reprocessed once or twice per reprojec- In this work we argue for the reuse of shading information tion which is prohibitively expensive for many production across frames through image warping techniques. To trans- scenes. Additionally, shader authoring is complicated by the fer this information efficiently we introduce a novel image need to write variants of the vertex and fragment programs warping algorithm based on fixed point iteration. We de- of each shader instrumented with the required reprojection scribe a heuristics-based variant of our approach that runs code. c 2012 The Author(s) Computer Graphics Forum c 2012 The Eurographics Association and Blackwell Publish- ing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. H. Bowles, K. Mitchell, R. Sumner, J. Moore & M. Gross / Iterative Image Warping For these reasons we focus on methods that are decou- plied to the inversion of computed tomography data by Chen pled from the scenes geometry such as the render cache in- et al. [CLC∗08]. However they address the limited case for troduced by Walter et al. [WDP99, WDG02], which caches which a global inverse warp function exists (i.e. there are no shading points as samples in 3D space and reuses them overlaps/folds in the warp), and is not applicable to the warps across frames. However this system requires substantial considered in this work. We previously relaxed this restric- changes to the graphics pipeline (the presented system re- tion [Bow10] and illustrated the behavior of the iteration in lies on ray tracing to populate the cache) and it is not clear the presence of disocclusions in the context of stereoscopic how this method could be implemented efficiently on raster- 3D post-conversion and temporal upsampling. The temporal ization architectures. upsampling approach from Yang et al. [YTS∗11] also employs an iteration equivalent to fixed point iteration but they Instead of caching data as shading points, image-based do not identify the iteration as such or consider its conver- rendering (IBR) methods use images as the cache primitive, gence properties further. In this work we study the behavior and use image warping techniques to construct the required of the iteration in detail and establish a general framework viewpoint. Some methods render from a large database of for image warping using fixed point iteration. images that sample a global plenoptic function, i.e. the radiance passing through a scene [LH96,GGSC96]. Others store image representations of individual elements in the scene 3. Method ∗ [Max96,SS96,SLS 96,SGHS98]. The Talisman system was We aim to warp a source image Is representing a rendered proposed as a hardware implementation of the latter [TK96]. view of a scene to a target image Iw that resembles the scene Due to their modest computational requirements these ap- from a different viewpoint in space or time. In the next sec- proaches are well suited to solving the rendering efficiency tion we describe the concept of the warp in detail and then problem. However, these approaches usually assume that the introduce an efficient solution method which employs itera- incoming radiance from surfaces in the scene is static which tive methods to solve the warp efficiently. may be too restrictive for many scenarios. This issue is mitigated in post-rendering warping which 3.1. Image Warp Definition 2 2 is the application of image warping to the reprojection of We define the warp as a vector field V : R ! R that de- rendered images to nearby viewpoints in space and time. scribes how each point in the source image should be trans- While post-rendering warps also assume that the radiance lated in order to produce Iw. For a particular point xs in the function is static, for small reprojection offsets the result- source image, the warped image coordinates xw are given by ing error tends to be small. Existing image warping methods can be categorized by their data access patterns. For- x = x +V(x ); (1) ward warps scatter information for the source frame into the w s s 2 target frame, which can be implemented on graphics hard- with xs;xw 2 R . For a particular pixel at point xw in the ware in the following ways. Each pixel can be splatted as warped image Iw, we wish to find the location(s) xs in the individual primitives [Wes90,ZPvBG02] which is conceptu- source image Is that satisfy Equation1. When multiple solu- ally simple but efficiently and accurately filtering irregularly tions xs exist we take the one with the minimum depth z(xs) scattered samples is challenging on GPU hardware. Another to maintain depth order. We now introduce the specific warp approach is to connect the pixels with triangles and raster- fields studied in this work. ize them into the target view [MMB97], which prevents is- Temporal Reprojection The warp field for temporal re- sues like holes appearing in the warped image. However both projection is given by per-pixel motion vectors, a common approaches result in a heavy geometry processing workload by-product of modern rendering pipelines which indicate the (over 1M primitives per warp at standard resolutions). While screen space motion of the rendered surfaces [Ros07]. These this dense warp geometry can be reduced by using a coarse motion vectors can be computed by projecting the position regular grid [DER∗10] or by adaptively grouping pixels into of the scene geometry at two points in time and computing coherent blocks [CW93, DRE∗10], such methods are an ap- the derivative using finite differences. The warp field is then proximation of the true warp. defined as Inverse warps reverse the data access pattern, i.e. they Vtemporal(xs;t) = tM(xs) (2) gather information from the source frame to fill the target frame. Since a closed-form expression for the inverse where M is the motion vector field, and t is the difference in warp is often not available, special heuristics for the warp time to reproject the viewpoint. From second order finite dif- type are used to obtain an approximate solution [KTI∗01, ferences we can obtain acceleration vectors A and can per- And10, SKS11], or a manual search is conducted through form nonlinear temporal reprojection using the Taylor ex- the source image [McM97, Mar98]. A more efficient way to pansion perform this search is to employ an iterative method called t2 V (xs;t) = tM(xs) + A(xs): (3) fixed point iteration. Such an approach has recently been ap- nonlinear 2 c 2012 The Author(s) c 2012 The Eurographics Association and Blackwell Publishing Ltd.

Load more