Real-Time Deep Image Rendering and Order Independent Transparency A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy Pyarelal Knowles August, 2015 School of Computer Science and Information Technology College of Science, Engineering and Health RMIT University, Australia i Candidate Declaration I certify that except where due acknowledgement has been made, the work is that of the author alone; the work has not been submitted previously, in whole or in part, to qualify for any other academic award; the content of the thesis/project is the result of work which has been carried out since the official commencement date of the approved research program; any editorial work, paid or unpaid, carried out by a third party is acknowledged; and ethics procedures and guidelines have been followed. Pyarelal Knowles Date ii Acknowledgements A big thank you to HP and Nvidia, Paul Phillips and Mike Wang, for the talk presenting trip money and giving, for some time, the latest computer parts needed to time my work. This was quite important to the work in finding out answers to some problems about close memory making things slow and showing how the work gets better with new parts. This work would not have been possible without the direction, help and handling from the people who see over my work, Geoff Leach and Dr Fabio Zambetta. A lot of time passed and I have learned much and enjoyed a lot. I seriously thank them for the change in my life that is now part of who I am and the heavy push for that hard to find order of ten times that was just not really possible but then finally was. At many times in this work there have been some hard number problems. Dr Musa Mam- madov has often given his time, which there is not a lot of, to talk through these problems with me. I thank him very much for his kind help which has been of much use. Thank you to the people who mark this work. I know it is hard and you do not have much time. Your work is not taken without care. My family and friends have been amazing whether helping with reading parts of the work or other problems in life. I would especially like to thank my parents. I have none better, could not have asked for better and am glad to be where I am because of them. I would also especially like to thank my school friends, Jesse Archer, Michael Patton and Dr Tsz Ho Wong (without order), who keep my mind from leaving reason too much, who I interrupt again and again with questions and who then give much wanted help. Lois Dent and Dave Hogan played a big part in my learning and the road to this work for which I have much thanks. Finally, thanks to the up-goer five text editor which allowed me to write these thanks, even though it only allows the ten hundred most used words. iii Contents Contents iv 1 Introduction 2 2 Background 6 2.1 GPU Rasterization Rendering ............................. 6 2.2 Transparency and OIT ................................. 14 2.3 Deep Images ...................................... 18 2.4 GPU Programming ................................... 19 3 The Layered Fragment Buffer 26 3.1 Introduction ....................................... 26 3.2 Related Work ...................................... 27 3.3 The Linked List LFB .................................. 28 3.4 The Linearized LFB .................................. 29 3.5 Performance Results .................................. 32 3.6 Conclusion ....................................... 35 4 Sorting with Backwards Memory Allocation 36 4.1 Introduction ....................................... 36 4.2 Background ....................................... 37 4.3 Backwards Memory Allocation ............................ 37 4.4 Results .......................................... 40 4.5 Conclusion ....................................... 44 5 Sorting with Register-Based Block Sort 45 5.1 Introduction ....................................... 45 5.2 Background ....................................... 46 5.3 Register-Based Block Sort ............................... 50 5.4 Results .......................................... 53 5.5 Conclusion ....................................... 57 6 Sorting with Spatial Coherence 58 6.1 Introduction ....................................... 58 6.2 Coherence in Deep Images ............................... 58 6.3 Sorting Coherent Fragment Lists ........................... 59 6.4 The Coalesced Linearized LFB ............................ 61 6.5 Results .......................................... 63 6.6 Conclusion ....................................... 66 iv 7 Deep Image Sampling 67 7.1 Introduction ....................................... 67 7.2 Background ....................................... 68 7.3 Related Work ...................................... 73 7.4 Connectivity in Deep Images ............................. 76 7.5 Deep Image Interpolation and Raycasting ...................... 80 7.6 Deep Image Mipmapping ............................... 82 7.7 Results .......................................... 84 7.8 Discussion and Future Work .............................. 88 7.9 Conclusion ....................................... 90 8 Depth of Field 91 8.1 Introduction ....................................... 91 8.2 Background ....................................... 93 8.3 Circle of Confusion ................................... 95 8.4 Related Work ...................................... 96 8.5 Hierarchical Fragment Reduction ........................... 98 8.6 Results ..........................................103 8.7 Future Work ......................................106 8.8 Conclusion .......................................107 9 Conclusion and Future Work 108 Bibliography 110 v Abstract In computer graphics some operations can be performed in either object space or image space. Image space computation can be advantageous, especially with the high parallelism of GPUs, improving speed, accuracy and ease of implementation. For many image space techniques the information contained in regular 2D images is limiting. Recent graphics hard- ware features, namely atomic operations and dynamic memory location writes, now make it possible to capture and store all per-pixel fragment data from the rasterizer in a single pass in what we call a deep image. A deep image provides a state where all fragments are available and gives a more complete image based geometry representation, providing new possibilities in image based rendering techniques. This thesis investigates deep images and their growing use in real-time image space applications. A focus is new techniques for im- proving fundamental operation performance, including construction, storage, fast fragment sorting and sampling. A core and driving application is order-independent transparency (OIT). A number of deep image sorting improvements are presented, through which an order of magnitude performance increase is achieved, significantly advancing the ability to perform transparency rendering in real time. In the broader context of image based rendering we look at deep images as a discretized 3D geometry representation and discuss sampling techniques for raycasting and antialiasing with an implicit fragment connectivity approach. Using these ideas a more computationally complex application is investigated — image based depth of field (DoF). Deep images are used to provide partial occlusion, and in particular a form of deep image mipmapping allows a fast approximate defocus blur of up to full screen size. 1 Chapter 1 Introduction Figure 1.1: The Sponza atrium model rendered using order-independent transparency. This thesis explores deep images and their use in real-time image space rendering. They are elevated as a unifying concept for a recent growing body of work and we focus on improving core operations with regard to some primary applications such as order independent transparency (OIT), shown in figure 1.1. A deep image contains multiple values per pixel, which for OIT allows geometry to be sorted in image space for correct transparency. We now elaborate, motivating the work by showing the importance of deep images, outlining some existing problems and areas with opportunity for improvement, and provide context for discussing the contributions and thesis structure. Object space and image space are categories of techniques which operate in different stages in the rendering pipeline and on different data: object space on the original geometry and image space on the pixel values in the rendered image, as discussed in section 2.1.1. There are many cases where performing operations in image space improves speed, accuracy and ease of implementation. A large part of this is due to GPU hardware being well suited, and even designed, to accelerate image based operations, as discussed in section 2.1.6. A primary example is the z-buffer [Cat74], which solves the hidden surface problem per-pixel. Deferred shading [DWS+88; ST90] only performs expensive shading for visible surfaces. Even geometry can be modified in image space with per-pixel displacement mapping. Most image space techniques use regular 2D images — a grid with just a single value per cell, or pixel. These images are “flat”, computed by discarding and compositing geometry, and can be limiting for many image space techniques. Therefore, we investigate keeping all frag- 2 ments during rasterization, shown in figure 1.2, forming a deep image, visualized in figure 1.3, which is necessary for new, faster, more accurate and robust techniques. Examples include order-independent transparency (OIT), translucency and absorption