Master of Science in Engineering: Game and Software Engineering July 2020 Analysing Variable Rate Shading’s Image-Based Shading in Deferred Lighting Composition A comparison between image-based shading and uniform shading for games Filip Lundbeck Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Engineering: Game and Software Engineering. The thesis is equivalent to 20 weeks of full time studies. The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree. Contact Information: Author(s): Filip Lundbeck E-mail: fi[email protected] University advisor: Dr. Prashant Goswami Department of Computer Science Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 Abstract Background. The shading cost of a pixel is only getting more expensive with more realistic games. Resolution of games is equally pushed to display the all the details in a scene. This causes rendering a frame to be very expensive. Dynamic Resolution Rendering has been used to uniformly decreases resolution to gain performance but with the new release of image-based shading through Variable Rate Shading could be the new way to gain performance with less impact on image quality. Objectives. The goal is to see if the adaptive shading possibilities of Variable Rate Shading can show equal or better results, in regards to performance and image qual- ity, compared to the uniform shading of Dynamic Resolution Rendering. Methods. This investigation is performed by implementing them into the Deferred Lighting pass in a Deferred Renderer. The performance is measured by the render pass time of the Deferred Lighting and the image quality is measured by comparing the final frames of Variable Rate Shading and Dynamic Resolution Rendering against the original resolution through SSIM. Results. Overall Variable Rate Shading show comparable performance results to Dynamic Resolution Rendering but the image quality is closer to the original reso- lution. Conclusions. Using image-based shading on the deferred lighting pass allow the possibility of extracting similar performance gains as dynamic resolution rendering but allows maintaining higher image quality. Keywords: Rendering, Adaptive Shading, Uniform Shading, Variable Rate Shading, Dynamic Resolution Rendering i Sammanfattning Bakgrund. Kostnaden att ljussätta en pixel blir dyrare med realistiska spel. Up- plösningen av spel är ökas i samma tak för att visa upp detaljerna. Att rendera en blidruta för användare blir därför allt dyrare. Dynamic Resolution Rendering är ett sätt att enhetligt minska upplösningen för att öka prestandan fast med den nyligen släppta variable rate shading finns det nya sätt att öka prestandan men med mindre påverkan på bildkvalitén. Syfte. Målet är att se om adaptiv ljussätningsning via Variable Rate Shading kan tillåta liknande eller bättre resultat, med hänsyn på prestanda och bildkvalité, jäm- fört med den enhetliga ljussättningen av Dynamic Resolution Rendering. Metod. Denna undersökning kommer att utföras genom att implementera Variable Rate Shading och Dynamic Resolution Rendering i en Deferred Renderer. Prestan- dan kommer att mätas genom att ta tid för ljussättningspasset och bildkvalitén kommer att mätas genom att jämföra den slutgiltiga bildrutan av båda teknikerna mot den ursprunliga upplösningen via SSIM. Resultat. Övergripande visade Variable Rate Shading jämförelsebar prestanda när den applicerades på Deferred Lighting passet fast hade bildkvalité som liknande mer den originalupplösningen. Slutsatser. Variable Rate Shading visade sig vara jämförbar i prestandan som i jämförelse med dynamic resolution rendering, fast gav bättre möjlighet att bibehålla bildkvalitén. Nyckelord: Rendering, Adaptiv ljussättning, enhetlig ljussättning, Variable Rate Shading, Dynamic Resolution Rendering iii Acknowledgments I would like to thank Avalanche Studios Group for allowing me to conduct my thesis at the Stockholm office which has been very educational and exciting. I would also like to thank Duncan Williams for being my supervisor at the company with several interesting discussions that has helped me with my thesis. Lastly I would like to thank the Graphics Team at Avalanche Studios Group for more insight into graphics programming. v Contents Abstract i Sammanfattning iii Acknowledgments v 1 Introduction 1 1.1 Aim and Research Questions . 3 1.2 Outline . 4 2 Background 5 2.1 Compute Shader . 5 2.2 Deferred Rendering . 5 2.3 Dynamic Resolution Rendering . 6 2.4 Variable Rate Shading . 7 2.5 Structural Similarity Index . 8 3 Related Work 9 4 Method 11 4.1 Implementation . 11 4.1.1 Variable Rate Shading . 11 4.1.2 Dynamic Resolution Rendering . 13 4.2 Heuristics . 13 4.2.1 Haar Wavelet Transform . 14 4.2.2 Luma & Albedo . 15 4.2.3 Silhouette Rendering . 16 4.2.4 Nvidia’s Content Adaptive . 17 4.2.5 Combined . 17 4.3 Evaluation . 17 4.3.1 Scenes Under Test . 19 4.4 Validity . 20 4.5 Reliability . 20 4.6 Delimitations . 21 5 Results and Analysis 23 6 Discussion 37 vii 7 Conclusions and Future Work 39 7.1 Future Work . 39 References 41 A Supplemental Information 45 viii List of Figures 1.1 In-game capture from Red Dead Redemption 2 at 1920x1080 output resolution on PlayStation 4. 1 2.1 Top: Four G-Buffers in Killzone 2 (from left to right), Depth, Albedo, Normal and Specular. Bottom Left: Deferred Composition of lights and G-Buffers. Bottom Right: Final frame after post-processing. (Source: DEFERRED RENDERING IN KILLZONE 2 [26]) . 6 2.2 Dynamic Resolution Rendering works by varying the size of the view- port below the size of the render target. (Source: Dynamic Resolution Rendering Article [11]) . 6 2.3 Visualization of the shading rate texture feature in Variable Rate Shading (Source: Nvidia VRWorks - Variable Rate Shading [20]) . 7 4.1 Pseudo code for the tiling compute shader (in hlsl) used to determine the adaptive shading for Variable Rate Shading. 12 4.2 Pseudo code for edge detection using Haar Wavelet Transform (in c) used to evaluate the desired shading rate. 14 4.3 Scenes under test - Area in rectangle is further investigated . 19 5.1 Left: Global SSIM value. Right: Performance including both the time for the overhead and the lighting pass. (Note: It is not a linear increase between resolutions) . 29 5.1 Left: Global SSIM value. Right: Performance including both the time for the overhead and the lighting pass. (Note: It is not a linear increase between resolutions) . 30 5.2 Local SSIM Values for the different heuristics in Desert Scene at 1200p. Comparison between resolutions can be seen in the appendix in figure A.1. 31 5.3 Local SSIM Values for the different heuristics in Forest Scene at 1200p. Comparison between resolutions can be seen in the appendix in figure A.2. 31 5.4 Local SSIM Values for the different heuristic in Office Scene at 1200p. Comparison between resolutions can be seen in the appendix in figure A.3. 31 5.5 Local SSIM Values for the different heuristic in Sewer Scene at 1200p. Comparison between resolutions can be seen in the appendix in figure A.4. 32 ix 5.6 A zoomed in section of the Forest Scene on the final image using different heuristics at 1200p (Crop at 240x170 resolution). 33 5.7 A zoomed in section of the Sewer Scene on the final image using different heuristics at 1200p (Crop at 240x170 resolution). 34 5.8 A zoomed in section of the Office Scene on the final image using different heuristics at 1200p (Crop at 240x170 resolution). 35 5.9 A zoomed in section of the Desert Scene on the final image using different heuristics at 1200p (Crop at 240x170 resolution). 36 A.1 Local SSIM values in Desert Scene at 1200p, 1600p and 2400p. 46 A.2 Local SSIM values in Forest Scene at 1200p, 1600p and 2400p. 47 A.3 Local SSIM values in Office Scene at 1200p, 1600p and 2400p. 48 A.4 Local SSIM values in Sewer Scene at 1200p, 1600p and 2400p. 49 A.5 Box plot of desert scene with 1000 samples . 50 A.6 Box plot of sewer scene with 1000 samples . 51 A.7 Box plot of office scene with 1000 samples . 52 A.8 Box plot of forest scene with 1000 samples . 53 x List of Tables 2.1 Memory usage required for Shading Rate Texture at different resolutions. 8 4.1 Factors of interests for test cases and evaluation. 17 5.1 Median time (ms) for upsampling (DRR) and tiling (VRS) at different resolutions using different heuristics. 24 5.2 Median render time (ms) for the deferred lightning pass and global SSIM value at different resolutions using different heuristics in the Desert scene. More detailed values can be seen in the appendix at A.5. 24 5.3 The percentage of time (%) taken for Variable Rate Shading and Dy- namic Resolution Rendering based on the time of the native resolution in the Desert scene when only comparing the deferred lighting pass. 24 5.4 Median render time (ms) for the deferred lightning pass and global SSIM value at different resolutions using different heuristics in the Office scene. More detailed values can be seen in the appendix at A.7. 25 5.5 The percentage of time (%) taken for Variable Rate Shading and Dy- namic Resolution Rendering based on the time of the native resolution in the Office Scene when only comparing the deferred lighting pass.