A Volume Rendering Engine for Desktops, Laptops, Mobile Devices
Total Page:16
File Type:pdf, Size:1020Kb
Iowa State University Capstones, Theses and Graduate Theses and Dissertations Dissertations 2012 A Volume Rendering Engine for Desktops, Laptops, Mobile Devices and Immersive Virtual Reality Systems using GPU-Based Volume Raycasting Christian John Noon Iowa State University Follow this and additional works at: https://lib.dr.iastate.edu/etd Part of the Biomedical Engineering and Bioengineering Commons, Computer Engineering Commons, and the Radiology Commons Recommended Citation Noon, Christian John, "A Volume Rendering Engine for Desktops, Laptops, Mobile Devices and Immersive Virtual Reality Systems using GPU-Based Volume Raycasting" (2012). Graduate Theses and Dissertations. 12419. https://lib.dr.iastate.edu/etd/12419 This Dissertation is brought to you for free and open access by the Iowa State University Capstones, Theses and Dissertations at Iowa State University Digital Repository. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. A volume rendering engine for desktops, laptops, mobile devices and immersive virtual reality systems using gpu-based volume raycasting by Christian John Noon A dissertation submitted to the graduate faculty in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Co-majors: Human Computer Interaction; Computer Engineering Program of Study Committee: Eliot Winer, Co-Major Professor James Oliver, Co-Major Professor Stephen Gilbert Chris Harding Tim Bigelow Iowa State University Ames, Iowa 2012 Copyright © Christian John Noon, 2012. All rights reserved. ii TABLE OF CONTENTS! LIST OF FIGURES! v LIST OF TABLES! xi ABSTRACT! xii 1! INTRODUCTION! 1 1.1! What is Volume Rendering?! 1 1.2! Medical Imaging! 3 1.3! Benefits of Volume Rendering! 4 1.4! Real-World Volume Rendering Applications! 7 1.5! Motivation! 9 1.6! Dissertation Organization! 12 2! THE VOLUME RENDERING PIPELINE! 13 2.1! Computer Graphics and the OpenGL Rendering Pipeline! 13 2.2! Volumetric Data! 15 2.3! The Volume Rendering Pipeline! 17 2.3.1! Segmentation! 18 2.3.2! Gradient Computation! 18 2.3.3! Resampling! 19 2.3.4! Classification! 21 2.3.5! Coloring! 22 2.3.6! Shading! 24 2.3.7! Compositing! 25 2.4! Volume Rendering Techniques ! 26 2.4.1! Iso-surface Surface Rendering! 27 2.4.2! Image Splatting! 28 2.4.3! Shear Warp! 29 2.4.4! Texture Slicing! 29 2.4.5! Raycasting! 30 2.5! Raycasting Execution! 30 2.6! A Real-World Example of Raycasting! 31 3! ADVANCED VOLUME RAYCASTING AND APIs! 39 3.1! Advances in Volume GPU-based Raycasting! 39 3.1.1! Rendering Speed Optimization! 39 iii 3.1.2! GPU Texture Optimization! 42 3.1.3! Lighting and Shadowing! 44 3.1.4! Clipping! 47 3.1.5! Rendering Multiple Volumes! 49 3.1.6! Other Advancements! 51 3.2! Volume Rendering APIs! 51 3.2.1! Desktop APIs! 52 3.2.2! Immersive Virtual Reality APIs! 54 3.2.3! Mobile Device APIs! 56 3.2.4! Commercial and Open Source Volume Rendering Applications! 56 3.3! Research Issues! 58 4! METHODOLOGY! 61 4.1! Developing the Rendering Core Foundation! 62 4.1.1! OpenSceneGraph! 62 4.1.2! DCMTK! 63 4.1.3! VR Juggler! 64 4.2! The Desktop Sandbox Application! 65 4.2.1! Architecture! 65 4.2.2! Features! 65 4.2.3! User Interface! 76 4.2.4! Challenges and Contributions! 78 4.3! The Immersive Sandbox Application! 80 4.3.1! Architecture! 80 4.3.2! Features! 81 4.3.3! User Interface and Interaction! 83 4.3.4! Challenges and Contributions! 90 4.4! The Mobile Sandbox Application! 92 4.4.1! Architecture! 92 4.4.2! Raycasting Complications! 97 4.4.3! Memory Limitations! 101 4.4.4! GPU Fragment Operation Bandwidth! 102 4.4.5! GPU Asynchronous Processing and Synchronization! 111 4.4.6! Features! 116 4.4.7! User Interface! 128 4.4.8! Challenges and Contributions! 133 5! VIPRE! 135 5.1! VIPRE Architecture! 135 5.1.1! The vipre Library! 137 5.1.2! The vipreDICOM Library! 138 5.1.3! The vipreViewer Library! 138 5.1.4! The vipreRaycaster Library! 139 iv 5.1.5! The vipreOTSlicer Library! 140 5.1.6! The vipreVATSlicer Library! 140 5.2! Advanced Volume Raycasting Techniques! 141 5.2.1! Empty Space Skipping using Octrees! 141 5.2.2! Phong Illumination! 146 5.2.3! Multi-Pass Rendering for Backface Depth Rasterization! 148 5.3! Bridging Academic Research and Volume Rendering APIs! 153 6! CONCLUSIONS AND FUTURE WORK! 155 6.1! Summary and Conclusions! 155 6.2! Future Work! 158 6.3! Acknowledgements! 158 BIBLIOGRAPHY! 160 v LIST OF FIGURES Figure 1: A Full body view of a virtual forensic autopsy [71] (Top). 2 Volume rendering of the UTCT Chameleon dataset [45] (Bottom-Right). A close up view of the lungs and throat of a three week old infant [71] (Bottom-Right). Figure 2: An X-ray of the chest (Top-Right). A CT image of the chest 4 (Top-Left). An Ultrasound of the abdomen (Bottom-Right). An MRI of the knee (Bottom-Left). Figure 3: Illustration demonstrates the CT process and how a set of 2D 5 slices can generate a 3D volumetric dataset. Figure 4: Multimodal view of a head, tumor, cortical activations and 5 fiber tracts (Right). Several views of a clipping skull for neurosurgical planning (Left). [104] Figure 5: Volume rendering of hurricane Isabel [5] (Top Right). Volume 6 rendering of a frog (Top Left - http://www.cs.utah.edu/~jmk/ images/frog2.jpg). Volume rendering of the internal structure of an engine block (Bottom Left). Volume rendering of the electrostatic potential surrounding the fourth tandem repeat in the Candida Albicans Adhesin Als5p (Bottom Right - http:// ec.asm.org/content/vol9/issue3/cover.dtl). Figure 6: Screenshot of the Sinus Endoscopic system’s interface. 8 Figure 7: Screenshot of the user interface of BodyViz, a volume 9 rendering application designed for surgical planning and medical training. Figure 8: A combination of DOSE and CT data. Red contours show the 10 target volume outline, blue contours show the rectum and pink contours show the bladder (Top). Visualization of the dose distribution on areas that have high CT values. The bladder can be seen in the middle because a contrast agent was used during CT scanning to highlight softer tissues. Figure 9: The OpenGL rendering pipeline. (http://www.songho.ca/ 14 opengl/gl_pipelin e.html) Figure 10: Shows the difference between an isotropic and anisotropic 16 grid. vi Figure 11: The volume rendering pipeline. 17 Figure 12: Diagram of raycasting in 2D where each ray is cast from the 20 eye-point in a perspective projection. Image courtesy of [45]. Figure 13: Diagram of a single voxel C surrounded by 8 neighboring 21 voxels (Left). Diagram of how Trilinear Interpolation can be used to compute the value of C (Right). (http:// en.wikipedia.org/wiki/Trilinear_interpolation) Figure 14: The same volume rendering image using two different color 23 transfer functions with the same opacity transfer function. Figure 15: A black pool ball in a dim room (Left). A black pool ball in a 25 dim room with a small flashlight shown on it (Right). Figure 16: An iso-surface surface rendering of a human skull. (http:// 27 www.aravind.ca/images/ivis_gallery/isoColour.png) Figure 17: Examples of image splatting on a full head dataset [190]. 28 Figure 18: Shear-warp sampling always takes place in orthogonal 29 direction slices. Figure 19: Texture slicing sampling generating view-aligned slices 30 parallel to the image plane. Figure 20: Pseudo code of the volume rendering algorithm using 33 raycasting. Figure 21: Geometric representation of the volume as a surface. 34 Figure 22: Rendering of the volume using a grayscale color transfer 34 function at only the ray/volume intersection points. Figure 23: Rendering of the volume using a grayscale color transfer 35 function and linear opacity transfer function at only the ray/ volume intersections points. Figure 24: Rendering of the volume using a grayscale color transfer 36 function and linear opacity transfer function while resampling and compositing along the entire ray. Figure 25: Rendering of the volume using a muscle/bone color transfer 37 function and linear opacity transfer function while resampling and compositing along the entire ray. Figure 26: An octree division and its tree representation. 40 vii Figure 27: Raycasting using octrees and hierarchical enumeration [47]. 41 Figure 28: A slice of the Head dataset is partitioned using growing 42 boxes (Left). The growing box set converted into a BSP tree (Right). [55] Figure 29: A hand dataset rendered using Phong illumination (Left), 47 shadow mapping (Middle) and deep shadow maps (Right). Notice deep shadow maps are the only technique to produce semitransparent shadows. Figure 30: Interactive exploded view illustration with increasing degrees- 49 of-explosion [110]. Figure 31: Multi-volume rendering by independently slicing each volume 50 and depth sorting the slices into a slice stack. Figure 32: Combination of multiple datasets using multiple rendering 51 modes. From left to right: pre-integration with illumination, transparent isosurfaces, pre-integration with one clipping plane and a corresponding 2D slice. Figure 33: Architecture diagram of the desktop sandbox application. 66 Figure 34: An image of the sandbox application rendering a bounding 67 box encapsulating the volumetric dataset. Figure 35: Examples of different volume rendering techniques 70 supported by the desktop sandbox application including compositing (Top Left), MIP (Top Right) and MinIP (Bottom). Figure 36: A chest cavity CT scan of rendering using different coloring 73 schemes including Cardiac, Muscle and Bone, NIH and Stern from top left to bottom right respectively.