Image-Based Rendering for Volume Visualization by Using Principle Component Analysis by Yinghan Xu, B.Sc. Dissertation Presented to the University of Dublin, Trinity College in fulfillment of the requirements for the Degree of Master of Science in Computer Science University of Dublin, Trinity College September 2017 Declaration I, the undersigned, declare that this work has not previously been submitted as an exercise for a degree at this, or any other University, and that unless otherwise stated, is my own work. Yinghan Xu August 28, 2017 Permission to Lend and/or Copy I, the undersigned, agree that Trinity College Library may lend or copy this thesis upon request. Yinghan Xu August 28, 2017 Acknowledgments I am heartily thankful to my supervisor Dr. John Dingliana for his guidance and support throughout my masters study which set up a strong understanding of the subject, which led to the completion of this thesis. I would like to show my gratitude to all my classmates for helping me throughout the master. I would also like to thank my parents for their support, trust, and encouragement. Last but not the least, I would like to acknowledge and appreciate my girlfriend Xinyue, for spending her time to proofread this dissertation. Yinghan Xu University of Dublin, Trinity College September 2017 iv Image-Based Rendering for Volume Visualization by Using Principle Component Analysis Yinghan Xu, M.Sc. University of Dublin, Trinity College, 2017 Supervisor: John Dingliana This dissertation investigates and analyses imrovement to a PCA approach for image- based rendering in C++ based on paper [2]. This technique allows to visualize volume data with lower GPU requirement and lower computing complicity than direct volume rendering. First, eigenspaces were calculated with a volume data set, pre-rendered using a standard ray-caster, from a spherically distributed range of camera positions. The parallel programming with CUDA was used to speed up the reconstruction process. Then, PCA function in OpenCV is used to calculate eigenspaces and reconstruct results. Artefacts in the result of paper [2] were smoothed by MedianBlur with small computing costs. Results quality, memory cost and frame rate were tested and evaluated under different conditions. The result with different resolution of images, cells dimension, number of eigenvectors were given for real-world application. v Contents Acknowledgments iv Abstract v List of Figures ix Chapter 1 Introduction1 1.1 Motivation....................................2 1.2 Methodology..................................3 1.3 Contribution...................................3 1.4 Summary of Chapters.............................4 Chapter 2 Background5 2.1 Image-based rendering.............................5 2.2 Principle component analysis.........................7 2.3 Volume rendering................................9 2.3.1 Direct volume rendering........................9 2.3.2 Image-based volume rendering..................... 10 2.4 Parallel programming.............................. 12 2.4.1 CUDA.................................. 12 2.4.2 OpenCL................................. 12 vi Chapter 3 Design and Implementation 14 3.1 Basic PCA implementation........................... 14 3.1.1 Put the observed object into the matrix............... 14 3.1.2 Subtract the mean........................... 15 3.1.3 Calculate the covariance matrix.................... 15 3.1.4 Calculate the Eigenvectors and Eigenvalues.............. 16 3.1.5 Sort Eigenvalues............................ 16 3.1.6 Choose the number of components and calculate scores....... 16 3.1.7 Reconstruct the original image..................... 17 3.1.8 Whole image PCA and cell-based image PCA............ 17 3.2 Reconstruction with CPU........................... 18 3.2.1 Crop to cell and put the image pixel value into the matrix..... 18 3.2.2 Calculate eigenvectors......................... 18 3.2.3 Reconstruct the original cell image and put cells together to a whole image.................................. 19 3.3 Reconstruction with CUDA (GPU)...................... 19 3.3.1 Parallel programming design for reconstruction........... 19 3.3.2 CUDA Implementation details..................... 20 3.4 Reconstruction with OpenCV......................... 22 3.4.1 Calculate PCA in OpenCV...................... 22 3.4.2 Reconstruction............................. 24 3.4.3 Speed up reconstruction by OpenCL (GPU)............. 25 3.5 Smooth artefacts between cells......................... 26 3.5.1 Mean blur................................ 27 3.5.2 Gaussian Blur.............................. 27 3.5.3 Median Blur............................... 27 3.5.4 Bilateral Blur.............................. 28 vii Chapter 4 Results and Evaluation 29 4.1 Evaluation methods............................... 29 4.1.1 SSIM................................... 30 4.1.2 HDR-VDP-2............................... 30 4.2 Reconstruction................................. 31 4.2.1 Small training images results...................... 31 4.2.2 Reconstruction speed comparison between CPU and GPU (CUDA) 33 4.2.3 Large training images results...................... 33 4.2.4 Reconstruction speed comparison between CPU and GPU (OpenCL) 36 4.3 Smooth artefacts of edges........................... 37 4.3.1 Blur and Median Blur Comparison.................. 37 4.3.2 Kernel size chosen for Median Blur.................. 39 4.3.3 Comparison between smoothed images and unsmoothed images.. 40 4.4 Image quality, frame rate (reconstruction speed) and memory cost..... 43 4.4.1 Same resolution data set........................ 44 4.4.2 Different resolution data set...................... 47 Chapter 5 Conclusions and Future Work 50 5.1 Conclusions................................... 50 5.2 Future work................................... 52 Appendix A Abbreviations 53 Bibliography 55 viii List of Figures 2.1 Parameterization of the Lumigraph[15].....................6 2.2 Projection process (Reconstruction) [14]....................8 2.3 A volume rendered cadaver head using view-aligned texture mapping and diffuse reflection[5]................................9 3.1 Kernel for blur filter[8].............................. 27 4.1 Head training data with 36 images....................... 31 4.2 Left: reconstructed by 15 eigenvectors. Right: reconstructed by 20 eigen- vectors...................................... 32 4.3 Left: reconstructed by 30 eigenvectors. Right: original image........ 32 4.4 Reconstruction speed comparison between normal CPU and CUDA..... 33 4.5 The SSIM value for these four images are all around 0.967. The top left image used 4 eigenvectors with 5*5 cell dimensions. The top right image used 13 eigenvectors with 10*10 cell dimensions. The bottom left image used 40 eigenvectors with 20*20 cell dimensions. The bottom right image used 90 eigenvectors with 30*30 cell dimensions................ 34 4.6 These four zoomed in images are the same part of area (120*120 pixels) corresponding to figure 8 four images...................... 35 4.7 FPS performance comparison between GPU and only CPU with different cells dimension.................................. 36 4.8 Blur and Median Blur Comparison with Kernel Size 1............ 37 ix 4.9 Blur and Median Blur Comparison with Kernel Size 3............ 38 4.10 Blur and Median Blur Comparison with Kernel Size 5............ 38 4.11 Cells dimension are 10*10. Image quality with the different size of kernel in different number of components....................... 39 4.12 Cells dimension are 20*20. Image quality with the different size of kernel in different number of components....................... 39 4.13 Cells dimension are 30*30. Image quality with the different size of kernel in different number of components....................... 40 4.14 The dataset used here is 720*720 head images. The chart compares qual- ities (SSIM) between smoothed image and unsmoothed image in different cells dimension. Reconstruction with 10*10, 20*20 and 30*30 cells are shown here. The kernel size 3 used in smoothed image............ 41 4.15 The dataset used here is 720*720 head images. The chart compares qual- ities (HDR-VDP-2) between smoothed image and unsmoothed image in different cells dimension. Reconstruction with 10*10, 20*20 and 30*30 cells are shown here. The kernel size 3 used in smoothed image....... 41 4.16 Zoomed in (200*200) comparison between unsmoothed images and smoothed images. The left column is unsmoothed images. The right column is smoothed images, whose kernel size is 3. The first row's images are con- structed by 7 eigenvectors for 10*10 cells dimension. The second row's images are constructed by 25 eigenvectors for 20*20 cells dimension. The third row's images are constructed by 60 eigenvectors for 30*30 cells di- mension...................................... 43 4.17 The relationship between number of eigenvectors used and SSIM value... 44 4.18 The relationship between SSIM value and frame per second (720*720 res- olutions). The corresponding used eigenvectors are labeled upon lines.... 45 x 4.19 The relationship between HDR-VDP-2 quality value and frame per second (720*720 resolutions). The corresponding used eigenvectors are labeled upon lines..................................... 46 4.20 The relationship between SSIM value and frame per second (540*540 res- olutions). The corresponding used eigenvectors are labeled upon lines.... 48 4.21 Three different kinds of resolution data set with the same content. 20*20 cells dimension are used............................. 49 A.1 These three pair of