The Performance of Image Difference Metrics for Rendered HDR Images Jørn Skjerven Master’s Thesis Master of Science in Media Technology 30 ECTS Department of Computer Science and Media Technology Gjøvik University College, 2011 Avdeling for informatikk og medieteknikk Høgskolen i Gjøvik Postboks 191 2802 Gjøvik Department of Computer Science and Media Technology Gjøvik University College Box 191 N-2802 Gjøvik Norway The Performance of Image Difference Metrics for Rendered HDR Images Jørn Skjerven 1st July 2011 The Performance of Image Difference Metrics for Rendered HDR Images Contents Contents ................................................... iii 1 Abstract ................................................... 1 2 Acknowledgements ............................................ 3 3 Introduction ................................................. 5 3.1 AreasofResearch ................................. .......... 5 3.2 ProblemDescription .............................. ........... 5 3.3 Justification .................................... .......... 5 3.4 ResearchQuestion ................................ .......... 5 3.5 ResearchMethodology............................. ........... 6 3.6 ThesisOrganization.............................. ............ 6 4 State of the Art ............................................... 9 4.1 ImageAssembly ................................... ......... 9 4.1.1 CameraResponseFunction . ........ 10 4.1.2 GhostRemoval .................................. ...... 11 4.1.3 ImageStorage .................................. ...... 12 4.2 ToneMappingOperators............................ ........... 13 4.2.1 Global Tone Mapping Operators . ......... 15 4.2.2 Local Tone Mapping Operators . ......... 18 4.3 PerceptualExperiment ............................ ............ 32 4.4 PerceptualUniformEncoding. ............. 33 4.5 ImageQualityMetrics ............................. ........... 34 4.5.1 Measurement Metrics . ........ 35 4.5.2 PerceptualMetrics ............................. ......... 36 5 Experiment .................................................. 49 5.1 SceneSetup ...................................... ........ 49 5.1.1 SimpleScene ................................... ...... 50 5.1.2 AdvancedScene ................................. ...... 51 5.2 CaptureofImages................................. .......... 52 5.3 ToneMapping ..................................... ........ 55 5.4 PerceptualExperiment ............................ ............ 58 5.5 EncodingFramework............................... .......... 60 5.6 ImageComparison ................................. ......... 61 6 Results ................................................... 65 6.1 Results from Perceptual Experiment . ............... 65 6.2 Results from Image Quality Metrics . .............. 67 6.3 CorrelationofFindings. .. .. .. .. .. .. .. .. .. .. .. .. .. ............. 69 iii The Performance of Image Difference Metrics for Rendered HDR Images 6.3.1 Metric vs Metric Correlation . ........... 74 7 Conclusion and Further Research ................................... 79 7.1 Conclusion ...................................... ......... 79 7.2 FurtherResearch ................................. .......... 83 A Output from Tone Mappers ....................................... 85 B Protocol Used for Perceptual Experiment .............................. 95 B.1 SimpleScene..................................... ......... 95 B.2 AdvancedScene................................... ......... 97 C Output from Perceptual Matrices ................................... 99 D Output from Metrics ............................................ 101 E Output from Correlation ......................................... 107 Bibliography ................................................... 125 iv The Performance of Image Difference Metrics for Rendered HDR Images 1 Abstract HDR is a field in image processing that has received a lot of attention in the later years. Techniques for capturing, tone map back to viewable data has been proposed. Many different ideas have been pursuited, some with a background in the Human Visual System (HVS), but the same problem with determining the quality of these reproductions still exist. In low dynamic range imaging, the solution to the problem has been either to do a visual inspection and compare the reproduction against an original, but as this is a labour intensive and time consuming and highly subjective process, and the need for automated measures which can predict quality has resulted in different image difference metrics. As for comparison of HDR and LDR, this is no trivial task. Currently, no method of automated comparison has been deemed a viable solution due to the difference in in dynamic range. In this master thesis, we present a novel framework extending on recent research which enables us to compare HDR and LDR content, and from this using standard image difference metrics to eval- uate the quality of these. These measures are tested against data from a perceptual experiment to verify the stability and quality of the framework. Initial results indicate that the proposed framework enables us to evaluate the quality of such reproductions on the tested scenes, but that some problems are still unsolved. Keywords: High dynamic range, perceptual uniform encoding, tone mapping operators, image difference metrics, perceptual experiment. 1 The Performance of Image Difference Metrics for Rendered HDR Images 2 Acknowledgements This master thesis is the ending of my three year study at the Master in Media Technology in the Faculty of Computer Science and Media Technology at Gjøvik University College. I would like to thank and show my deepest appreciation for my excellent supervisor Ivar Farup for his kind support, motivation and help during the last half year when this thesis is written. Through the many visits to his office and small talk the work grew forward and ended up in this report. I would also like to thank professor Jon Yngve Hardeberg for his excellent class Advanced Color Ima- ging during the first half of the second semester where my interest for color and color science really came to life. I would also like to express my deepest gratitude to the staff of The Norwegian Color Research Laboratory that supported me and given good feedback during my Master Thesis, especially Marius Pedersen, Peter Nussbaum, Arne Magnus Bakke, Gabriele Simone, Dibakar Pant and Raju Shrestha. Without them, I would not have gotten where I am today. I would also like to thank my family and girlfriend for her kind support during long days of work and late nights with work. Last but not least, I offer my regards to all those who have taken part of my experiment or supported me in any respect during the work with this Master Thesis Jørn Skjerven July, 2011 3 The Performance of Image Difference Metrics for Rendered HDR Images 3 Introduction Since the beginning of man, pictures of different scenes has been depicted on different media. One thing that always has been an important, is how we perceive the motive. One of the essential parts of this is the dynamic range and how this is mapped. On most media it is not possible to reproduce a perfect copy, so different techniques has been developed to produce a more realistic rendering. In paintings, Leonardo da Vinci is accredited the technique Chiaroscuro, which emphasizes the differ- ence between light and dark to create an illusion of high dynamic range. This was further adapted into photography and film, with Ansel Adams and his zone system to transfer natural light into specific values on negatives. Today the same problems exist with digital cameras. Systems cannot capture the same range that the eye can, and a lot of adaptation processes that are not fully under- stood influences how we perceive motives. 3.1 Areas of Research The thesis touches a wide area of research. The main work lies in the field of High Dynamic Range(HDR), the capturing, rendering and displaying of such data on Low Dynamic Range(LDR) environments, and how these to can be compared using known measures of quality in the form of image difference metrics. 3.2 Problem Description Today the cameras get more and more elaborate, but still there are cases where the equipment cannot give a dataset which produces an accurate reproduction of a scene. By introducing HDR, we can extend the dynamic range that is available in the raw data, and in post processing, we can choose the different elements that are to be included. HDR imaging can also be used to emphasize other aspects of images and give hyperreal reproduction which seems more real than reality itself. This makes it very hard to evaluate which reproductions are good ones, and earlier research has mainly relied on subjective measures to evaluate quality of the reproductions against the rendering intent. 3.3 Justification In this thesis the hypothesis that by using our framework, quality metrics developed for LDR images can be used to compare HDR against LDR reproductions are proposed. A direct comparison is not possible due to the difference in dynamic range. Some research in the field has been done, and by using the state of the art techniques we will extend this to a wider set of metrics and image attributes. 3.4 Research Question From the problem described above, we will look into three main topics: The quality of different tone mapping operators, the quality of image difference metrics when comparing HDR vs LDR reproduc- tions and usage of original scenes as a reference in perceptual experiments. From these topics,