Color-Image Quality Assessment: From Metric to Application Jens Preiss Color-Image Quality Assessment: From Metric to Application Vom Fachbereich Maschinenbau an der Technischen Universität Darmstadt zur Erlangung des akademischen Grades eines Doktor der Naturwissenschaften (Dr. rer. nat.) genehmigte Dissertation vorgelegt von Dipl.-Phys. Jens Preiss aus Reutlingen Berichterstatter: Dr. rer. nat. Philipp Urban Prof. Dr.-Ing. Edgar Dörsam Prof. Dr.-Ing. Michael Goesele Datum der Einreichung: 18.08.2014 Datum der mündlichen Prüfung: 17.12.2014 Darmstadt 2015 D17 Kurzfassung Die Bewertung der visuellen Qualität von Bildern in der digitalen Bildverarbeitung ist eine entscheidende Voraussetzung für die meisten bildverarbeitenden Systeme. Für eine solche Bewertung der Bildqualität werden hauptsächlich objektive Bewertungen, welche automa- tisch die Bildqualität durch einen Computeralgorithmus voraussagen, verwendet. Die große Mehrheit der objektiven Bewertungen sind sogenannte Bildabstandsmetriken. Diese sagen den wahrgenommenen Unterschied zwischen einem verzerrtem Bild und einer Referenz vo- raus. Aufgrund des beschränkten Verständnisses des menschlichen visuellen Systems ist die Bewertung der Bildqualität kompliziert und noch immer ein offenes Forschungsfeld. Die Mehrheit der Bildabstandsmetriken vernachlässigt Farbinformationen, was schnellere Berech- nungen zulässt. Auch wenn ihre Leistungsfähigkeit für viele Anwendungen ausreichend ist, sind die Metriken nicht in der Lage, die Qualität für eine Vielfalt an Farbverzerrungen korrekt vorauszusagen. Viele Bildabstandsmetriken berücksichtigen außerdem nicht die Betrachtungs- bedingungen, welche einen großen Einfluss auf die wahrgenommene Bildqualität haben können (z. B. ein großer Bildschirm in einem Büro verglichen mit einem kleinen Mobilgerät im hellen Sonnensschein). Das Hauptziel meiner Forschung war die Entwicklung einer neuen Bildabstandsmetrik, genannt iCID, die Bilder auf Standardbetrachtungsbedingungen normalisiert und chromatische Merk- male extrahiert. Die neue Metrik wurde dann als Zielfunktion eingesetzt, um sowohl Gamut- Mapping als auch Tone-Mapping zu verbessern. Beide Methoden stellen wesentliche Trans- formationen zur Farbbildwiedergabe dar. Die Leistungsfähigkeit der vorgeschlagenen Metrik wurde durch visuelle Experimente als auch durch Vergleiche mit subjektiven Beurteilungen bestätigt. Die visuellen Experimente of- fenbaren signifikante Verbesserungen gegenüber hochmodernen Gamut-Mapping- und Tone- Mapping-Transformationen. iCID weist für Gamut-Mapping-Verzerrungen die signifikant höchste Korrelation zu subjektiven Beurteilungen auf und übertrifft für konventionelle Verz- errungen (z. B. Rauschen, Unschärfe und Kompressionsartefakte) fast alle hochmodernen Metriken. Abstract In digital imaging, evaluating the visual quality of images is a crucial requirement for most image-processing systems. For such an image quality assessment, mainly objective assess- ments are employed which automatically predict image quality by a computer algorithm. The vast majority of objective assessments are so-called image-difference metrics which predict the perceived difference between a distorted image and a reference. Due to the limited un- derstanding of the human visual system, image quality assessment is not straightforward and still an open research field. The majority of image-difference metrics disregard color information which allows for faster computation. Even though their performance is sufficient for many applications, they are not able to correctly predict the quality for a variety of color distortions. Furthermore, many image-difference metrics do not account for viewing conditions which may have a large impact on the perceived image quality (e.g., a large display in an office compared with a small mobile device in the bright sunlight). The main goal of my research was the development of a new image-difference metric called improved Color-Image-Difference (iCID) which normalizes images to standard viewing con- ditions and extracts chromatic features. The new metric was then used as objective function to improve gamut mapping as well as tone mapping. Both methods represent essential trans- formations for the reproduction of color images. The performance of the proposed metric was verified by visual experiments as well asby comparisons with human judgments. The visual experiments reveal significant improvements over state-of-the-art gamut-mapping and tone-mapping transformations. For gamut-mapping distortions, iCID exhibits the significantly highest correlation to human judgments and for conventional distortions (e.g., noise, blur, and compression artifacts), iCID outperforms almost all state-of-the-art metrics. Acknowledgments First of all, I would like to thank my supervisor Dr. Philipp Urban who advised me during my graduation and taught me how to be a good scientist. In close collaboration with him, I had the best conditions to successfully pass the time of my graduation. Prof. Edgar Dörsam as the head of our institute provided an extraordinarily productive and positive environment which allowed me to fully focus on my research. I also thank Prof. Michael Goesele who did not hesitate to review this thesis for spending so much of his valuable time. This thesis would not have been possible without my wife Yvonne who helped me immensely with her infinite patience, understanding and sacrifice. It is invaluable to have someoneyou can rely on when you need it the most. I am deeply grateful to her and my family for their continuous support at any given moment. Finally, I wish to thank my colleagues at the Institute of Printing Science and Technology for having a great time inside and outside the institute. They have always been helpful, sharing their knowledge, and — good friends. I am especially indebted to the members of the color group for the successful collaboration. This work was supported by the German Research Foundation within the Emmy-Noether Program of “Perceptually Optimal Reproduction of Color Images considering Device Limits”. Darmstadt, February 2015 Contents Notations v Publications xi 1 Introduction 1 1.1 Motivation . 1 1.2 Objectives . 4 1.3 Methodology . 4 2 Background 7 2.1 Visual Psychophysics . 7 2.1.1 Contrast-Sensitivity Functions . 7 2.1.2 Visual Masking . 8 2.1.3 Multi-Channel Model of the Human Visual System . 9 2.2 Visual Resolution . 10 2.3 Color Spaces . 11 2.3.1 RGB Color Spaces . 12 2.3.2 CIEXYZ . 13 2.3.3 CIELAB . 13 2.3.4 LAB2000HL . 13 2.4 Color Gamut Mapping . 14 3 Image-Difference Metrics 15 3.1 Image Quality . 15 3.1.1 Image Quality Assessments . 16 3.2 Introduction to Image-Difference Metrics . 17 3.2.1 Existing Image-Difference Metrics . 18 3.2.2 Color in Image-Difference Metrics . 20 3.3 Selection of Image-Difference Metrics . 22 i CONTENTS 3.3.1 Mean-Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR) . 22 3.3.2 Structural Similarity (SSIM) Index . 22 3.3.3 Multi-Scale SSIM (MSSIM) Index . 24 3.3.4 Feature Similarity Index (FSIM) . 25 3.3.5 Visual Information Fidelity (VIF) . 25 3.3.6 MeTriX MuX Visual Quality Assessment Package . 26 4 Image-Quality-Assessment Databases 27 4.1 General Properties of Image-Quality-Assessment Databases . 27 4.2 Analysis of Subjective Data . 29 4.2.1 Mean Opinion Scores . 29 4.2.2 Hit Rates . 30 4.2.3 Hit-Rate Ratio . 31 4.2.4 Significance Analysis of Hit Rates . 32 4.3 Conventional Distortions . 33 4.3.1 LIVE Image Quality Assessment Database . 34 4.3.2 Tampere Image Database . 35 4.3.3 Further Conventional Image Databases . 37 4.4 Gamut-Mapping Distortions . 37 4.4.1 Empa Gamut-Mapping Database . 38 4.4.2 Dugay Gamut-Mapping Database . 39 4.5 Combined Gamut-Mapping Database . 39 5 The Color-Image-Difference (CID) Metric 43 5.1 Related Publications . 43 5.2 The Need of a New Image-Difference Metric . 44 5.2.1 Color in Image-Difference Metrics . 44 5.2.2 Viewing Conditions in Image-Difference Metrics . 45 5.3 Image-Difference Features . 47 5.3.1 Feature Extractions . 49 5.3.2 Image-Difference Comparisons . 50 5.3.3 Combining the Image-Difference Features to the Color-Image-Difference (CID) Metric 53 5.4 Elaborating the CID Metric . 55 5.4.1 Working Color Space . 55 5.4.2 Image-Appearance Model . 56 5.4.3 Training of the CID Parameters . 57 ii CONTENTS 5.5 Results and Discussion . 59 5.5.1 Impact of Image-Difference Features on Color Distortions . 59 5.5.2 Impact of the Multi-Scale Approach . 61 5.5.3 Impact of Color on Conventional Distortions . 64 5.5.4 Image-Difference Metrics on the Combined Gamut-Mapping Test Set . 65 5.6 Conclusions . 67 5.7 My Contributions . 68 6 Optimizing Gamut Mapping With Image-Difference Metrics 69 6.1 Related Publications . 69 6.2 Gamut Mapping as a Constrained Optimization Problem . 70 6.2.1 Gamut Mapping . 70 6.2.2 Constrained Gamut-Mapping Optimization . 70 6.2.3 Capability of CID As Objective Function . 71 6.3 Optimizing Gamut Mapping by the Color-Image-Difference (CID) Metric . 72 6.3.1 Modifications of the CID Metric . 72 6.3.2 Optimizing Gamut Mapping . 74 6.3.3 Evaluation of CID-Based Gamut-Mapping Optimization . 77 6.4 Improving the Color-Image-Difference (CID) Metric . 78 6.4.1 Lightness Inversion . 79 6.4.2 Chromatic Ringing . 80 6.4.3 Chromatic Edges . 80 6.4.4 Lightness Banding . 81 6.4.5 The Improved CID (iCID) Metric . 82 6.4.6 Parameter Adjustment . 82 6.5 Performance of the Improved Color-Image-Difference (iCID) Metric . 85 6.5.1 Properties of the iCID Metric . ..