Rochester Institute of Technology RIT Scholar Works Theses Thesis/Dissertation Collections 3-1-2008 Image segmentation and pigment mapping of cultural heritage based on spectral imaging Yonghui Zhao Follow this and additional works at: http://scholarworks.rit.edu/theses Recommended Citation Zhao, Yonghui, "Image segmentation and pigment mapping of cultural heritage based on spectral imaging" (2008). Thesis. Rochester Institute of Technology. Accessed from This Dissertation is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. Image Segmentation and Pigment Mapping of Cultural Heritage Based on Spectral Imaging by Yonghui Zhao A dissertation proposal submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Chester F. Carlson Center for Imaging Science Rochester Institute of Technology March 10, 2008 CHESTER F. CARLSON CENTER FOR IMAGING SCIENCE ROCHESTER INSTITUTE OF TECHNOLOGY ROCHESTER, NEW YORK CERTIFICATE OF APPROVAL Ph.D. DEGREE DISSERTATION The Ph.D. Degree Dissertation of Yonghui Zhao has been examined and approved by the dissertation committee as satisfactory for the dissertation required for the Ph.D. degree in Imaging Science Dr. Roy S. Berns, dissertation Advisor Dr. Joseph G. Voelkel Dr. David W. Messinger Dr. Mark D. Fairchild Date DISSERTATION RELEASE PERMISSION ROCHESTER INSTITUTE OF TECHNOLOGY CHESTER F. CARLSON CENTER FOR IMAGING SCIENCE Title of Dissertation: Image Segmentation and Pigment Mapping of Cultural Heritage Based on Spectral Imaging I, Yonghui Zhao, hereby grant permission to Wallace Memorial Library of R.I.T. to reproduce my thesis in whole or in part. Any reproduction will not be for commercial use or profit. Signature Date Image Segmentation and Pigment Mapping of Cultural Heritage Based on Spectral Imaging by Yonghui Zhao Submitted to the Chester F. Carlson Center for Imaging Science in partial fulfillment of the requirements for the Doctor of Philosophy Degree at the Rochester Institute of Technology Abstract The goal of the work reported in this dissertation is to develop methods for im- age segmentation and pigment mapping of paintings based on spectral imaging. To reach this goal it is necessary to achieve sufficient spectral and colorimetric accuracies of both the spectral imaging system and pigment mapping. The output is a series of spatial distributions of pigments (or pigment maps) composing a painting. With these pigment maps, the change of the color appearance of the painting can be simulated when the optical properties of one or more pigments are altered. These pigment maps will also be beneficial for enriching the historical knowledge of the painting and aid- ing conservators in determining the best course for retouching damaged areas of the painting when metamerism is a factor. First, a new spectral reconstruction algorithm was developed based on Wyszecki’s hypothesis and the matrix R theory developed by Cohen and Kappauf. The method achieved both high spectral and colorimetric accuracies for a certain combination of illuminant and observer. The method was successfully tested with a practical spectral imaging system that included a traditional color-filter-array camera coupled with two 5 optimized filters, developed in the Munsell Color Science Laboratory. The spectral imaging system was used to image test paintings, and the method was used to retrieve spectral reflectance factors for these paintings. Next, pigment mapping methods were brought forth, and these methods were based on Kubelka-Munk (K-M) turbid media theory that can predict spectral reflectance fac- tor for a specimen from the optical properties of the specimen’s constituent pigments. The K-M theory has achieved practical success for opaque materials by reduction in mathematical complexity and elimination of controlling thickness. The use of the gen- eral K-M theory for the translucent samples was extensively studied, including deter- mination of optical properties of pigments as functions of film thickness, and prediction of spectral reflectance factor of a specimen by selecting the right pigment combination. After that, an investigation was carried out to evaluate the impact of opacity and layer configuration of a specimen on pigment mapping. The conclusions were drawn from the comparisons of prediction accuracies of pigment mapping between opaque and translucent assumption, and between single and bi-layer assumptions. Finally, spectral imaging and pigment mapping were applied to three paintings. Large images were first partitioned into several small images, and each small image was segmented into different clusters based on either an unsupervised or supervised classification method. For each cluster, pigment mapping was done pixel-wise with a limited number of pigments, or with a limited number of pixels and then extended to other pixels based on a similarity calculation. For the masterpiece The Starry Night, these pigment maps can provide historical knowledge about the painting, aid conser- vators for inpainting damaged areas, and digitally rejuvenate the original color appear- ance of the painting (e.g. when the lead white was not noticeably darkened). Acknowledgements I would like to first thank my advisor, Dr. Roy S. Berns, for his patience, inspiring ad- vices and ceaseless efforts in teaching me how to become a research scientist. Without his supervision, this research work would not have been a reality. He has also been a wonderful role model and a great mentor for me. I would like to thank Dr. Joseph G. Voelkel, Dr. David W. Messinger and Dr. Mark D. Fairchild for serving on my thesis committee and reviewing my thesis. I highly appreciate their help and consideration for arranging my candidacy exam and final defense. Thanks again for your valuable time and guidance. I would like to thank Lawrence A. Taplin for his help on operating the spectral camera, writing efficient programs, and inspiring discussion - Thank you Lawrence. I would also like to extend my gratitude to all the people of the Munsell Color Science Laboratory for the help given during the various stages of my thesis, especially Dr. Mitchell R. Rosen, Dr. David R. Wyble, Dr. Philipp Urban, Mr. Rodney L. Heckaman, Dr. Yongda Chen, Mahdi Nezamabadi, Mahnaz Mohammadi, Dr. Hongqin Zhang, Changmeng Liu, Dr. Jiaotao kuang, Valerie Hemink and Colleen M. Desimone. I would also like to thank the artist, Bernard Lehmann, who had drawn two won- derful paintings for my research. Both paintings are very good targets for both imaging and reproduction. I would also like to thank the Andrew W. Mellon Foundation, the National Gallery of Art, Washington, the Museum of Modern Art, New York, the Institute of Museum and Library Services, and Rochester Institute of Technology for their financial support. Finally, I would like to thank a very special person, my dear husband, for his help on the proofreading of this dissertation. Thank you for supporting me all the time and giving me a shoulder to lean on when times were hard. This work is dedicated to my parents, my husband, Ligang and my lovely son, Eric for their love and support. Contents 1 Introduction 34 1.1 Motivation . 34 1.2 Dissertation Outline . 37 1.3 Terminology . 39 2 Background Literature Review 41 2.1 Multispectral Imaging . 41 2.1.1 Historical Development . 41 2.1.2 Camera Model . 43 2.1.3 Spectral Reconstruction Algorithms . 45 2.1.3.1 Direct Reconstruction . 45 2.1.3.2 Reconstruction by Interpolation . 48 2.1.3.3 Learning-based Reconstruction . 50 2.1.4 Summary . 53 2.2 Color-Match Prediction for Pigment Materials . 54 2.2.1 Introduction . 54 2.2.2 Surface Correction . 55 2.2.3 Kubelka-Munk Theory . 58 CONTENTS 9 2.2.3.1 Historical View . 58 2.2.3.2 General Theory . 59 2.2.3.3 Limitations of Kubelka-Munk Theory . 61 2.2.4 Pigment Mixing Models . 62 2.2.4.1 Introduction . 62 2.2.4.2 Two-Constant Kubelka-Munk Theory . 63 2.2.4.3 Single-Constant Kubelka-Munk Theory . 63 2.2.4.4 Determination of Absorption and Scattering Coeffi- cients . 64 2.2.5 Summary . 66 2.3 Pigment Identification . 67 2.3.1 Introduction . 67 2.3.2 Modern Analytic Techniques . 68 2.3.3 Near Infrared Imaging Spectroscopy . 69 2.3.4 Visible Reflectance Spectroscopy . 73 2.3.5 Summary . 79 2.4 Pigment Mapping . 80 2.4.1 Introduction . 80 2.4.2 Pigment Mapping Algorithms . 81 2.4.3 Summary . 86 3 Image-Based Spectral Reflectance Reconstruction 88 3.1 Introduction . 88 3.2 Matrix R Theory . 89 3.2.1 General Theory . 89 3.2.2 Some Applications . 91 CONTENTS 10 3.3 Matrix R Method . 94 3.3.1 Spectral Transformation . 94 3.3.2 Colorimetric Transformation . 95 3.3.3 Combination of Two Transformations . 96 3.4 Experimental . 97 3.5 Results and Discussions . 99 3.6 Conclusions . 107 4 Pigment Mapping for Opaque Materials 110 4.1 Introduction . 110 4.2 Database Development . 112 4.2.1 Methods . 112 4.2.1.1 Characterization of White Paint . 113 4.2.1.2 Characterization of Other Paints . 114 4.2.1.3 Further Simplification . 115 4.2.2 Experimental . 118 4.2.3 Results and Discussions . 119 4.2.4 Summary . 120 4.3 Colorant Formulation for Simple Patches . 122 4.3.1 Pigment Database . 122 4.3.2 Color-Matching Prediction . 123 4.3.3 Summary . 126 4.4 Pigment Mapping for a Complex Image . 128 4.4.1 Introduction . 128 4.4.2 Experimental . 129 4.4.3 Pigment Database . 129 CONTENTS 11 4.4.4 Image Segmentation . 130 4.4.5 Pigment Mapping . 132 4.4.6 Summary . 135 4.5 Conclusions . 136 5 Improvement of Spectral Imaging by Pigment Mapping 138 5.1 Introduction .