ADVANCED TOPICS IN SCIENCE AND TECHNOLOGY IN CHINA ADVANCED TOPICS IN SCIENCE AND TECHNOLOGY IN CHINA Zhejiang University is one of the leading universities in China. In Advanced Topics in Science and Technology in China, Zhejiang University Press and Springer jointly publish monographs by Chinese scholars and professors, as well as invited authors and editors from abroad who are outstanding experts and scholars in their fields. This series will be of interest to researchers, lec- turers, and graduate students alike. Advanced Topics in Science and Technology in China aims to present the latest and most cutting-edge theories, techniques, and methodologies in var- ious research areas in China. It covers all disciplines in the fields of natural science and technology, including but not limited to, computer science, mate- rials science, life sciences, engineering, environmental sciences, mathematics, and physics. DongmingLu Yunhe Pan

Digital Preservation for Heritages Technologies and Applications

With 121 figures Authors Prof. Dongming Lu Prof. Yunhe Pan College of Computer Science and College of Computer Science and TechnologyTechnology Zhejiang University, Hangzhou Zhejiang University, Hangzhou 310027, China 310027, China E-mail: [email protected] E-mail: [email protected]

ISSN 1995-6819 e-ISSN 1995-6827 Advanced Topics in Science and TechnologyinChina

ISBN 978-7-308-06599-3 Zhejiang University Press, Hangzhou

ISBN 978-3-642-04861-6 e-ISBN 978-3-642-04862-3 Springer Heidelberg Dordrecht London New York

Library of Congress Control Number: 2009935706 c Zhejiang University Press, Hangzhou and Springer-Verlag Berlin Heidelberg 2010 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law ofSeptember 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

Cover design: Frido Steinen-Broo, EStudio Calamar, Spain

Printedonacid-free paper

Springer is a part of Springer Science+Business Media (www.springer.com) Preface

Cultural heritages include rich information related to social, historical and cultural values. Affected by climate, environmental and other factors, some valuable heritage information is threatened through destruction or disappear- ance, and some is still not utilized sufficiently. How to investigate and utilize such information effectively is a significant scientific and technological issue. Archaeologists, museologists and conservators are working on issues such as the excavation of precious heritage items, the exhibiting of this valuable in- formation and the strengthening of their outline structure, which aims to conserve and utilize the heritage items as well as their values. The development of information technology has shown its significant role inlargeandfast digitalization, personalization and so on. Information tech- nology is more and more important in heritage preservation, including, but not limited to, digitalization, digitally-aided research, conservation, exhibi- tion and utilization. First introduced in the 1980s, information technology was initially used to store information about relics, and then some digital- ization and exhibition applications were implemented. Currently, information technologyisappliedinmanydifferent aspects in heritageinformation preser- vation. Digitalization can store the heritageinformation in digital format therefore prolonging the “life” of the heritage items. Digitally-aided research technologies can help to improve the effect and efficiency of the archaeological research.Digitally-aided conservation technologies can simulate the conser- vation effect and monitor the heritage items in real-time, hence avoiding unpredictedlosses. Digital exhibition technologies can remove thetimeand space limitations of traditional exhibitions, and can also exhibit the implicit values more vividly. Digital utilization can synthetically utilize the historical, cultural and scientific values of cultural heritage items by applying modern science and technology The digital heritage research group of Zhejiang University has been de- voting significant time in this area since 1997, initially taking the Dunhuang Mogao Grottoes as an example and opening up a new area of mural restora- VIPreface tion, cave exploration and Dunhuang style pattern design. During thepast tenyearswehavefound many new and valuable topics related to heritage preservation, and the research was further extended to heritage information acquisition, digitally-aided research, digital conservation anddigital exhibi- tion and utilization. In addition, these technologies are applied to more Chi- nese heritages, such as the Jinsha Site in Sichuan Province, the Liangzhu Site and the Hemudu Site in Zhejiang Province. The research was financially supported by Key Projects of the National Natural Science Foundation of China (No. 69733030), the National Basis Research Program of China (No. 2002CB312106), the National High-Tech Research and Development Program of China (Nos. 2003AA119020 and No. 2006AA01Z305), the National Re- search and Development Program of China (No. 2004BA810B04), the Pro- gram for New Century Excellent Talents at University (No. NCET-04-0535), the Program for “151 Talents” in Zhejiang Province, and the Program for Changjiang Scholars and Innovative Research Team (No. IRT0652). Motivated by the systematic introduction of our work as well as recent developments in the area, we aim to provideacomprehensive and up-to-date coverage of digital technologies in cultural heritages preservation, including digitalization, digitally-aided research, conservation, digital exhibition and digital utilization. Processes, technical frameworks, key technologies, as well as typical systems and applications are discussed in the book. Chapter 1 introduces the significance and the goal of digital heritage preservation. In addition, some technical requirements for heritagepreser- vation are also discussed. Chapter 2 covers some basic knowledgeaboutdigital preservation tech- nologies, including the basis of information acquisition and perception, in- formation analysis and recognition, digital exhibition and interaction. These are fundamental to the following chapters. Chapter 3 describes the digital acquisition technologies needed for various typical heritage items, including the archaeological excavation field, museum preserved sculptures and artifacts, the large site scene and large paintings and murals. Chapter 4 discusses some digitally-aided research techniques during the process of archaeological research, among which computer aided investigation of archaeological sites, excavation and computer aided quantitative analysis and research aredetailed. Chapter 5 presents various digitally-aided conservation technologies, in- cluding digitally-aided investigation, dynamic environmental monitoring of cultural heritages and digitally-aided restoration of cultural heritages. Chapter 6 proposes various technologies for digital exhibition and inter- action. Distinguishedbytheir objectives, theonline heritage exhibition, the reconstructed archaeological sites exhibition and interactive experience in the exhibition hall are introduced. Preface VII

Chapter 7 brings forward the framework of heritage utilization by tak- ing Dunhuang style pattern re-creation and semantic modelingfor ancient Chinese buildings as an example. Chapter 8 gives some systematic examples based on the technologies in- troduced in Chapters 3 to 7, including digital preservation for the Mogao Grottoes, digital preservation for the Jinsha Site, digital reconstruction of the Hemudu site and digital exhibition of the Liangzhu relics. Chapter 9 summarizes the current status and the prospects for future development. The book is intended for researchers and students in the fields of computer science and technology, museology, and archaeology. We hope it can be a ref- erence book for computer science researchers and students, especially within the areas of virtual reality, computer graphics, computer animation, image processing, and so on. The book can also be used as a reference book for museologyand archaeology researchers, whoareworking on digitalheritage preservation applications or projects. We would like to express our gratitude to Dr. Qingshu Yuan, who has helped with collecting materials for this book. We also deeply appreciate the contribution to this book by our colleagues and the PhD candidates. Our thanks would go to Dr. Changyu Diao for collecting part of the materials for Chapter 1, 2, and 3, Mr. Xifan Shi for collecting part of the materials for Chapter 1, 2, and 4, Mr. Jianming Liu for collecting part of the materials for Chapter 5, 7, 8, and 9, as well as Dr. Yabo Dong and Mr. Ming Xia for collecting part of the materials for Chapter 5. We would like to extend our heartfelt gratitude to Caiping Huang for her helpful work on material checking. We also owe a special debt of gratitude to Dunhuang Academy, Chengdu Relics Archaeological Institute, Liangzhu Museum, and Hemudu Site Mu- seum, for their cooperative research work.

Dongming Lu, Yunhe Pan Hangzhou, China August, 2009 Contents

1 Introduction ...... 1 1.1 Cultural Heritage, the Crystallization of History ...... 1 1.2 Cultural Heritage Preservation and Its Objectives ...... 3 1.3 New Requirements of Digital Technologies for Heritage Preservation ...... 5 References...... 6

2 The Basis of Digital Technologies for Cultural Heritage Preservation ...... 9 2.1 Basis of Information AcquisitionandPerception...... 9 2.1.1 Digital Photography and Processing ...... 10 2.1.2 3D Scanning and Processing ...... 12 2.1.3 3S Technology ...... 14 2.1.4 Sensing andWirelessTransmission ...... 16 2.2 Basis of Information AnalysisandRecognition...... 18 2.2.1 Image Processing ...... 18 2.2.2 Intelligent Information Processing ...... 22 2.3 Basis of DigitalExhibitionandInteraction...... 25 2.3.1 Animation...... 25 2.3.2 Real-time Rendering...... 27 2.3.3 Stereo Display ...... 29 2.3.4 NaturalInteraction...... 31 2.4 Summary...... 32 References...... 32

3 Digitalization of Cultural Heritage ...... 35 3.1 Information Acquisition from Archaeological Excavation Sites 36 3.1.1 Preventing Loss of Information from Archaeological Sites...... 36 3.1.2 Process and Technical Framework of Information Acquisition from Archaeological Excavation Sites . . . . . 37 X Contents

3.1.3 Key Technologies for Information Acquisition from ArchaeologicalExcavationSites...... 41 3.1.4 Typical System for Information Acquisition from ArchaeologicalExcavationSitesandApplications..... 44 3.2 Information Acquisition of Museum Preserved Sculptures andArtifacts ...... 44 3.2.1 Digital Technology Makes Sculptures and Artifacts Remain“YoungForever”...... 45 3.2.2 Information Acquisition Process and Technical Framework for Museum Preserved Sculptures and Artifacts...... 46 3.2.3 Key Technologies for Information Acquisition of Museum Preserved Sculptures and Artifacts ...... 48 3.2.4 Devices and Applications...... 55 3.3 Information AcquisitionfromLargeScenes ...... 55 3.3.1 Process and Technical Framework of Large Scene InformationAcquisition ...... 55 3.3.2 Key Technologies of LargeSceneInformation Acquisition...... 56 3.3.3 TypicalApplications...... 61 3.4 Information Acquisition of LargePaintingsandMurals...... 63 3.4.1 Process and Technical Framework of Acquisition of Large Paintings and Murals ...... 63 3.4.2 Key Technologies for Information Acquisition of LargePaintingsandMurals...... 64 3.4.3 Typical Devices and Applications...... 67 3.5 SummaryandProspects...... 69 References...... 70

4Archaeological Research Aiding Technologies ...... 71 4.1 Digital Technology and Archaeological Research ...... 71 4.2 Process and Technical Framework of Archaeological Research Aiding ...... 72 4.2.1 Process of Archaeological Research Aiding Technologies 72 4.2.2 Technical Framework of Archaeological Research Aiding Technologies ...... 73 4.3 Typical Applications...... 82 4.3.1 Utilization of RS ...... 82 4.3.2 Digital Measurement of Large-size Archaeological Sites 83 4.3.3 Computer Aided Bronze Ware Identification Expert System...... 84 4.3.4 Reconstruction Simulation of Stilt Style Buildings of the Hemudu Site...... 85 4.4 Summary andProspects...... 85 References...... 87 Contents XI

5 Digitally Aided Conservation and Restoration of Cultural Heritages ...... 89 5.1 Digitally Aided Investigation ...... 90 5.1.1 Current Situation Investigation by Digitally Aided Technologies...... 90 5.1.2 Process and Technical Framework of Digitally Aided Current Situation Investigation ...... 90 5.1.3 Key Technologies of Digitally Aided Investigation . . . . 95 5.1.4 Typical Digitally Aided Current Situation Investigation System...... 98 5.2 Dynamic Environmental Monitoring of Cultural Heritages . . 99 5.2.1 Process and Technical Framework of Dynamic Environmental Monitoring ...... 100 5.2.2 Key Technologies of Dynamic Environmental Monitoring...... 102 5.2.3 Typical Dynamic Environmental Monitoring System . . 106 5.3 Digitally Aided Restoration of Cultural Heritages...... 107 5.3.1 Process and Technical Framework of Digitally Aided Restoration...... 107 5.3.2 Key Technologies in Digitally Aided Restoration . . . . . 109 5.3.3 An Introduction to Typical Application of Digitally AidedConservationandVirtualRestoration ...... 115 5.4 SummaryandProspects...... 118 References...... 119

6 The Impact of Digital Technologies on the Exhibition of Cultural Heritages ...... 121 6.1 OnlineHeritageExhibition ...... 122 6.1.1 Online Exhibitions BreakingConstraints of Time and Space...... 123 6.1.2 Process and Technical Framework of Online Heritage Exhibitions...... 123 6.1.3 Key Technologies for Online HeritageBrowsing ...... 126 6.1.4 Typical Online Heritage Exhibition Applications . . . . . 132 6.2 DigitalExhibitionsofReconstructedArchaeologicalSites.... 135 6.2.1 Archaeological Sites Exhibition of Reconstructed Original Appearance...... 135 6.2.2 Process and Technical Framework of a Digital Reconstruction Exhibition...... 136 6.2.3 Key Technologies of Digital Reconstruction Exhibition 138 6.2.4 Typical Applications for Digitized Reconstruction andExhibitionofSites...... 141 6.3 Interactive Experience in theExhibition Hall ...... 144 6.3.1 Interactive Experience that Enhances a Sense of Participation ...... 145 XII Contents

6.3.2 Process and Technical Framework of the Interactive Experience in theExhibition Hall ...... 146 6.3.3 Key Technologies of Interactive Experience in ExhibitionHall ...... 148 6.3.4 Typical Application of Interactive Experience System . 154 6.4 Summary andProspects...... 156 References...... 157

7Digital Development and Utilization of Cultural Heritages’ Information ...... 159 7.1 Culture Heritages’Value ...... 160 7.2 Process and Technical Framework of Digital Development andUtilization ...... 162 7.3 Key Technologies for Development and Utilization ...... 166 7.3.1 SourceMaterialExtraction...... 166 7.3.2 Expression and Extraction of Ancient Murals’ ArtisticStyle...... 167 7.3.3 Artistic Style Learning BasedRe-creation ...... 171 7.3.4 Computer-aided Imitation of Murals...... 174 7.4 Introduction of Typical System for the Development and Utilization ...... 176 7.4.1 Computer Aided Art Design and Creating System.... 177 7.4.2 Semantic Modeling for Chinese Ancient Buildings .... 179 7.5 SummaryandProspects ...... 181 References...... 181

8 Applications of Digital Preservation Technologies for Cultural Heritages ...... 183 8.1 Digital Preservation Project for the Mogao Grottoes ...... 184 8.1.1 Digital Acquisition of the Dunhuang Grottoes...... 186 8.1.2 Microclimate Monitoring in the Mogao Grottoes . .... 188 8.1.3 Digitally-Aided Imitation of the Dunhuang Murals . . . 191 8.1.4 ColorSimulationoftheDunhuangMurals...... 192 8.1.5 Dunhuang-style Pattern Creation and Product Development...... 192 8.2 Digital Preservation Project for the Jinsha Site ...... 196 8.2.1 Information Management and Sharingfor ArchaeologicalSites ...... 196 8.2.2 Acquisition and Exhibition of the Excavation Field . . . 198 8.3 DigitalReconstructionProjectoftheHemuduSite ...... 201 8.4 Digital Exhibition of the LiangzhuRelics...... 204 8.5 Summary ...... 208 References...... 209

9 Summary and Prospect ...... 211

Index ...... 215 1 Introduction

Cultural heritagescontainrichinformation regarding society, history and cultural values. How to investigate and utilize such information effectively is a significant scientific and technological issue.

1.1 Cultural Heritage, the Crystallization of History

The history of human civilization has left a wealth of precious cultural her- itage. Cultural heritage mentioned in this book is the tangible heritage con- taining various valuable historical, artistic and scientificartifacts. Heritage Museum-Encyclopedia of China considers all historical cultural relics, no mat- ter whether they can be moved or not, to be part of cultural heritage (Ed- itorial Department of Encyclopedia of China, 2004). Cultural heritage has various forms, including architectural structures, cave temples, tombs, stone inscriptions, murals, heritage sites, and so on. It also covers important ob- jects, works of art and literature, manuscripts, books, and other articles of interest (UNESCO, 1972). Some famous examples of cultural heritage are listed below to show the historical, artistic and scientific values of heritages. As the largest and most complete ancient building in China, the Forbid- den City, also known as the Palace Museum is the imperial palace of the Ming and Qing Dynasties. the Forbidden City covers over 720,000 m2 andhasmore than 9,000 rooms. All architectures were built of wood on a greenish white marble base, roofed with yellow glazed tiles and decorated with magnificent colored drawings, demonstrating the excellent skills ofChinese architecture of more than five hundred years ago. Currently, the Palace Museum preserves a number of precious cultural relics, many of which are unique national trea- sures.Thetotalnumber of relics in the Palace Museumis estimated to be about 1/6 of the total in China. the Forbidden City represents a condensed version of the history of Chinese civilization. The Dunhuang Mogao Grottoes (Yasuo and Dong, 1982) are an artistic treasure and consist of 492 caves containing over 45,000 m2 of murals, 2,415 21Introduction painted sculptures, and more than 4,000 Apsaras statues. In the year 1900, the Library Cave (the 17th cave) was discovered accidentally when the 16th cave was cleared of silt. It contains over 50,000 pieces of cultural relics includ- ing writingsof previous dynasties, hand paintings on paper and on silk, and embroideries dating from the 4th to the 11th century. the Library Cave is a significant discovery in Chinese archaeological history. This heritage site has important historical and scientific values in the study ofChinese and Central Asian history. The so-called Dunhuangology is mainly concerned with studies of the Library Cave writings and the Dunhuanggrotto art. Fig.1.1shows the famous architecture of nine floors of the Dunhuang Mogao Grottoes.

Fig. 1.1. The architecture of nine floors of the Dunhuang Mogao Grottoes (With permission from Changyu Diao)

In 1974, terracotta warriors and horses (Qin Shihuang Terracotta War- riors and Horses Museum, 1983) were found in the eastern Mausoleum of Qin Shihuang in Lintong, Xi’an, the largest ancient military museum in the world. More than 2,000 pieces of pottery warriors and horses, more than 30 war chariots, over 40,000 bronze weapons of variouskinds,aswellasmany other relics were excavated fromthreepits.Life-size pottery warriors and horses are arranged in an orderly manner, full of power and grandeur. These warriors represent different regiments of army units, such as chariot soldiers, infantry,andcavalry. The terracotta warriors and horses, as the epitome of the powerful army of the Qin Dynasty, are invaluable for studying the history, politics, militarism, economics, culture, art, and technologyof the Qin Dynasty. The terracotta warriors and horses were created in animated 1.2 Cultural Heritage Preservation and Its Objectives 3 poses. The gestures and facial expressions show the distinct personalities and characters of various people at the time. They represent the peak of the de- velopment of ancient Chinese clay sculptures. Fig. 1.2 shows the first and largest pit of the terracotta warriors and horses ofQin Shihuang.

Fig. 1.2. The first pit, excavated in 1974, is the largest pit of terracotta warriors and horses ofQin Shihuang.Itcoversanareaof 230×62 metres and contains more than 6,000 pieces of terracotta warriors and horses (With permission from Qingshu Yuan)

The Egyptian pyramids (Macaulay, 1982) built 4,500 years ago are called “One of the Seven Wonders of the Ancient World”. From the 3rd to the 13th dynasty,ittooktendynasties to build the pyramids. The pyramids are filled with the ancient Egyptians’ wisdom and they still retain many mysteries. They attract many scientists, archaeologists, historians, and tourists from all over the world. Fig.1.3showstheGreat Sphinx ofGiza.

1.2 Cultural Heritage Preservation and Its Objectives

The primary objective of cultural heritage preservation is to protect the au- thenticity and integrity of cultural heritage (UNESCO, ICCROM, ICOMOS, 1994). Authenticity and integrity are two important concepts conceived in the Convention Concerning the Protection of the World Cultural and Natu- ral Heritage (UNESCO, 1972), and also are the basic objectives of cultural heritage preservation. The preservation target is to uncover theimplicit value, and find its historical context from cultural heritage itself. 41Introduction

Fig. 1.3. The pyramids and the Great Sphinx ofGiza. The largest pyramid is for Khufu, the 2nd pharaoh of the 4th ancient Egyptian dynasty. The Sphinx was built according to Khufu’s facial features (With permission from Qian Fan)

At present, cultural heritage preservation includes mainly archaeological excavation, archaeological research,archive management, conservation, exhi- bition, and utilization. Archaeological excavation includes field excavation, underwater excava- tion, aerial excavation, and so on. By variousmeans,wecancollect unearthed and other artifacts properly, make an archaeological interpretation, and fi- nally complete an archaeological excavation report. Digital technology has expanded archaeological excavation work in various ways. For example, the total station makes measurements of the unearthed cultural relics’ positions more efficient and accurate. 3D scanners can easily record the geological sur- face shape information of the archaeological site. Archaeological research is dedicated to finding the historical, artistic, and scientific value of cultural heritage, and investigation the historical process, the state of civilization, and the technological level. These goals are achieved through analyzing comprehensivelythephysical characteristics, pattern char- acteristics, geographical distribution, and related documentation of cultural heritage items. Digital technology plays an important role in the process of archaeological research; for example, techniques for dating relics based on their shape analysis, and techniques that use 3D visualization to predict the location of heritage sites. All these techniques lend support to archaeological research. Archive management includes collecting, recording,organizing, adding, deleting and modifying cultural heritage information. Archives of cultural 1.3 New Requirements of Digital Technologies for Heritage Preservation 5 heritage usuallyincludethenumber, size, weight, material, unearthed posi- tion, unearthed time, dynasty, status, photos, and other information of the relics. With the development of digital technology, digital images, digital videos, 3D models, and other multimedia information have gradually become integral components of cultural heritage archival management. Conservation work includes monitoring, safely storing, and restoring her- itage items. For example, the surrounding environment such as thetemper- ature, humidity, density of carbon dioxide, and the state of microorganisms and vegetation should be monitored in grottoes. The principle of restoration work is to “restore so as to be the same as before”, that is, to maintain the cultural characteristics of the heritageitemsoastoensureitsauthentic- ity. Digital technology can improve environmental monitoring efficiency. It also helps to strengthen security of heritage information through biological authentication and digital encryption, especially on the net. In the work of restoration, digital technology can help to analyze the physical characteristics of cultural heritage items and restore their original status. Exhibition work is an important means of cultural diffusion and educa- tion, including exhibitions in museums, on site, and even on the Web. Not only is the basic information about the cultural heritage item itself exhibited, but also related historical, cultural and scientificinformation. Digital tech- nology has brought new forms to exhibitions. For example, virtual reality can make visitors step into the scenes of ancient life through an interactive im- mersion exhibition, and also give visitors the opportunity to enjoy the images of cultural heritage freely throughthenetwork. Utilization of cultural heritage leads to a better understanding of the culture. Tourism, film, television, game production, imitation, printing, and publication are all utilizations of cultural heritage. With the help of digital technology, it is possible to promote the utilization of cultural heritage val- ues through, for example, pattern creation, animation, and recreation of an ancient cultural style. It is also possible to solve conflicts between the conser- vation and utilization of cultural heritage, for example, the conflict between conservation and tourism.

1.3 New Requirements of Digital Technologies for Heritage Preservation

Digital preservation technologies have been applied at home and abroad,and have achieved initial success. However, some problems still exist and new requirements are also being raised. Intermsof archaeological excavations, althougharchaeologists can ob- tain a lot of information about excavation sites through photographs and drawings, this has not yet achieved the goal of recording all the information. By the time archaeological excavation work is completed, lots of important information produced during excavation might be lost forever. 6 1 Introduction

In terms of archaeological research, there are still many important cultural heritage relics yet to be discovered. Also, secondary damagetorelicsmay occur when archaeologists physically measure and examine them. In addition, many senior experts need to be assembled in one place and time during heritage analysis and identification. In terms of archive management, the advantages of digital archive man- agement have gradually become apparent. Major heritage units have now turned their attention to digital archive management. However, special equip- ment is in urgent need to transfer old materials from non-digital to digital forms, such as drawings, films, slides, microfilms, photographs, audio tapes, and video tapes. Such digitalization devices include scanners, digitizers, audio and video capture cards. In terms of conservation, software can assist experts to mark disease re- gions and create conservation plans for cultural heritageitems.Sensors, wire- less network segments, and other equipments can help monitor temperature, humidity, and carbon dioxide concentration of the environment, enabling us totake necessary measures to reduce the environmental impact on cultural heritage items. Virtual restoration experiments have the advantageofbeing abletobe reversed or repeated compared withdirect physical restoration. They can also reduce difficulties and risks in the restoration of cultural her- itage items. In terms of exhibitions, visitors can only visit a museum during opening hours, and the content of the exhibition remains the same for all visitors that cannot meet any personalized needs. Also, cultural heritage items suffer a lot from weather or man-made destroy, and original condition of them can not beexhibited Now augmented reality technologyismaturing and becoming applicable to museum exhibitions. In addition, exhibitions need to allow visitors to participate in an interactive experience to get a real feeling for the heritage relic. Excessive utilization of cultural heritage items has caused severe loss to the cultural heritage. This situation needs to be changed urgently. It is nec- essary to determine how to maximize development and utilization without damaging cultural heritage. Therefore, digital technology will play an impor- tant role in the development and utilization of heritage information because of the non-contact and non-destructive characteristics of its virtual role. With the emergence of all kinds of new theories and equipments, preser- vation technologies are also improving with time. In chapters 3 to 7, digital preservation technologies will be introduced in order of their application, as recording, researching, exhibition, conservation, and utilization of cultural heritage relics.

References

Editorial Department of Encyclopedia of China (2004) Heritages & Museum- Encyclopedia of China. China Encyclopedia Press, Beijing (in Chinese) References 7

Macaulay D (1982) Pyramid. Houghton Mifflin, Boston Qin Shihuang Terracotta Warriors and Horses Museum (1983) Terra-cotta War- riors & Horses at the Tomb of Qin Shihuang. Cultural Relics Press, Beijing (in Chinese) UNESCO (1972) UNESCO World Heritage Convention, article 1, 2. http://whc. unesco.org/en/conventiontext/. Accessed 10 Sept 2009 UNESCO, ICCROM, ICOMOS (1994) Nara Document on Authenticity, arti- cle 13. http://www.international.icomos.org/charters/nara e.htm/. Accessed 10 Sept 2009 Yasuo I, Dong XC (1982) Dunhuang. Shanxi People’s Publishing House, Taiyuan (in Chinese) 2 The Basis of Digital Technologies for Cultural Heritage Preservation

Cultural heritage is unique and non-renewable while digital information about cultural heritage has the advantages of being permanently stored, and conveniently copied and shared. Hence it can provide a new method for sci- entific conservation, research, interactive exhibition, as well as utilization of cultural heritage. Current developments in computer networks, multimedia, virtual reality, and artificial intelligence have provided a solid foundation for the digitalization of cultural heritageinformation. Digital information about cultural relics covers many aspects, including both visual information, such as image, video, and 3D models, and non-visual information, such as components, internal structure, and the conservation en- vironment. Digital information goes throughthefollowing processes: acqui- sition and perception, analysis and recognition, transmission and exhibition. In this chapter, we will describe the technological basis of these processes, which is fundamental to Chapters 3 to 7.

2.1 Basis of Information Acquisition and Perception

Acquisition and perception of cultural heritage information refer to the recording of various properties and status of cultural relics, such as their shape, color, texture, material, internal structure, evolutionary process, and their conservation environment, such as temperature, humidity, brightness, air components, and so. It is used mainly for permanent digital archiving of heritage information and real-time monitoring, which supports subsequent tasks for ensuring data reliability. Information acquisition and perception of culturalheritageareclosely associated with technologies such as digital pho- tography, 3D scanning, 3S (GIS, RS, and GPS), environment perception and wireless transmission, which will beintroduced in this section. 10 2 The Basis of Digital Technologies for Cultural Heritage Preservation

2.1.1 Digital Photography and Processing

Commonly used in heritageinformation documentation, digital photography and processing can acquire, modify and record 2D information into digital format.

• Digital Photography

High resolution and high photorealism are important issues in heritageimage acquisition. Digital photography is more suitable for heritage image acquisi- tion than traditional photography because it is real-time to show, and conve- nient to edit. The resolution of digital photography is also now highenough to meet the requirements of acquisition for cultural heritage information. The first digital camera in the world was invented in 1975 by Steven Sasson, an engineer from the Kodak Electronic Applications Research Centre (Sasson, 2007). It weighed 8.5 pounds and was driven by 16 AA batteries. Thepicturestaken by it were recorded on tape, containing 10,000 pixels. After only three decades, the resolution of shutter digital cameras can now reach about 40 million pixels or more, and significant improvements have been made in lenses, viewfinders, shutters, LED displays, etc. Digital photography can providethe results in an instant, allowing peopletocheck the results immediately, and ensuring the quality of the photography. The data files of digital cameras are now usually stored in magnetic devices or in optical recording media. Common storage medias available include Compact Flash, Secure Digital Memory Card, Smart Media, Memory Stick, MultiMedia Card, Extreme Digital-Picture Card and MicroDrive, etc. Thephotosensitive technologies that are used in digital photographyin- clude mainly CCD (Charge Coupled Device), Super CCD, and CMOS (Com- plementary Metal Oxide Semiconductor)(Jeff, 2003). Digital photographing devices available presently include SLR (Single Lens Reflex) digital cameras and digital cameras with back. The latter can be further divided into shutter types and scanning types. SLR digital cameras, using electronic photosen- sitive components instead of the films of traditional cameras, can make full use of traditional SLR camera lenses and mechanical shutters, but also have new features like increased capacity and the ability to offer a real-time pho- tographic effect. The major providers ofSLR digital cameras include Canon, Fuji, Kodak, Konica Minolta, Nikon, Olympus, and so on. Compared with SLR digital cameras, digital back cameras usually have higher resolution and a better color effect. The photosensitive components of shutter back cam- eras are similar to those of SLR digital cameras in working mode, while the shooting of scanning back cameras, similar to the bulb shooting of tradi- tional cameras, is often used for static objects. The major providers of digital back cameras include Better Light, Foveon, Fuji, Jobo, Kodak, Leaf America, MegaVision, Phase One, Sinar Bron Imaging, and so on. Fig. 2.1(a) shows a shutter type digital back camera produced by Phase One, which can record 2.1 Basis of Information Acquisition and Perception 11

7,216×5,412 pixels with 16 bits per RGB channel in a single shoot (Phase One, 2006). Fig. 2.1(b) shows a scanning type digital back camera produced by Better Light, which can record 10,200×13,600 pixels with 16 bits per RGB channel (Better Light, 2007).

Fig. 2.1. Two types of digital back camera in the current market. (a) Phase One P45+; (b) Better Light Super 10K-HS

• Digital Photo Processing

Digital photo processinggenerallyincludes rotation, cropping, zoom, mosaic, and exposure adjustment, which can be achieved conveniently using digital photo editing tools. It also offers functions such as white balance correction, sharpening, noise reduction, and other effects. Color space is the color range that a color image can reach, such as CMYK (Cyan, Magenta, Yellow, and Key (Black)) and RGB (Red, Green, and Blue)(Abhay, 2003).CMYKisthe color space commonlyusedby output devices such as printing presses and printers. RGB color space has several different types, such as Adobe RGB, Apple RGB, ColorMatch RGB, CIE RGB, sRGB, and so on. Among them, Apple RGB is the default color space for Apple’s display device, and is widely used in graphic designandprinting phototypesetting. CIE RGBisthecolor space standard established by the International Commission on Illumination. For digital photography and processing, sRGB and Adobe RGB are the two most commonlyused color spaces. In the processing of digital photos of cultural relics, white balance is one of the most important issues, because the color of objects will change under different light conditions. Photographs taken in different light may have different color temperatures while the photography of cultural heritage relics has highrequirementsfor color fidelity. White balance makes white objects appear white in photos, and thus balances the appearance of other colors. In fact, color is determined by the light. Thingsthatappeartobe white in normal light may not be white in dark light. The white color in fluorescent light is in fact not white. For all these situations, if we can adjust thewhite balance appropriately, we can ensure that all thecolors in a photo will be represented faithfully. There are many modes of white balance suitable 12 2 The Basis of Digital Technologies for Cultural Heritage Preservation for different shooting scenes, such as automatic white balance, tungsten light white balance, fluorescent light white balance, indoor white balance, and manually adjusted white balance.

2.1.2 3D Scanning and Processing

3D scanning can record the detailed surface information of heritageitems, while 3D processing can handle further tasks such as de-noising,triangula- tion, texture mapping, and so on. They are combined to record the surface information of heritageitems.

• 3D Scanning

Shape information of cultural relics is usually achieved using 3D scanning devices. 3D scanning acquires 3D information of an object’s surface using technologies based on stereo vision, light wave, or sound wave time-of-flight distance measurement. The major specification parameters of 3D scanning include scanning precision, scanning speed, sampling rate, scanning scope, and so on. 3D scanning technology originated in the 1960s (William et al., 1989). In the 1980s, the Cyberware Company developed the world’s first 3D scanners. 3D scanning technology is applied to the acquisition of digital informa- tion of cultural relics in various ways, for example, using a robot arm to make a measurement (contact measurement), or applying optical principles to make a measurement (non-contact measurement). The main tool for con- tact measurement is a trilinear coordinates measuring instrument, which uses a small probe to measure the spatial position of a relic’s surface, point by point, thereby reconstructing the whole surface of the relic. The accuracy of contact measurement is quite high,reaching micrometers. However, because contact with the relic may harm the surface, it is seldom used in heritage surface acquisition. Non-contact measurements, especiallyoneswithhigh speed andhigh sam- pling rate, are more suitable for obtaining digital 3D information of objects with complex shapes. Time-of-flight measurement and triangulation mea- surement are most commonly used methods. In time-of-flight measurement, the laser scanner sendsalaser pulse and then starts timing until the sensor captures the pulse signal reflected by object surface. The roundtrip distance between the scanner and target point can be determinedbymultiplying the laser speed and the flight time. Using laser phase difference, the improved time-of-flight measurement can give a more precise estimate of the flight time. Laser time-of-flight measurement can acquire 3D information from tar- gets as far as several kilometres away, which is especially suitable for the 3D reconstruction of large-scale site scenes. Fig. 2.2(a) is a laser scanner made by Trimble, which can achieve 12 mm accuracy at 100 m distance (Trimble, 2005). 2.1 Basis of Information Acquisition and Perception 13

Triangulation measurement is more suitable for obtaining 3D information of small-scale objects. The high-precision 3D shape of small objects can be obtained using spot lasers, line lasers, and structured light approaches. These three approaches all calculate the spatial position of the point by using a triangle composed of the target point, the emitting point, and the sensor point. Currently, line laser scanning and surface structured light scanning are the two most popular methods. However, surface structured light scanning is more efficient because it can obtain 3D information from more than one million sample points at one time. Fig. 2.2(b) shows a structured light scanner made by 3D CaMega Co., Ltd, which can acquire 1.3 M points in a single scan at 0.06 mm accuracy (3D CaMega, 2009).

Fig. 2.2. Two types of 3D scanners in the market. (a) Trimble 3D GX Scanner; (b) 3D CaMega CS-400 Scanner

• 3D Scanning Data Processing

Information acquired by 3D scanning devices is usually expressed as point clouds in computers. Caused by various errors, there will be noises in point clouds, which should first be removed. To remove the noise, filtering methods should first be used. Then triangulation should be carried out, which trans- forms the point cloud into a triangular mesh model. Finally throughregistra- tion and merging of the triangular mesh model, an integrated polygon mesh model will be obtained. Alternatively, we can obtain the complete point cloud model throughtheregistration and merging of the point cloud first, followed by 3D triangulation. But the current 3D triangulation algorithm is unstable in judging the normal vector orientation in the model surface, and therefore is unsuitablewhen dealing with complex 3D models. The mesh model which integrates all the scanned data may still have holes caused by self-shelter of the scanned object and other factors. These holes should be filled by planar or curved surfaces. To ensure the reliability of the data, the repaired region needs to be marked to show its difference from the original scanned data. Because of differences in accuracy among the 14 2 The Basis of Digital Technologies for Cultural Heritage Preservation scanned data, an integrated polygon grid model may have the problem of uneven distribution of triangular meshes, which can be solved by curvature based polygon mode reconstruction method. The method constructs a dense distribution of triangles in a highcurvaturesurface, and a sparse distribution of triangles in a low curvature surface. The number of polygons needs to be large enough to avoid a loss of detail from the original data and to record surface details reliably. High-precision texture photos should also be taken and mapped to the sur- face of 3D models. The photo texture mapping algorithm includes automatic mappingbased on contour and interactive mappingbased on corresponding points. Usually a series of textures is required to cover the whole surface of a 3D model, so fusion should be carried out in the overlapping areas. Gener- ally, the texture fusion algorithm based on the overall minimum difference can achieve a good result, which is often used in 3D data processing of heritage.

2.1.3 3S Technology

3S technology is a technology integrating Remote Sensing (RS), Geographic Information Systems (GIS), and Global Positioning Systems (GPS).3Sin- volves information acquisition, processing, and application. It has already been applied widely in archaeology because of its fast speed, real-time capa- bility, high precision and largequantify properties in acquisition and process- ing.Integrated applications of RS, GIS and GPS can efficiently collect and process information from archaeological sites, and provide decision support for archaeological excavations.

• GPS

GPS uses a constellation of several medium earth orbit satellites that trans- mit precise radio wave signalstolocate a place on earth,tell the current time, or navigate moving objects on earth.Thesystem,which theUnited States started to develop in the 1970s, took 20 years and cost 20 billion U.S. dollars to complete in March 1994. The system consists of 24 satellites with a global coveragerateof98% and enables overall real-time 3D navigation and positioning in the sea, on land, and in the air. It provides a 24-hour service and is highly automated, precise and efficient. GPS consists of three major parts: a space segment, a ground control segment, and a user’s receiver. As the precise location of the satellite is known, from the GPS observations, we can determine the distance from the satellite to the receiver. One equation can be established by each satellite, and the location of the observation point (X, Y, Z ) can be solved using 3 equations by 3 satellites. Taking into account the error between the satellite clock and the receiver clock, we need to use four or more satellites to form four equations to solve the four unknowns, X, Y, Z, and the clock. Finally, we can obtain the latitude and longitude of the observation point and its elevation. 2.1 Basis of Information Acquisition and Perception 15

GPS is widely used in archaeological excavations, especially in site exca- vation explorations in which geographical information is very important.

• RS

Developed in 1960s, RS has its broad sense and narrow sense. In a broad sense, it is a detection technologythat remotely senses objects and natural phenomena using electromagnetic waves, gravitational fields, electric fields, mechanical waves (sound waves, seismic waves), and so on, without direct contact. In a narrow sense, it is a technology that is used to studytheshapes, sizes, locations, and properties of objects on the earth and their correlations with the environment. The radiation features of electromagnetic waves, from ultraviolet to microwave, of various objects on the earth are obtained using various sensors placed on aerospace carriers (including near-earth carriers) at different heights. Those features are then formed into images, which are then transmitted and processed. Through such procedures, the attributes of objects on the earth are identified, and their temporal and spatial changing rules are explored. Multi-sensors, high-resolution, and multi-temporal data are the distinctive features of contemporary development of RS technology. The application and analysis of RS information is currently undergoing anumberof changes from the analysis of single remote sensing data to the analysis offused information from multiple data sources, from static analysis to dynamic monitoring anal- ysis, from qualitative investigation to computer-aided automatic quantitative investigation. Aerial RS has become an important aspect of RS development for the reason of its mobility and high-resolution. RS archaeology, as its name suggests, is the nondestructive detection of objects on the ground, underground, or underwater using RS technology. To be specific, we detect, record, and analyze archaeological sites and their re- gional environments from four levels, namely aerospace, aviation, ground and underground, using geophysical means such as electromagnetic and seismic waves, and gravitational, magnetic, and electric fields. Using RS archaeol- ogy, the information obtained is no longer limited by visible light and au- dible sound waves detected by human eyes and ears. Any trivial changes or abnormalities in attributes detectable by the instruments can be recorded; therefore, RS technology can provide much more detailed archaeological in- formation.

• GIS

GIS is a geographical technology rapidly developed in the 1960s as a result of interdisciplinary studies. Based on the geographical space database and geographical coordinates (latitude and longitude), GIS provides a variety of space and dynamic geographic information. It has the ability to handle a 16 2 The Basis of Digital Technologies for Cultural Heritage Preservation variety of geographic data and provides services for geography researchers and geographic decision-making. GIS acts as a computer hardware and soft- ware system; but internally it is a geospatial information model consisting of computer programs andgeographicaldata. With the emergence of “Digital Earth”, GIS is developing from a 2D sys- tem into a multi-dimensionaldynamic system or even network system, to meet the urgent needs of the theoretical development and many other fields such as resources management, environment monitoring, and city planning. One of these technical developments is based on developing a client/server ar- chitecture to ensure that the data and procedures on the server are accessible to its end-users. Another is to develop, through the Internet, Internet GIS or Web-GIS to access various geospatial data remotely, including graphics and images, and to carry out various kinds ofgeospatial analysis. This kind of development furthers the connection of GIS to the information highway through modern telecommunication technologies.

• Integration of GPS, RS, and GIS

GPS,RS,andGIS have their own distinctive advantages. By integrating them, we can develop a system that is powerful in acquisition, positioning, and analysis. RS and GPS are collection tools; RS plays the role of the data source and GPS plays the role of positioning. Their combined results need to be integrated in GIS. GIS can be regarded as the processing hub, while RS and GPS are the two wings of information acquisition.

2.1.4 Sensing and Wireless Transmission

Sensing and wireless transmission techniques are used for real-time heritage sites information acquisition, such as temperature, humidity, gas density, and so on, and transmission them for further analysis.

• Sensing Technology

Environmental information is acquired by reading the data from sensors. Temperature, humidity, photosensitive, and gas sensors are widely used in environmental monitoring systems. Inappropriate temperature or humidity may lead to distortion, cracking or even mildewing of relics. Temperature sensors can be classified as metal thermal resistor, semiconductor resistor or thermoelectric force sensors. Hu- midity sensors can be classified as water-affinity or non-water-affinity types. Currently, temperature and humidity sensors are always integrated into one module while maintaining high accuracy and stability. For instance, the Sen- sirion SHT75 (Sensirion Company, 2009) can achieve accuracies within 1.8% for humidity and 0.3◦Cfor temperature. 2.1 Basis of Information Acquisition and Perception 17

Some types of relics, such as paintings and silks, are quite sensitive to light. Light sensors can be classified as photocells, photo resistors, photodiodes, and phototransistors. Photo resistors are widely used because of their high sensitivity, small packagesizeandlow cost. Inappropriate gas density may also have a negative influence on the con- servation of relics. For instance, in an environment with high carbon dioxide density andhumidity,thecarbon dioxide dissolved in water will lower thepH value of the surface of relics, and thus cause deterioration. Gas sensors today can detect many types of gases such as carbon monoxide, carbon dioxide, sulfur dioxide, etc. They can be divided into semiconductor, electrochemical, solid electrolyte, contact combustion, photochemical and polymer gas sen- sors. For instance, Telaire 6004 CO2 sensor module (GE Corporation, 2006) is a photochemical carbon dioxide sensor that provides a digital RS232 inter- face for data acquisition, and can reach a high level of accuracy (5%) in the range of 0∼5000 ml/m3.

• Wireless Communication

Because many cultural heritage sites are located in the wild where wired com- munication infrastructures are usually unavailable, wireless communication is the only choice for environmental data transmission. Wireless communication technologies used in heritage conservation can be classified into those aiming at low power consumption and those aiming at long range data delivery. Low power wireless communication is a basic requirement, because of the difficulties involved in deploying power supply infrastructure in the wild, and the constraints of heritage conservation. Wireless Sensor Network (WSN) is one of the most representative low power communication technologies. In WSNs, hundreds of low power, low cost sensor nodes operate as a large scale self-organized multi-hop mesh network. Although each of them could only communicate with each other over a limited distance, the whole network could relay environmental data to the sink node. WSN systems can work even continuously for more than 1 year with two AA batteries, thus greatly reducing the battery replacement or charging overhead in maintaining the system. As WSN is a newly emerged technology, there are still many unresolved problems. One is hardware. As the hardware of sensor nodes is highly de- pendent on their application and sensor nodes are often deployed in harsh environments, we often have to customize robust hardware to ensure the stable running of the system. Another issue is software, including highef- ficiency operating systems, communication protocols (time synchronization, media access control, routing, etc.) and application related techniques, such as localization and data query.Forfurther information, readers can refer to a survey of WSN technology (Akyildiz etal., 2002). 18 2 The Basis of Digital Technologies for Cultural Heritage Preservation

Although WSN has advantages in data transmission in heritage site, its limited communication range makes it unsuitable for long distance data communications. Therefore, long range communication technologies need to beemployed.Fordata relaying within severalkilometres, digitaldata transceivers such as the HAC-LM 433 MHz data transceiver (HAC Com- pany, 2009) are very attractive. It has a high data transmission power, and its single hop communication range can reach even more than 1 km. By em- ploying some simple data transmission control mechanisms such as polling, we can easilyorganize data transceivers into a data relaying network covering several kilometers. For relaying data in a larger area, a mobile or satellite net- work might bemoresuitable. We can access these networksinmostplaces in the world, and less effortisrequired to implement data transmission control mechanisms.

2.2 Basis of Information Analysis and Recognition

Cultural heritageinformation analysis and recognition refers to the analy- sis, extraction and processing of digital information from cultural heritage sites, including images, graphics, and other information. Through analysis and recognition of digital information, we can highlight the data character- istics of the original information. This section will focus mainly on heritage related analysis and recognition techniques.

2.2.1 Image Processing

Heritage digital information mainly includes images, videos, 3D models, and soon.3Dmodel processing was discussed in subsection 2.1.2. Here we will focus on the basic operations of heritageimage processing, including image enhancement, segmentation, and recognition. The primary purpose of enhancement is to process images to make them more suitable for specific applications compared to the original; the purpose of segmentation is to divide heritage images into meaningful region so as to facilitate understanding of the images; while image recognition is the major component of image understanding.

• ImageEnhancement

Image enhancement can be divided into two categories: spatial domain method and frequency domain method (Ruan, 2005). Spatial domain refers to the image plane itself and such methods are based on the direct processing of the image’s pixels. Frequency processing is based on the Fourier transform. Image enhancement based on gray scale transformation is the simplest of all imageenhancement technologies. Let r and s be the pixel values before and 2.2 Basis of Information Analysis and Recognition 19 after processing respectively, then the image enhancement based on gray scale transformation can be expressed as:

s = T (r). (2.1)

Here, the goal of image enhancement is to find the mappingfunction T ,from r to s. Common enhancement functions include the linear function (direct ratio and inverse ratio), the logarithmic function (logarithm and exponen- tial transformation), and the power function (the nth power and nth root transformation). Another commonly used spatial domain method for image enhancement is histogram processing. The histogram is the basis for a variety of spatial domain processing technologies. The histogram of adigital image with a gray level scope of [0, L−1] is the discrete function h(k)=r/n,in which r is the number of pixels with the gray level k, and n is the total num- ber of pixels. Histogram equalization and histogram matching are two basic but very useful image enhancement algorithms. Other spatial domain-based image enhancements include enhancement using arithmetic/logic operations, median filter smoothness, and space filtering. Frequency-based imageenhancement is usually based on theFourier transform. It is implemented through frequency domain filtering, the pro- cess of which is as follows: first, the input image is pre-processed, then the Fourier transform of the imageiscarriedoutandafilteringfunction is used. Finally, the inverse fourier transform of the image is computed and some post processing is completed (Fig. 2.3).

Fig. 2.3. Image enhancement based on frequency domain filtering

The key to frequency domain enhancement is to find a suitable filter. Cur- rently, filters can be divided into smooth frequency filters, frequency domain sharpening filters, and homomorphic filters (Ruan, 2005). In heritage preservation, image enhancement can not only improve im- age quality for the convenience of post-processing, but also can be used for heritage restoration, especially for paintings. For example, Pei et al. (2004) have used mixed color contrast enhancement of color images and a texture synthesis algorithm for the restoration of ancient Chinese paintings.

• Image Segmentation

Image segmentation (Sonka et al., 1999) is an important step after image en- hancement and before the analysis of the image data from cultural items. Its main goal is to divide an image into different objects or regions. For example, inthecourseof disease detection, the first task is to segment the imageinto 20 2 The Basis of Digital Technologies for Cultural Heritage Preservation a series of regions with diseases. Common image segmentation algorithms in- clude threshold-based, edge-based, region-based, and connectivity-based seg- mentation . Threshold-based segmentation is thesimplest method and relies mainly on the gray scale value of local pixels. Pixels whose gray scale values are within a certain range are set as foreground, while others are regarded as background. This method does not take spatial information into consideration and is therefore likely to generate fuzzy borders. Edge-based segmentation relies on edge detection algorithms to find ob- ject edges in images. Edges are recognized as discontinuities in gray scale, color, or texture of an image. The problem of this approach is that edges maybe erroneously detected when there are none and vice versa. This is caused by image noise or unwanted information from the image. Region-based segmentation should meet the conditions of complete seg- mentation, that is to say, there should benooverlapping area between seg- mented regions Ri (i = 1, 2, ··· , s) andanimage R is equivalent to the union of all the regions, as follows: s R = Ri,Ri ∩ Rj = φ, i = j. (2.2) i=1 Italso meets maximum regional consistency. There should benotwoor more regions which have the same regional consistency condition, as follows:

H(Ri) =True. (2.3)

H(Ri ∪ Rj) =False,i= j, i, j = 1, 2, ··· , s. (2.4) where Ri and Rj are theadjacent regions and H isthesameregion decision function. There are three basic region growing algorithms: region merging, region splitting, and regional merging and splitting. The images segmented by a region growing method often contain either too many or too few regions. To improve the effect of classification, a variety of post-processors are brought forward. Simple post-processors can reduce the number of small regions after the segmentation of images. Some more complex post-processing approaches can combine the information obtained by region growing algorithms with the information obtained by edge-based segmentation. Connectivity-based segmentation relies mainly on the active contour model.Itsidea is to provideaninitial outline expressedbyspline curves, and then to keep modifying edges until the specific conditions are met throughit- erations and the contraction and expansion of energy functions. This method can accurately extract the target region and is commonly used in medical image processing. The clustering method based on characteristic space (Comaniciu and Meer, 2002) and the graph cut algorithm (Shi and Malik, 2000) have also 2.2 Basis of Information Analysis and Recognition 21 been widelyapplied to image segmentation andhave achieved good results. These methods have therefore become a focus for imagesegmentation studies in recent years.

• Texture Recognition

Texture recognition is also commonlyused in culturalheritage preservation. For example, heritage diseases often have obvious textural characteristics. Fig. 2.4 shows a mural image with streptomyces infections.

Fig. 2.4. Typical streptomyces infections of murals

Texture is a surface or structural property of an object. There are many definitions of texture; the most common one is something composed of inter- related elements, so-called textons. Texture recognition has attracted people’s attention since the 1980s and has now become a focus of study. The key to texture recognition is feature extraction, whose main purpose is to map dif- ferences in texture structure to different values. There are a variety of texture feature extraction methods. Commonly used methods can be divided into two categories: structural methods and the statistical methods. Structural meth- ods regard textures as a result of the arrangement of textons in accordance with certain specific laws. The Fourier spectral analysis technique is used to determine thetextonsand their arrangement laws. However, this method is applicable only to textures with strong regularities. Statistical methods are more popular. They have a dominant position in texture analysis, and use various kinds of statistical features as texturefeatures. 22 2 The Basis of Digital Technologies for Cultural Heritage Preservation

Texture feature extraction methods include mainly: the space frequency- based method, the co-occurrence matrix-based method, the fractal model- based method, and the filter-based (such as Gabor filter (Fogel and Sagi, 1989)) method.

2.2.2 Intelligent Information Processing

Inarchaeological research,intelligence-intensive work is required,such as analysis, reasoning, classification, and recognition. Computers can assist ar- chaeologists with some of the routine, tedious work. Obviously, computers must have a certain degree of intelligence, e.g. principal component analysis, intelligent reasoning, and expert systems.

• Principal Component Analysis

Sometimes more than one different types of characteristics are involved in de- picting an archaeological object. The most important characteristics can be identified using PCA (Principal Component Analysis). PCA is an exploratory statistical analysis method which concentrates the information dispersed in a group of variables into other variables. When used to describe the inter- nal structure of data sets, PCA has the effect of reducing data dimensions. The main idea of the method is to reduce the original multiple variables to p variables according to the correlation between original variables, usinga dimension reducing technology. To be specific, every composite variable is a principal component, recorded as PRIN i (i=1,2,···,p),andeachprinci- pal component is expressed by the linear combination of original variables so that they reflect the information volume of the original variables and yet are uncorrelated with each other, to reduce information correlation between variables.Theamountof information extracted from the original variables by these p principal components is in a descending order. The information volume of each principal component PRIN i is measuredbyits variance. The contribution of principal component variance is equivalent to the eigenvalue λi of the correlation matrix of original variables, and the combination coeffi- cient of each principal component is the eigenvector ti corresponding to the p eigenvalue λi. The contribution rate of variance is expressed as fi = λi/ λj; j=1 the larger fi,themoreimportantthe corresponding principal component will be. One of the aims of PCA is to reduce the dimension of eigenvectors. In practical applications, it is enough to use the preceding m components with m contributionrate sum fj  85%. In addition, each principal component j=1 has a special meaning, representing the characteristics of a particular aspect of the object. If someone is concerned about only one aspect of the object, they can arrange the order according to this principal component and then 2.2 Basis of Information Analysis and Recognition 23 obtain the required information.

• Fuzzy Reasoning and Expert Systems

Archaeological, historical or other related knowledge are very important in identifying, authenticating and analyzing ancient objects. A system that can simulate the performance of experts, will be convenient and helpful. Expert System (ES) is a knowledge-based computer program which can provide ex- pert level knowledge to specific problems. It has already been widely used in various fields, providing strong support to computer-aided decision-making. ThecoreofESisitsknowledge base, and reasoning is based on the knowl- edge. Knowledge is actually a rule, that is, if condition X is satisfied, then Y will take place. However, this kind of ideal reasoning rule actually does not exist; for example, no one can be sure that it will not rain today even though there is not a cloud in the sky now. Therefore, it happens frequently that the knowledge base is unable to process this kind of uncertain rule. In addition, as knowledge accumulates in the knowledge base, contradictory knowledge will also appear. For example, although it is known that if condition X issatis- fied, then Y will takeplace, we may encounter a case in which Y takes place with condition X not being satisfied. Shall we remove the original knowledge or not? If yes, then the knowledge in the knowledge base will be reduced; if no,itmeansthatthiscaseisofnousefortheknowledgebaseatall.When the knowledgeinthebaseof a common expert system accumulates to a cer- tain degree, new knowledge may be troublesome because of its conflict with old knowledge. To solve this problem, we can rely only on fuzzy processing technology, which requires us to record not onlytherules but also the correct probability of this rule. In this way,thesystem can deal with uncertain or conflicting rules. In mathematical language, the fuzzy concept is fuzzy sets. The basic idea offuzzy sets is to make the absolute membership relation in the classical sets flexible. In brief, elements in sets are no longer confined to 0 or 1; they can be any number from 0 to 1. For example, the weather forecast generally used to state whether it would rain or not thenextday. But now it has been changed to the rainfall probability forecast, which is in fact more accurate, with higher probability indicating greater rainfall possibility. Assuming that the probabilities of rain on any two days are respectively 90% and 60%,and the weather forecast says only that it will rain, althoughtheforecast is not ambiguous, the information we have acquired is in fact reduced, because we cannot identify the possibility clearly. Fuzzy reasoning produces results not by judging whether an element belongs to a set but by offering the possibility of an element belonging to a set. Generally, we use Artificial Neural Networks (ANNs) to proceed fuzzy reasoning. 24 2 The Basis of Digital Technologies for Cultural Heritage Preservation

• Synthesis Reasoning

Artistic re-creation is an important part of the development and utilization of cultural heritage information. Artistic re-creation refers to extracting the style characteristics of cultural heritage items and applying them to other objects so as to generate objects with similar style. For example, the pattern creation system can create patterns with Dunhuang stylethrough learning. Intelligent reasoning is the technological basis for re-creation. Synthesis reasoning (Pan, 1996) is an important branch of intelligent rea- soning. Artificial intelligence generally solves problems by three major meth- ods: reasoning, exploration and constraint satisfaction. Traditional reasoning refers to the thinkingform which aims to make a new judgment based on one or several existing judgments. It includes deductive reasoning, inductive reasoning and analogical reasoning. As the traditional reasoning of AI schol- ars can no longer meet new needs following the development of AI, the logic analysis school has developed fuzzy reasoning and non-monotonic reasoning, while the knowledge school has developed rule-based reasoning, semantic net- work reasoning, and case-based reasoning. As a result, the scope of reasoning has greatly broadened. Meanwhile, research on graphics and images also has led to the development of spatial reasoning, geometric reasoning and visual reasoning.Infact, current reasoning is no longer the reasoning of the tradi- tional sense. What it denotes is a form of thinking or a method for obtaining a new thing from one or a few things through certain rules. Synthesis rea- soning is generated in such a situation. Synthesis reasoning consists of three elements: synthesis source S, synthesis constraint C, and synthesis result R. Definition 1 Synthesis source (abbreviated as S):Asynthesissourceis the expression of information with the following structures:

S = {P, M, F}. (2.5) where P is the set of components, P = (P1,PP2, ···,PPn); it shows that a source is composed of n parts. M is the structure; it is a method that describes how n parts comprise an entity, source S. F is the source influence field; it describes the distribution of influencing force in the synthesis reasoning. F is composed of two parts: F = {FP , FM }, FP isthe set of fieldfunctionsof one component. Every element of FP describes the field distribution of the influence force of the corresponding components in the synthesis reasoning; FP =(FPP1, FPP2,···, FPPn). FM is the field function of a structure; it describes the distribution of the influence force of a source structure in the synthesis reasoning. Definition 2 Synthesis space (abbreviated as SS) is composed of the sources through superposition in synthesis reasoning. We can record the syn- thesis space which is composed of m sources and that has n parts as: 2.3 Basis of Digital Exhibition and Interaction 25 m n (FPPij,PPij,FMMjMj)(2.6) j=0 i=0

Eachlocation in thesynthesis space is a potential synthesis result. We can see that the synthesis space is a complex space. Consider a syn- thesis space which is composed of m sources that have n parts; a part that has m sources constitutes an m-dimensional space (which can be called a part space, recorded as PS) constructed by their respective field functions. In this space, we can obtain a new component in accordance with the require- ments of synthesis requirement C. The new componentisthepart which corresponds to the synthesis result. Similarly, structure M with m sources also constitutes an m-dimensional space, known as the structure space. In this way,thesynthesis space is actually the superposition of n part spaces and one structure space; it is the n+1 superposition of m-dimensional space. Definition 3 Synthesis reasoning is a reasoning method which constitutes the synthesis space SS by synthesis source S and obtains synthesis result R by positioning in the space according to thesynthesis requirement C.

2.3 Basis of Digital Exhibition and Interaction

Digital exhibition and interaction refers to utilizing digital techniques to dis- play to the public as much as possible of the heritage information acquired or restoreddigitally, meanwhileencouraging visitors to participate in the display scenes and exchangeinformation with the system. Digital exhibition and interaction of cultural heritage items relies mainly on animation, rendering, stereo displaying, and natural interaction.

2.3.1 Animation

Animation plays an important role in displaying vividly the implicit value of cultural heritage items, including classical allusions and manufacturing pro- cesses. Animation can be divided into two forms, 2D and 3D animation.

• 2D Animation

Traditional 2D animation relied mainly on hand drawing. With the con- tinuous development of computer technology, traditional hand drawing was gradually replaced by computer, changingfrom colorizing with pigment to non-pigment colorizing, from capturingframe by frame by cameras to arbi- trary settings of virtual cameras, from films to computerized automatic ren- dering and imaging. Adhering to traditional animation operation processes, computer 2D animation generation has resolved many problems that could not be solved by traditional methods. It has not only improved the ability to 26 2 The Basis of Digital Technologies for Cultural Heritage Preservation handle the large-scale integrated generation of series, but also has improved animation generation efficiency. Traditional key frame-based animation requires artists to have rich ex- perience and superb painting skills. To speed up animation generation, re- searchers have proposed many new and effective computer-aided methods, differing from the traditional art design and generation processes. These methods can be divided into the following categories (Watt and Watt, 1991; Jin etal., 1997): (1) Video stream based methods. A video stream not only provides visual information including color, texture and contour, but also shows how the visual information changes with time, which can offer references for the gen- eration of animation in various aspects. The method ofgenerating animation based on video streams relies on the ideas of imitation or simulation and can be divided into the following three parts: a) Convert the contents of video streams automaticallyintoimage sequences with cartoon styles and visual effects. b) Extract some useful reference information from video streams for the generation of animation. c) The method of facial expression animation driven by performance. (2) Traditional generation process-like method. During the drawing pro- cess, a very important reason why animators can convert hand-drawn char- acters into livelyanimationisthat they have the3Ddescriptions in their minds. However, in current 2D animation generation, because of the lack of adequate 3D information, the computer can play the role of an interactive assistant for animators to only a certain extent. Therefore, to express and utilize 3D information in the animators’ minds may be an effective approach for computer-aided artistic animation. (3) Animation reuse method. This method attempts to extract the move- ments and drawing skills from existing animations, and to turn them sys- tematically into a variety of templates with various cartoon-styles, and apply them directly to the generation of new animations. (4) 3D animation-like method. This method aims to learn from the 3D ani- mation generation processes and methods. Referring to the non-photorealistic rendering methods, it improves thetraditional 2D animation generation methods and processes to form new types of artistic animation modeling and drawing methods, which integrate modeling, rendering, and movement generation techniques.

• 3D Animation

3D animation is entirely dependent on computer graphics. 3D animation can be classified into several types, including key frame, deformed objects, processing, joint, human, and physically based animation (Parent, 2002). The concept of key frame originated from traditional cartoon creation. Skilled animators design key images of cartoons or the so-called key frames, 2.3 Basis of Digital Exhibition and Interaction 27 then other animators design the tween frames. In 3D computer animation, the tween frames are generated by computers. Interpolation takes the place of animators who design tween frames, and all the parameters affecting images become key frame parameters, such as location, rotation angle and texture. Key frame technology is the most basic technique in computer animation. Another method is spline-driven animation, by which users appoint the move- ment trace by interaction. A notable characteristic of traditional animation is that it bestows each character with personality and exaggerates the effect by shape deformation. Computer animation researchers have proposed many shape deformation methods, most of which are closely related to object expression; for example, one of the methods transforms objects by moving or controlling the object vertex. Major deformation methods include the object expression relevant method, object expression irrelevant method, constraints based method, and the axis deformation method. Procedural animation refers to animation in which objects are controlled by a process. Procedural animation often involves the deformation of objects, but it is different from the animation of flexible objects mentioned above. In the animation offlexible objects, object deformation is arbitrary and can be freely controlled by the animators. In procedural animation, object deforma- tion follows certain mathematical rules. In 3D computer animation, the human bodyisasubject in which re- searchers are always interested. Forward kinematics and inverse kinematics are two major approaches for human joint animation. The former uses joint rotation angles to set the key frames and then obtain the positions of re- lated body parts. The inverse kinematics method designates the position of terminal joints and then lets the computer calculate the position of each intermediate joint automatically. As a new animation technology, physically based animation began to de- velop in the late 1980s (Kim et al., 2002). In recent years, it has become a 3D modeling and movement simulation technology with potential advantages in graphics. Although it is much more complex than traditional animation, it can simulate various natural phenomena vividly. It has taken real attributes of objects into consideration, such as their weight, rotation inertia moment, flexibility, and friction. It has also applied the kinetics principle to automat- ically generate object movement.

2.3.2 Real-time Rendering

To implement digital exhibition of cultural heritages, firstly we need to do some acquisition and modeling tasks, and then we output the results of ac- quisition and modeling by display. The process of output generation is called rendering. The goal of rendering is to render true-to-life images using com- puter, namelyrealism. Besides realism, real-time properties should also be ensured, otherwise we will have a great sense of delay during exhibition. 28 2 The Basis of Digital Technologies for Cultural Heritage Preservation

Since Sutherland, the founder of computer graphics, published The Ulti- mate Display (Sutherland, 1965), rendering technology has developed rapidly. Usually, rendering is implemented by a graphics rendering pipeline, which consists of three stages: application, geometry, and rasterizer (Tomas et al., 2008)(Fig. 2.5).

Three typical stages of a graphics rendering pipeline

The application stage is implemented in software. Developers can have control of this stage. The main task of this stageistogenerate triangle sets for the geometry stage. Asthegeometry and rasterizer stages following the application stage are usually implemented by hardware and are very difficult for users to control, minimizing the number of triangles in the application stage can improve the efficiency of the system. Culling (Zhang et al., 1997) and Level of detail (LoD)(Puppo etal., 1997) are both techniques belonging to the application stage. Culling is an acceleration technique that cuts the triangles outside view frustum or will not be shown in the viewport. LoD generates a set of models of different complexity and the viewing distance will determine which one is used. Scene management driven by user interactions is also implementedduring theapplication stage. The geometry stage is a very important stageinthe graphics rendering pipeline, which is responsible for dealing with polygon and vertex. The stage can be divided into several stages including model and view transform, vertex shading, projection, clipping, and screen mapping (Tomas et al., 2008).The goal of the model and view transform stage is to transform different objects into thesameworld coordinate system to achieve thetranslation, scaling, rotation and other operations of target and meanwhile to adjust the position ofsight and view. Model and viewpoint transformations are both expressed bya4×4 matrix. The vertex shading stage is responsible for light effect in the vertex levelby computing ashading equation at various object points. The aim of the projection stageistotransform 3D model into 2D space. There are two types of projection transforms, namely orthographic projection and perspective projection. The projection transform is also expressed by a 4×4 matrix. The purpose of the clipping stage is to send the graphic elements existing inside the view port to the rasterizer stage and to clip the graphic elements outsidethe view port. In the screen mapping stage, we need to map theclippedgraphic elements to the screen. Therelationships between these stages may not be the same in all pipeline implementations. A simple exampleisshown in Fig. 2.6. Thetask that needstobe completed in the rasterizer stageistodistribute colors to pixels based on the given vertex, color and texture coordinates 2.3 Basis of Digital Exhibition and Interaction 29

Fig. 2.6. Subdivided pipelines in the geometry stage after the geometry stage. Information on each pixel is stored in the color buffer, which contains colors including three channels, red, green and blue. The existing graphic system generally adopts the double-buffer mechanism; that is, the screen content is rendered in a back-buffer in an alternate manner. The advantage of this method is that it can solve the problem of operating the contents displayed on screen when theobserver is operating andhandling the graphicelementssoastoavoidtheflickering of the screen. Rasterizer is also usually completed in hardware. Generally, developers do not need to care about the implementation de- tails of the above geometry stage, but only need to call the graphic library to complete rendering tasks. At present, OpenGL (Khronos Group, 2009) and Direct3D (Microsoft, 2009) are the most popular 3D graphics libraries. OpenGL is the product ofSGI, issued in the form of head files and a static link library. It supports functions including viewpoint calculation, lighting, mix- ing, anti-aliasing, texture mapping, frame buffer, and other functions and sup- port for network operations. Direct3D is a component of DirectX developed by Microsoft. It is based on Microsoft’s COM (Component Object Model) technology and has a good object-oriented feature. In respect of function, it supports lighting,vertexblending,vertexshader, bump map, and so on. It is a library based on Microsoft’s Windows platform, and therefore cannot be applied in other operating systems.

2.3.3 Stereo Display

In culturalheritage digital exhibitions, it is required to output therendering results vividly to the visitors. From MIT’s Whirlwind-I CRT (Cathode Ray Tube) monitor in 1950 up to present, display technology has undergone sev- eral generations of development and updates, and now provides good support for the display and output of cultural heritageinformation. Computers can allow visitors to sense the in-depth 3D information of objects through constructing stereo image pairs andletting them immerse themselves in the virtual environment. The technology for constructing stereo image pairs is called stereo display (Marr and Poggio, 1979). The reason why peoplecanhave such a stereo sense is that the image pair constructedby computers has built-in in-depth 3D information. As Fig. 2.7(a) shows, when animage pair shown by the display is non-stereo, for every point that users can see with both their left and right eyes, the sights of both of the user’s eyes intersect at this point; therefore, this imagehasinformation of the same depth and does not produce any stereo sense. However, as Fig. 2.7(b) shows, 30 2 The Basis of Digital Technologies for Cultural Heritage Preservation when a user’s left eye sees the point Q1, and theright eye sees the point Q2; the two sights intersect at the point Q,giving the user a spatial depth sense, which leads to stereo vision.

Fig. 2.7. Principles of non-stereo and stereo display. (a) Plane display with no depth information; (b) Stereo display

There are two means of displaying the 3D image pair. One is the time paralleldisplay, whichdisplays two images at thesametimeand makes each eye see different images with the help of devices. The other is the time mul- tiplexed display which displays alternately at a certain frequency and makes each eye see theimage in turn. To see 3D images, the user needstowear glasses which, based on their respective working principles, can be divided into active glasses and passive glasses. The frame of active glasses is equipped withbatteries and the lenses are controlledbya liquid crystal modulator. The glasses can control the on-off status (the transparency or opacity of lens) of the liquid crystal modulator according to the infrared control signals that are sent based on the image switchingfrequency of two eyes, and implements the time multiplexed display. Passive stereo glasses are usually designed based on the polarization principle. The glasses consist of two orthogonal polariz- ingfilters, which let through only the polarized light with one direction. The images displayed by devices will respectively become leftandright polarized light with the help of the filters. As a result, the images are shown to both eyes at the same time, but each eye sees a different image. Currently,stereodisplays include mainly the desktop stereo monitor dis- play system, the Head Mounted Display (HMD) and the multi-projector display systems. The desktop stereo display system is composed of desktop stereo monitors and stereo glasses. The HMD is a display device worn on the head. The main components of this display are two small LCD screens. It is placed in front of the eyes and mounted on the head. Based on the principles of generating stereo image pairs described above, it displays stereo images on its display and gives the users a stereo sense. Of course, there are a lot of problems that need to be taken into consideration in practical use; for ex- ample, how to make the eyes not feel exhausted, how to make the user have a broad perspective, and so on. Themulti-projector display system is also an important stereo display device. It has been widely used in the display of 2.3 Basis of Digital Exhibition and Interaction 31 cultural heritage in recent years. Such devices are built mainly by the manu- facturers of projectors with high precision and high brightness. CAVE (Cave Automatic Virtual Environment)(Carolina etal., 1993) is the first typical system of such devices. This system is composed of acuboidof size 10×10×9 feet. 3D image pairs with 1024×768 resolution are projected onto the wall and floor of the cuboid by the projector. Users are allowed to walk freely in CAVE and sense the stereo world with the help of stereo glasses.

2.3.4 Natural Interaction

The exhibition of cultural heritage information not only needs to offer visitors visual information through rendering and display technology, but also needs toallow visitors to interact with thesystem, allow them to personalize the content of the exhibition and enable them to enjoy the scenes freely. Usu- ally, in non-immersion heritageexhibitions, we use a keyboard and mouse, with which people are familiar, to interact with the system. In the immersion environment, we use natural interaction devices instead. Natural interaction refers to a situation in which people express themselves and act freely as if they were in the real world, such as the use of hand gestures, body gestures, eyesight, and voice to make interactions. It also refers to determining the location, viewpoint and interest point of visitors by tracking their positions. In the following, we will introduce some natural interaction methods that are widelyused in culturalheritageexhibitions.

• Tracking

The goal of tracking is to detect the positions and directions of users, such as view parameters andhand pose parameters. Based on these data, the system makes corresponding adjustments to the display content, for example, adjusting the stereo display content according to each visitor’s viewpoint and line of sight. The position tracking system usually has 6 degrees of freedom (DoF), translation on the X, Y and Z axis, and the rotations around them. Most position tracking devices have the following performance parameters: accu- racy, resolution, responsiveness, and robustness. Accuracy and resolution de- termine the ability of the tracker. Accuracy refers to the tracker’s output correctness. Resolution refers to the tracker’s ability to detect minimum vari- ations. Responsiveness is the tracker’s temporal performance parameter, in- cluding the sampling rate, the update rate and the latency. Robustness refers tothetracker’sability to avoid mistakes in unusual conditions. During the digital exhibition of cultural heritageinformation, the most important per- formance parameters that need to be considered are accuracy and latency, because accuracy has a direct impact on correction of the visiting process, and latency may directly influence the real time performance of interaction during the visiting process. Low accuracy may lead to a failure to interpret 32 2 The Basis of Digital Technologies for Cultural Heritage Preservation users’ actions during the interaction, while highlatency may cause sickness in the interaction process. At present, main tracking technology includes electromagnetic, inertial, ultrasonic, mechanical device, and visual tracking (Rolland et al., 2000).

• Gesture Interaction

Gesture is an important means by which human beings express themselves. It is an important research area in human-computer interaction, and is widely used in culturalheritageexhibitions. Data gloves are most widelyused in gesture interaction, for example, the sensinggloves created by VPL Company in 1987 (Sturman and Zeltzer, 1994). The gloves are made of lycra, and use optical fibers as sensors, with each joint of every finger equipped with a light ring to measure the angle offinger joints. Each finger of adataglove is usually equipped with two sensors to measure the bending angle of finger joints, which will be further used to identify gestures. Usually, data gloves and position trackingdevices are combined to work, because data gloves can read only the joint gestures of hands, while the specific position of the hand needs to be determined by the tracker. There is also gesture interaction technology based on markers, which utilizes cameras and image recognition techniques to determine gestures. The development of image processing technologyin recent years has also promoted the progress of natural gesture recognition technology which needsnoother devices but human hands. Gaze recognition and voice recognition are also important human-computer interaction methods that have laid a technological basis for the interactions in cultural heritage exhibitions.

2.4 Summary

Digital technology is widely used in various aspects of cultural heritage preser- vation, including archaeological excavations, archaeological research,archive management, conservation, exhibitions, as well as utilization. This chapter focuses mainly on the technological basis of special techniques, systems, and devices for the digital preservation of cultural heritageinformation, which is fundamental to Chapter 3 to 7.

References

3D CaMega (2009) CS-400. http://www.3dcamega.com/cpjs show.asp?id=1 (in Chinese). Accessed 10 Sept 2009 Abhay S (2003) Understanding Color Management. Cengage Learning, Florence, KY Akyildiz IF, Su W, Sankarasubramaniam Y, et al. (2002) Wireless sensor net- works: a survey. Computer Networks 38(4):393-422 References 33

Better Light (2007) Better light SuperModels. http://www.betterlight.com/sup- erModels.html. Accessed 20 Aug 2007 Carolina CN, Daniel JS, Thomas AD (1993) The design and implementation of the cave. In: ProceedingsofACM SIGGRAPH’93, Anaheim, CA,USA, 135-142 Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5):603-619 Fogel I, Sagi D (1989) Gabor filters as texture discriminator. Biological Cyber- netics 61:103-113 GE Corporation (2006) Telaire 6004 CO module. http://www.gesensing.com/do- wnloads/datasheets/920 356a.pdf. Accessed 10 Sept 2009 HAC Company (2009) HAC-LM data RF module. http://www.haccom.com/new- Ebiz1/EbizPortalFG/portal/html/ProductInfoExhib-it.html?ProductInfoExhib- it ProductID=c373e91521f83e108f7b916fde2381ed. Accessed 10 Sept 2009 Jeff S (2003) Professional Digital Portrait Photography. Amherst Media, Inc., New York, USA Jin XG, Bao HZ, Peng QS (1997) A survey of computer animation. Journal of Software 8(4):241-251 (in Chinese) Khronos Group (2009) OpenGL–the industry standard for high performance graphics. http://www.opengl.org. Accessed 10 Sept 2009 Kim YJ, Otaduy MA, Lin MC, et al. (2002) Fast penetration depth com- putation for physically-based animation. In: Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation.San Antonio, Texas, USA, 23-31 Marr D, Poggio T (1979) A computational theory of human stereo vision. In: Proceedings of the Royal Society of London, Series B, Biological Sciences 204(1156):301-328 Microsoft (2009) Microsoft DirectX. http://www.microsoft.com/australia/wind- ows/directx. Accessed 10 Sept 2009 Rolland JP, Davis L, Baillot Y (2000) A survey of tracking technology for virtual environments. Fundamentals of Wearable Computers and Augmented Reality, Lawrence Erlbaum Associates, Philadelphia Pan YH (1996) The synthesis reasoning. Pattern Recognition and Artificial Intel- ligence 9(3):201-208 (in Chinese) Parent R (2002) Computer Animation: Algorithms and Techniques. Morgan Kauf- mann, USA Pei SC, Zeng YC, Chang CH (2004) Virtual restoration of ancient Chinese paint- ings using color contrast enhancement and lacuna texture synthesis. IEEE Transactions on Image Processing 13(3):416-429 Phase One (2006) P45+. http://www.phaseone.com/Content/p1digitalbacks/P- plusseries/Pplus/P45+.aspx. Accessed 10 Sept 2009 Puppo E, Nazionale C, Scopigno R, et al. (1997) Simplification, LOD and multireso-lution-principles and applications. In: Eurographics’97, Tutorial Notes, Budapest, Hungary Ruan QJ (2005) Digital Image Processing Using Matlab. Publishing House of Electronics Industry, Beijing (in Chinese) 34 2 The Basis of Digital Technologies for Cultural Heritage Preservation

Sensirion Company (2009) SHT75 - Digital humidity sensor (RH&T). http://ww w.sensirion.ch/en/01 humidity sensors/06 humidity sensor sht75.htm. Accessed 10 Sept 2009 Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Transac- tions on Pattern Analysis and Machine Intelligence 22:888-905 Sonka M, Hlavac V, Boyle R (1999) Image Processing, Analysis, and Machine Vision: Second Edition. Thomson Learning,USA Sasson S (2007) We Had No Idea. http://stevesasson.pluggedin.kodak.com/. Ac- cessed 10 Sept 2009 Sutherland I (1965) The Ultimate Display. In: Proceedings of IFIP Congress, New York, USA, 506-508 Sturman DJ, Zeltzer D (1994) A survey of glove-based input. IEEE Computer Graphics and Applciations 14:30-39 Tomas AM, Eric H, Naty H (2008) Real-time Rendering, 2nd Edition. A K Peters Ltd., Wellesley Massachusetts, USA Trimble (2005) Trimble GX 3D Scanner for Spatial Imaging-Accurate Terrestrial Positioning Data for the Geospatial Industry. http://www.trimble.com/trimbleg- x.shtml. Accessed 10 Sept 2009 Watt A, Watt M (1991) Advanced Animation and Rendering Techniques. ACM New York, USA William E, Ramon B (1989) Handbook of Jig and Fixture Design (2nd Edition). Society of Manufacturing Engineers, Michigan, USA Zhang H, Manocha D, Hudson T, etal. (1997) Visibility culling using hierarchical occlusion maps. In: Proceedings of ACM SIGGRAPH’97, Los Angeles, CA, USA, 77-88 3 Digitalization of Cultural Heritage

A variety of cultural heritages are undergoing natural and man-made de- struction, thus some important information is being lost. For example, an enormous amount of information has been lost from excavations despite the great amount of precious relics and cultural information gained. Iwanami Shoten, a famous Japanese archaeologist, once claimed that an archaeologi- cal excavation report is in fact “a Medical Certificate of Death”. Cultural heritagedigitalization includes three important objectives: to prevent the loss of information about cultural relics; to provide original data for archaeological conservation, research, exhibition, and utilization; and to record the current status of cultural heritage items as much as possible for potential future use. With various new acquisition technologies and devices continually being developed, it is now possible to acquire heritage information and record it precisely as digital information in computers. For example, we can use digital photography to capture texture and color, 3D scanning to record shape, and radar and X-ray devices to record internal structure. Digital signals consisting of “0” and “1” can be duplicated numerous times without losses and kept for a long time in magnetic disks, CDs, or other storage devices. This chapter focuses on the digitalized acquisition and recording of in- formation from ancient heritage items, including their distribution, texture, shape, color, etc. Section 3.1 introduces various digital techniques used in archaeological excavations for acquiring information such as the distribution of cultural layers and heritage items, as well as the structure and appear- ance of unearthed relics. Section 3.2 discusses the digitalized acquisition of information about sculptures and artifacts preserved in museums, especially information about their, 3D shape, surface color and texture. Section 3.3 de- scribes information acquisition from largearchaeological sites, focusing on the recording of information concerning their structural layout, 3D shape and surface color. Section 3.4 covers the digitalized acquisition of informa- 36 3 Digitalization ofCultural Heritage tion about large-scale paintings and murals, focusing on the recording of high accuracy imageinformation.

3.1 Information Acquisition from Archaeological Excavation Sites

Archaeological excavation aims to explore human history and extract details of the development of human society. It helps to understand the origin and development of the human civilization. During excavations, archaeologists can gradually obtain information about cultural heritage items buried underground, such as the cultural layer, re- mains, relics and their distribution and locations. The processing of archae- ological excavation information plays an important role in archaeological re- search. However, the process of archaeological excavation itself may cause damage to cultural heritage items and the excavated sites are always unre- coverable. Although a variety of methods such as photography, measurement, and mapping have been applied to record excavation information, they are not fully dimensional and have inadequate content. Besides, because of varia- tion in knowledge among individuals, important information might be omit- ted. Thus, part of the information from cultural heritage sites or unearthed relics may disappear forever. Therefore, it is important to develop modeling and expression techniques to preserve the process information accumulated during the excavation of archaeological sites as much as possible.

3.1.1 Preventing Loss of Information from Archaeological Sites

Archaeology has become a multidisciplinary science including humanity, sci- ence, and modern technology. In 1816, Christian Thomson, a scholar from Denmark,published A Guide for Archaeology in North,symbolizing the es- tablishment of the pre-historical three-age system (Stone Age, Bronze Age, and Iron Age). In 1871, Heinrich Schilemann, a German archaeologist, used the stratigraphy method for the first time to excavate the city of Hisar- lik on the Turkish Peninsula in Asia Minor. In 1921, Andersson, a Swedish scholar, presided over the excavation of the site of Yangshao villageinYinchi County, Henan, China, initiating the pre-historical archaeological stratigra- phy of China. Presently, the field of archaeological excavation and research ismaturinggradually. Asarchaeological excavation proceeds, the information is recorded, using photos, hand drawings, and charts. However, at the same time, pre-existing information from each site may be lost. A new digitalized method with more detailsanddimensions should beadopted to make recordsonsite,and to avoid the loss of information from archaeological excavation. 3.1 Information Acquisition from Archaeological Excavation Sites 37

First applied in the 1970s, ground penetrating radarisnowservesasan efficient technique for acquiring more accurate digitalized information dur- ing excavation with its resolution and penetration capability being improved constantly. Meanwhile, 3D information acquisition have also undergone great development with precision and speed being improved. The requirements for high precision acquisition of surface and shape information during excavation can now be met. Similarly, digital photography and videography have also provided feasible solutions to the acquisition of still and motion image infor- mation during excavation. 3D reconstruction technology based on multi-view photogrammetry can also help to acquiring surface shape information from a site. With the development of virtual reality and computer graphics, and full dimensional information modeling supported by 3D site model represen- tation, 3D animation and 3D model deformation will come into use, aimed at recording the whole excavation process. The information that needs to be recorded during excavation can be divided into the following three categories: (1) Cultural Layer. Cultural layers are formed by accumulated soil with different properties, such as color and looseness. These different layers, as well as the relics buried within them, are always of different dates, so the cultural layer is important for dating the relics. However, by the end of an archaeological excavation, the cultural layers will have vanished completely. Therefore, recording the information of cultural layers is an important task during excavation. (2) The Unearthing Location of Relics. The unearthing location of relics refers to the position and orientation of relics buried underground. The rela- tionships between relics can generally bejudged through their distribution. Hence, recording the information concerning the location of unearthed relics will have a great influence on subsequent research. (3) Dynamic Information during Excavation. During excavation, informa- tion from archaeological sites, such as each layer’s shape as well as the color and texture of unearthed relics, is always changing.Thechanging informa- tion is also of great value for subsequent research, therefore it needs to be recorded continuously during excavation.

3.1.2 Process and Technical Framework of Information Acquisition from Archaeological Excavation Sites

• TraditionalExcavationProcesses

Traditional excavation processes for archaeological sites include: square ar- rangement, excavation, cleaning, and backfilling. Every step will generate a series of information about cultural items. In this section, we focus on how to acquire and record digitalized information generated at each step (Fig. 3.1). (1) Arrange Squares. The archaeological site area is divided into many equivalent squares when excavated. Such method is called square arrange- ment. A square is generally made up of the main body, the girder, and the 38 3 Digitalization ofCultural Heritage

Fig. 3.1. Information acquisition process for an archaeological excavation site key pillar. A girder is usually orientated in the direction of a magnetic nee- dle, for the convenience of measurement and exploration. These squares are called excavation units. Normally, we choose the southwest corner as the co- ordinate origin where latitude and altitude are measured using GPS. All base points, base lines, directions and specification information should be recorded simultaneously in digital format as coordinate references for the acquisition of process information during the archaeological excavation of follow-up squares. (2) Excavate Cultural Layers. To make an excavation, we should first clear the surface soil. We digfrom the square’s edge till we reach a more ancient layer. When excavating an ancient layer, we should make a local scopeandthengodownwards.Whenwegetthebottomofalayer,weshould take into account the thickness of the next layer and try to avoid damages to the relics or the next layer. The traditional method can only gain data such as photos and section maps of girders, while the surface information of the whole cultural layer could not be recorded. By 3D scanning, we can make accurate recording of the surface information of each cultural layer and provide valuable data for further research. 3.1 Information Acquisition from Archaeological Excavation Sites 39

(3) Discover Relics. A variety of objects can be found during archaeo- logical excavation, such as man-made relics, skeletons of human beingsand animals, plant specimens, mineral dyestuffs, soil samples for pollen analysis, and so on. When picking up relics, we should pay special attention to their buried status such as whether they were placed upwards, downwards, or in- clined,whether they were placed together or separately, and whether they were piled up or juxtaposed. We should also note whether there are related relics around them. In this process, cameras can only capture single-view imageinformation and may cause confusion in judging the relative positions of relics later. Using digital technology, we can accurately record the posi- tions of relics, not only facilitating their excavation but also providing more information for future archaeological research. (4) Unearth and Clean Relics. We use different methods to deal with different unearthed relics. For relics whose attributes change rapidly when unearthed, we should acquire information from the excavation site immedi- ately. For those stable, we can bring them to labs to acquire high precision information after they have been cleaned. By collecting details, locations, and cultural layer distribution information, we can produce a holograph of relics buried underground. The whole archaeological excavation process can be turned from a non-reversible process into a “book”, which can be browsed arbitrarily with the assistance of dynamic information from the excavation. This will greatly support future archaeological research and potential utiliza- tion.

• Technical Framework for Archaeological Excavation

Based on the characteristics of each step in the excavation process and the information that needed in each step, we can build a technical framework for archaeological excavation (Fig. 3.2). Here we describe the acquisition of exploration information, relic location information, cultural layer shape in- formation, relic details, and information relation expression techniques. (1) Acquisition of Archaeological Exploration Information. Acquisition of archaeological exploration information refers to the acquisition of status infor- mation of cultural heritage items using RS, magnetic, electrical, radioactive, and nuclear magnetic resonance (NMR) technologies during the exploration phase of an archaeological investigation. To avoid damagetotheheritage items, we generally use non-contact approaches in exploration. (2) Acquisition of Relic Location Information. Acquisition of relic location information refers to the precise measurement and record of the unearthing lo- cation and orientation of relics according to the excavation origin in a reliable and detailed way so that the original distribution of relics can be represented after the excavation. The acquisition of information is generally realized by a variety of approaches such as the total station and photogrammetry. 40 3 Digitalization ofCultural Heritage

Fig. 3.2. Technical framework for information acquisition fromexcavationsites

(3) Acquisition of Cultural Layer Shape Information. Acquisition of cul- tural layer shape information refers to the precise measurement and record of the space, shape and position of cultural layers of different dates for the convenience of subsequent research. It can be realized by 3D scanning or multi-view photogrammetry. After archaeological excavations, we can recon- struct the general cultural layer shape based on the interpolation and fitting of a series of photos from the excavation. (4) Acquisition of Relic Detail Information. The acquisition of relic detail information refers to the acquisition and record of the surface shape, color, texture, internal structure, and buried ageof relics with high precision, using 3D scanning,digital photography, spectrometry, fluoroscopy, 14C dating,and so on, in order to record the overall relic details. (5) Expression of Relation Information. The expression of related infor- mation refers to the expression of all integrated information recorded during each process of an archaeological exploration and excavation in a complete model, covering all static and dynamic information generated. This could be convenient for further archaeological research and utilization. 3.1 Information Acquisition from Archaeological Excavation Sites 41

Theabove methods are supportedbytechnologies such as ground pene- trating radar, 3D scanning, X-ray, and so on. The relationships between them are denoted in Fig. 3.2.

3.1.3 Key Technologies for Information Acquisition from Archaeological Excavation Sites

Key technologies for information acquisition from archaeological excavation sites mainly include acquisition of cultural layer shape information, acquisi- tion of relic locations information, information relation expression for exca- vation process information, which will be detailed in the following contents. • Acquisition of Cultural Layer Shape Information

Stratigraphy and typology are two most important subjects involved in ar- chaeology; stratigraphy is closely related to the acquisition of information about the surface shape of cultural layers. It includes the study of layer sequences and spatial relationships between layers. In archaeological excava- tion, we try to excavate layer by layer. When a cultural layer appears, we always excavate along the surface of the layer. Therefore, we have the oppor- tunity to use digital devices to record the shape information of every cultural layer and thus provide sufficient evidence for the dating. 3D laser scanners are used to record surface information of each layer. There are two types of 3D laser scanners: the line laser scanner, based on triangulation measurement (Boehler and Marbs, 2002), and the point laser scanner, based on the time of flight (ToF) method. Point laser scanners are suitable for acquiring 3D information from large scale scenes. Phase difference based point laser scanners are able to measure distances withhigher precision, within the rangeof tens of metres, and are especially suitable for acquiring information from cultural layers unit by unit. The key steps of acquisition by laser scanning include the registration of scanning data and GPS data, registration and merging of multi-view scanning data, and the simplification and optimization of the surface polygon model of each layer. We can also acquire 3D shape information of cultural layers by multi-view photogrammetry. Firstly, we take pictures of the surface shape in different orientations, then calculate the camera’s parameters by recognizing the cor- responding feature points in different photos. Finally, we calculate the 3D space positions of the corresponding points based on the triangulation mea- surement method, and use the information obtained to rebuild the 3D shape ofthewholelayer. Camera parameters are important in acquiring the 3D information by multi-view photogrammetry. Firstly, we set some feature points in the ar- chaeological scene. For a photo of the scene, we set the camera projection T parameter matrix to be m[3×4], and define a vector l =(m1, m2, m3) , in T which mi = (mi1, mi2,mi3,mi4) ;meanwhile we construct a matrix P based 42 3 Digitalization ofCultural Heritage

T on the 3Dcoordinate Pi =(Xi,YYi,ZZi, 1) of a feature point in the photo and its pixel coordinate (ui,vi). Letting Pl have the minimum value m, we can obtain the camera projection parameter matrix of this photo. By taking several photos, we can reconstruct the 3D shape information data of the cul- tural layer surface. The formula for reconstructing the photo parameters of a cultural layer is as follows: ⎡ ⎤ T T P1 0 −u1P1 ⎢ T T ⎥ ⎡ ⎤ ⎢ 0 P1 −v1P1 ⎥ mT ⎢ ⎥ 1 P l ⎢ . . . ⎥ ⎣ mT ⎦ = ⎢ . . . ⎥ 2 (3.1) ⎣ T T ⎦ mT Pn 0 −unP1 3 T T 0 Pn −vnP1 Laser scanning is an active 3D information acquisition technique, while multi-view photogrammetry is a passive one. As active laser scanning method actively constructs features, which are easy to recognized, the error of the data is low. However, point by point scanning may take from several to dozens of minutes to acquire all the 3D information. What’s more, some laser scanners are not abletowork under sunlight. With the passive method of photogrammetry, however, we are not required to set feature points in the scene but only use the surface texture information in several photos to build apolar coordinate system and establish a corresponding relation. We then calculate the 3D space position of each corresponding point by triangu- lation measurement to finally obtain the 3D shape. Since this method does not require scanning every point in the scene, information acquisition can be completed in a shorter time. But the sampling rate and precision are not as good as with laser scanning,asthefeature points in photos with different views are difficult to identify automatically using algorithms and misjudg- ments may also occur. • Acquisition of Information on Relic Locations The 3D coordinates of the location of unearthed relics need to be recorded in archaeological excavations. In recent years, total stations have been ap- plied increasingly to identify the unearthed locations of relics quickly and accurately. As a precise electronic surveying instrument, a total station can record data automatically with advantages of high speed, highefficiency, and real time. It integrates optical, electrical, and mechanical technologies, and is capable of measuring horizontal angle, vertical angle, distance, and height differences. It is called total station because it can complete almost all mea- surement work. A total station’s error in measuring angles can be minimized to secondsand the error in measuring distance can be minimized to millime- ters. With the assistance of computers, a total station can put measurement data into computers directly and avoid errors of artificial reading and record- ing. Using a total station, we can realize WYSWYG (What You See is What You Get). The information of every feature point of the relics obtained by 3.1 Information Acquisition from Archaeological Excavation Sites 43 a total station can beshown on screen immediatelysothat we can see the distribution of everyfeature point in real-time. Compared with an electronic or optical theodolite, a total station equipped with more specialized components has more functions than other goniometers and distance meters, so it is more convenient. • Information Relation Expression for Excavation Process Infor- mation

A traditional line graph drawing, simplifying the real world to a flat surface, is unable to describe 3D information, so it is impossible to integrate all line graphs, written records, and video materialstosimulate and represent the whole excavation process. It is also impossible to gather experts to conduct remote investigation on site excavation process. To analyze cultural informa- tion contained in heritage sites revealed by excavation fieldwork, including the unearthinglocations, culturallayer distribution, precise unearthed posi- tion and 3D information of relics, we need a comprehensive and integrated method for organizing and expressing all information generated during the excavation. In this way, we can record all information discovered, and meet the requirements of virtual archaeological excavation, exhibition and infor- mation retrieval. The archaeological information acquired by digital devices can be transformed into digital form, and stored in databases after being edited and optimized. Information relation expression model for excavation process is shown in Fig. 3.3.

Fig. 3.3. Information relation expression model for excavation process

Inaculturalheritage database, various kinds of information may be re- latedtoeachotherinsomeway. Relics unearthed at the same site can be classified into different groups according to their cultural layer distribution, distribution region and relic categories. Those unearthed at different sites can be classified based on their date, category, shape, and color. A comprehen- sive relic information expression system with highly dynamic relationships is crucial to the application of relic information. 44 3 Digitalization ofCultural Heritage

3.1.4 Typical System for Information Acquisition from Archaeological Excavation Sites and Applications

Researchers at Columbia University,USA have used total stations, 3D laser scanners anddigital cameras in archaeological excavations. They recon- structed a precise 3D model of an archaeological site using scanned 3D mod- els and digital images, then constructed an enhanced model by fusing video, panoramic, and GIS data, and finally achieved multi-model expression and interaction with augmented reality (Allen et al., 2004). The workflow is shown in Fig.3.4.

Fig. 3.4. The site digitization workflow used by Columbia University

During the archaeological excavation of the Jinsha Site in Chengdu, China, our group used a self-developed site information acquisition equip- ment to acquire high-precision 3D excavation unit information. This system accurately recorded the shape and location of unearthed relics, the cultural layers, the excavation status, and so on. The acquisition precision of the whole unit reached 3 cm, while the unearthing location reached 0.5 cm and a single relic reached 0.2 cm. The results are showninFig.3.5.

3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts

Museum preserved sculptures and artifacts are not only an important part of museum exhibitions but also important for archaeological research. In- formation acquired from them will promote the diversification of exhibition forms, improve research work such as remote research on relics and the virtual restoration of relics. In this section, we focus on process, key technologies, and typical systems of information acquisition of sculptures and artifacts. 3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts 45

Fig. 3.5. Acquisition results from excavation units at the Jinsha Site (the 9th excavation) by Zhejiang University. Both (a) and (b) have four excavation units (with the unit number denoted), and the inter-connected girders have been removed. (a) IXT9641, T9642, T9741, and T9742; (b) IXT7128, T7129, T7228, and T7229

3.2.1 Digital Technology Makes Sculptures and Artifacts Remain “Young Forever”

Although museum preserved relics are stored carefully with appropriate hu- midity and temperature, they still face the problem of corrosion or damage. Hence, it is important to use digital technology to record the current sta- tus of relics and make them remain “youngforever” in computers. With the rapid development of 3D scanning and digital photography in recent years, successful examples have also emerged one after another, providing complete solutions and operation specifications for the eternal conservation of the sta- tus information of relics. Technologies such as X-ray penetration and neutron tomography have provided high precision approaches for acquiring and stor- ing the internal structural information of relics, while various kinds of mass spectrum detection technologies and equipment have provided strong support for acquiring and recording the texture information of relics. The categories of information of museum preserved sculptures and artifacts that need to be stored include: (1) Shape. The unique shapes of sculptures and artifacts are an important embodiment of their artistic value. Acquiring and recording the 3D informa- tion of these items not only stores the current status of relics but also pro- vides reference information for their virtual restoration, aesthetic analysis, and technical analysis. (2) Surface Texture. Rich information of surface texture will provide a vivid appearance for sculptures and artifacts. High fidelity and highresolution surface texture data, combined with 3D models, can help to reconstruct true- to-life models, which may be used for a variety of applications such as virtual exhibition and research. (3) Internal Structure. For those elaborately designed sculptures and ar- tifacts, analyzing their internal structures can help people learn about their manufacture techniques and discover more about their artistic and scientific 46 3 Digitalization ofCultural Heritage value.Internal structures can also assist researcherstoworkoutacorrect and scientific plan for conservation. However, because of restriction of cost and operating environments, information acquisition technologies have not yet been applied widely to the internal structures of relics. In this section, we focus on the processes and key techniques of informa- tion acquisition of the 3D appearance and texture of relics.

3.2.2 Information Acquisition Process and Technical Framework for Museum Preserved Sculptures and Artifacts

Among various 3D scanning technologies, 3D structured light scanning is the most suitable for information acquisition of sculptures and artifacts. It has many advantages such as highefficiency, intensive sampling points and low error. The main steps in structured light scanning include: system calibra- tion, shape acquisition, surface registration and merging, camera calibration, texture acquisition, texture mapping, and multi-texture fusion (Fig. 3.6).

Fig. 3.6. Information acquisition process for museum preserved sculptures and artifacts

Toacquire and record the current status of sculptures and artifacts com- pletely and precisely, we should first calibrate the system and obtain the scanner’s parameters such as focal length, rotation and translation parame- ters. If a digital turntable is used, relative parameters between the scanner and the digital turntable should also be calibrated. These can help to merge multiple 3D surface data after we put the relic on the turntable and read the 3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts 47 rotating angle when scanning surfaces in different directions. After 3D surface shape information is scanned in multiple directions (or views),wematchthe data according to the relative location parameters or the rotating angle of the turntable. If adigital turntable is not available, then we should manually choose some feature points to register and merge. Cameras for shooting the surface texture should also be calibrated, so the reconstructed 3D model can be bound with the texture in the photo. Finally, multiple textures should be fused in overlapping areas of the same model to get a uniform color of the relics. The technical framework is illustrated in Fig.3.7.Arrows indicate supporting relationships.

Fig. 3.7. Technical framework of information acquisition of museum preserved sculptures and artifacts

This section focuses mainly on the acquisition and recording of the surface shape and color information of relics with the help of the structured light scanning and texture mapping and fusion. The 3D model registration and merging technologies will beintroduced in section 3.3. 48 3 Digitalization ofCultural Heritage

3.2.3 Key Technologies for Information Acquisition of Museum Preserved Sculptures and Artifacts

For the acquisition of museum preserved sculptures and artifacts, appropriate high precision structuredlightencoding anddecoding technologies and post- processing methodswill beintroduced. • High Precision 3D Structured Light Scanning Technology for Acquiring the Surface Shape of Relics

There are lots of successful cases of scanning the3Dsurface shape of relics, and a great variety of scanner are available in the market. The structured light scanner is believed to be the most suitable type because of its light weight and small size. Here we will briefly introduce the structured light encoding and sub-pixel decoding algorithms, which are applicable to the acquisition of a relic’s surface information (Liu, 2005). (1) Structured Light Encoding. Different encoded structured light is used to project patterns onto the object surface to construct active features, which will be acquired by sen- sors. The emitter and the sensor are called the light source andlightre- ceiver respectively. The light source, the receiver and the reflection point compose a triangle that is used for calculating the depth of the reflection point. Based on light source differences, the encoding method can be classi- fied into three types: single point measurement, single line measurement, and multi-line measurement (Fig. 3.8). The principle of these methods is same, but the pixel number reconstructed at one time is quite different.

Fig. 3.8. Three typical types of triangulation measurement. (a) Single point; (b) Single line; (c) Multi-line

The structured light pattern methods can also be classified as time-based, space-based, or direct encoding (Fig. 3.9). The time-based encoding method projects different patterns at different times. The values at different times compose the code of this point. The space-based method distinguishes dif- ferent points depending on their neighboring point information. The direct 3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts 49 encoding method distinguishes points directly according to the value of the point itself, without other information.

Fig. 3.9. Three typical types of structured light pattern. (a) Time-based encoding; (b) Space-based encoding; (c) Direct encoding

Time-based encoding patterns generally use gray scale images as they are abletoproduce a highly precise and intensive scanning result. The decod- ing process is also very simple. The limitation of the pattern based on time is that it is not applicable to dynamic objects. Space-based and direct en- coding patterns can be used to scan dynamic objects as they use only one projection pattern. But recognition errors may occur when there are large fluctuations and discontinuities in the surface. Astheencoding concentrates on one projection pattern, the sampling rate is also low. Therefore, the time- based method is the best choice for high precision 3D reconstruction of static relics. (2) Image Feature Extraction. Feature extraction is to decode the structured light code, extract feature information and compute the accurate corresponding relationships between points in a projection pattern (source) and points in captured images (re- ceiver). The system will project a picture encoded by the gray code and phase shifting (Giovanna et al., 1999) onto the surface of an object and pro- duce active features on the object surface, which will be further analyzed and extracted. For a 1024×768 projection pattern, at the moment of t0, t1, t2, t3,..., and t9 (for simplification, only t0 to t4 are shown), we project ten patterns and obtain ten corresponding images (Fig. 3.10). By analyzing the pixel value of different images, we can obtain the u value of the projection pattern for each image pixel. Ideally, the projection pattern ti has more white and black stripes as the number i grows, and the pattern t9 is composed of 1 pixel wide white and black stripes. Actually,wheni grows to a certain degree, the image captured will be not able to distinguish the pixel value precisely owing to the small width of the strips. Therefore, the phase shifting code is adopted if i is greater than certain degree. For example, we first use gray code from t0 to t6, and later use phase shifting code, which is composed of white and black stripes of 4 pixels width. After projecting apicture,wemovethewholepicture totheright horizontally by 1 pixel at each step to ensure the minimum 50 3 Digitalization ofCultural Heritage

Fig. 3.10. Gray code scanning intensity. The newly obtained picture will be used for the next projection. After 8 operations, the picture will return to the original view and the projection will end. Gray code and phase shifting are complementary toeachother. By decodinggray code and phase shifting, and extracting the scan line between white andblack stripes, we can obtain the corresponding relation- ship between pixel and projection code. However, this method still has a shortcoming since therelationship obtained is still at the pixellevel,which may cause large errors in subsequent triangulations. Hence, we must extract data at the sub-pixel level, which means the precision is within one pixel. Ideally, as shown in Fig. 3.11, El and Er are two integral scan lines ob- tained through the projection of two neighboring structured light stripes. El − Er is the corresponding picture area of one pixel projection width. This means that one projection pattern corresponds to several picture pixels.

Fig. 3.11. Extraction of sub-pixel border

Generally, the change between white and black stripes is gradual owing tophysical restriction. Er is assumed to be the translational borderina translational projection. For a pixel between Er and El, we let the gray scale 3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts 51 of this pixel be GA,andthegrayscaleofEl and Er be Gl and Gr.Then we can compute the coordinates ofpixel A at sub-pixel level by interpolation. (3) 3D Reconstruction of Surface Points. 3D reconstruction of scattered points aims to compute the 3D coordinates for each surface point. After structured light encoding and decoding,wecan obtain the coordinate of each pixel in the image, and its up value of horizontal coordinate in the corresponding projection image. Using thetriangulation measurement method, the surface points’ 3D coordinate can be calculated using the following method. After applying the camera and projector calibration method,⎡ we can ob⎤- axc 0 u0c ⎣ ⎦ tain the internal parameters of the camera and the projector, 0 ayc v0c ⎡ ⎤ 001 axp 0 u0p ⎣ ⎦ and 0 ayp v0p ,and theexternal parameters R and T between the cam- 0 01 era and the projector. Taking thecameraastheexample, axc = fc/dxc and ayc = fc/dyc, fc isthefocal length of the camera, and dxc and dyc are the horizontal and vertical physical sizes of each pixel in the sensor, such as CCDorCMOS. (u0c, v0c) are the pixel coordinates of the optical center of the camera. The projector’sinternal parameters are alike. R isa3×3 rotation matrix between the camera’s reference coordinate system and the projector’s coordinate system, and T is a 3 dimensional translation vector. Suppose the coordinates of one point in the camera coor- dinate and one in the projection coordinate are (Xc,YYc,Zc) and (Xp,YYp,ZZp) respectively, with the image pixel coordinates (uc, vc) and projector image horizontal coordinate up,thenitsatisfies: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤

Xp r11 r12 r13 Xc t1 ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ Yp = r21 r22 r23 Yc + t2 (3.2) Z r r r Z t p 31 32 33 c 3 Wecanobtain:

Xp = r11 Xc + r12 Yc + r13 Zc + t1 Z r X r Y r Z t (3.3) p = 31 c + 32 c + 33 c + 3 Using thesimilar triangles principlewecanobtain ⎧ ⎨xc/ffc = Xc/Zc y /ff Y /Z ⎩ c c = c c (3.4) xp/ffp = Xp/ZZp where xc, yc are the pixel coordinates in the sensor image plane, and xp is alike. xp Xp r11Xc + r12Yc + r13Zc + t1 Wecanobtain = = ,which can derive fp Zp r31Xc + r32Yc + r33Zc + t3 52 3 Digitalization ofCultural Heritage

xp t3 − t1 fp Zc =   (3.5) xc yc xp xc yc r11 + r12 + r13 − r31 + r32 + r33 fc fc fp fc fc

xc dxc (uc − u0c) uc − u0c By the definition of dxc, we can obtain = = , fc fc axc yc vc − v0c xp up − u0p in a similar way = , = fc ayc fp axp Substitute the above three formulas, wecanobtainthevalueof Zc, and xc yc finally obtain 3D coordinates of the point (Xc,YYc,Zc) by Zc ,Zc ,Zc . fc fc Using the 3D reconstruction method, we can obtain the point cloud of the relic surface. For a complete 3D mesh of the relic shape, we should also perform triangulation, registration, merging, and mesh simplification tasks.

• High Precision Surface Texture Mapping of Relics

By taking high resolution texture images and mapping them onto the relic mesh, we can obtain a complete 3D model of a relic. The key problem of texture mapping is computing the texture coordinates of each texture image based on the corresponding relationship between image and mesh model. (1) Texture Mapping. The texture mapping process is showninFig.3.12.

Fig. 3.12. Texture mapping process

The camera pin-hole model satisfies 3.2 Information Acquisition of Museum Preserved Sculptures and Artifacts 53 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎤ xw ⎡ xw u ⎢ ⎥ m11 m12 m13 m14 ⎢ ⎥ ⎣ ⎦ ⎢yw ⎥ ⎣ ⎦ ⎢yw ⎥ z v = M = m21 m22 m23 m24 (3.6) ⎣zw ⎦ ⎣zw ⎦ 1 m31 m32 m33 m34 1 1 where (xw, yw, zw) are the 3D model vertex coordinates and (u, v) are the image pixel coordinates. We name matrix M the projection matrix which is equal to ⎡ ⎤   ax 0 u0 0 ⎣ ⎦ R T 0 ay v0 0 (3.7) 0T 1 0010 Bymanuallychoosing enough corresponding points between themodel and the image, we can solve the unknowns R and T (if the camera’s internal parameters ax, ay, u0, and v0 are available),ormij (i=1,2,3; j=1,2,3,4)(if the camera’s internal parameters are unavailable). Then other texture coordinates (u, v) canbecalculatedbytheirworld coordinates (xw, yw, zw) by dividing the first and the second row by the third row of theaboveformula. To compute the image’s overlapping area on the model, three variables should be determined: the direction of each polygon, the validity of the poly- gon’s texture scale, and self-occlusions. The direction of a polygon can be determined by the projection matrix. If the z value is positive, then it is a positive polygon, and vice versa. Only the polygons with positive z can be displayed with the corresponding texture. Polygons with negative z values are back directed, and thus should not be displayed. The validity of the polygon’s texture scale can also be determined by the projection matrix directly. The image coordinates (u, v) of a vertex (xw, yw, zw) are calculated by the projection matrix. If the image coordinates are smaller than the image resolution, which means the texture coordinates are within [0,1], then the texture is valid for the vertex, otherwise invalid. Sometimes more than one polygon is projected onto the same texture area, but only one is visible, as a result of self-occlusion. If we compute all polygon vertexes with all the other polygons, it would be extremely time- consuming, as this computation has a time complexity of O(N 2) and a relic model usually contains millions of vertexes. Here we introduce a method with time complexity of O(N). Firstly, we create a set for each texture image pixel. Then, we project each polygon of the model onto the image plane according to its projection matrix and compute the pixels covered by each polygon. For each pixel coveredbythepolygon, we add thepolygon to the pixel’s corresponding set. Finally, in each set, if all the polygons are connected, then there is no self-occlusion, otherwise the visible polygon can be determined by the z value. 54 3 Digitalization ofCultural Heritage

(2) Fusion of Texture Images. Fusion of texture image includes two steps: global color balance and local color fusion. Global color balance is based on the adjustment of color difference in the textures’ overlapping area. We first transform RGB color to lαβ color space and carry out global balance in l channel, which means brightness, to achieve globalbrightness consistency. Then we conduct globalhomogeneous transformation for the αβ channelsoastoreachtheglobal balance of color. The following formula is adopted for the measurement of the luminosity of overlapping areas. 2 E T L→− − T L−→ Φ i,m,j = ( i→A ji m→A jm) ( )(3.8) i,j,m(i=m)

A L→− j i where isthetarget luminosity, ji is the luminosity of pixel in image , L−→ j m Φ i,m,j j jm is the luminosity of pixel in image , and ( ) = 1 if pixel is seen in both image j and image m and0otherwise. The transformation relationship between RGB and lαβ isshownas ⎡ ⎤ ⎡ ⎤ 1 ⎡ ⎤ ⎡ ⎤ l √ 00 L ⎢ 3 ⎥ 11 1 ⎣α⎦ √1 ⎣ − ⎦ ⎣M ⎦ = ⎣ 0 6 0 ⎦ 11 2 (3.9) β √1 1 −10 S 00 2

L = log L, M  =log M,S = log S (3.10) ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ L 0.3811 0.5783 0.0402 R ⎣M⎦ = ⎣0.1967 0.7244 0.0782⎦ ⎣G⎦ (3.11) S 0.0241 0.1288 0.8444 B The formula for relighting in αβ channel is: ⎡ ⎤ ⎡ ⎤ ⎛ ⎡ ⎤⎞  αi αi cos θi sin θi ti1 ⎣ ⎦ ⎣ ⎦ ⎝ ⎣ ⎦⎠ βi = Ti→A βi , Ti→A = − sin γi cos γi ti2 (3.12) 1 1 0 01 To achieve the fusion of local color, firstly, we should project several inter- overlapping texture colors onto every vertex in themodel.During thepro- jection process, the normal vector is adopted and the α value is used as the weight value. Then adjustment is made on each texture color based on the color of vertex by interpolation, to make the color at the joint area consistent and even. When making the adjustment, the gradual transition interpolation is made to brightness channel by HSV space to eliminate the visible joint seams. 3.3 Information Acquisition from LargeScenes 55

3.2.4 Devices and Applications

Our group has developed a special 3D shape information acquisition device and high precision texture mapping software for museum preserved relics, by making full use of the above technologies. They have succeeded in the acquisition and recording of many valuable relics in China from Jiansha Site inChengdu, the LiangzhuSiteinHangzhou, the Hemudu Site in Yuyao, Zhejiang Museum, and so on (Diao and Lu, 2007). The device is shown in Fig.3.13and some acquisition results are shown in Fig.3.14.

Fig. 3.13. NMLAB 3D Scanner developed by our group

3.3 Information Acquisition from Large Scenes

The goal of acquiring scene information from large sites is mainly to record the scene appearance and color. The results will be useful for virtual restoration, virtual exhibition, development planning, and so on.

3.3.1 Process and Technical Framework of Large Scene Information Acquisition

Like the acquisition process mentioned in subsection 3.2.2, the acquisition process of scene information also includes 3D scanning system calibration, shape acquisition, surface mesh registration and merging, camera calibration, texture acquisition, texture mapping, and multi-texture fusion. However, as the data sets from large scenes are much larger than museum preserved relics, 56 3 Digitalization ofCultural Heritage

Fig. 3.14. Some acquisition results from museum preserved relics. (a) The gold mask excavated from the Jinsha Site; (b) Bird-shaped pot excavated from the Hemudu Site; (c) Black pottery in the shape of a bean excavated from the Liangzhu Site; (d) Buddha preserved in Zhejiang Museum it is necessary to simplify the 3D model acquired by scanning. Otherwise computers will probably be unable to process these large data sets. The process is shown in Fig. 3.15. Large scenes generally contain a lot of information, such as composition, soil humidity,locationinformation and so on. Here we focus on the acquisition of shape and color information. The technical framework is shown in Fig. 3.16.

3.3.2 Key Technologies of Large Scene Information Acquisition

Compared with museum preserved objects, large scene information acqui- sition has to deal with problems caused by large data volume, using scene 3.3 Information Acquisition from LargeScenes 57

Fig. 3.15. Largesceneinformation acquisition process information filtering, multiple 3D mesh data registration and fusion.

• Registration of 3D Information Acquired from Large Scenes

To rebuild thewhole scene, we have to dividethe scene into several parts according to different perspectives. As measurements of different areas are always needed in the corresponding local coordinate systems, we should unify the local coordinate systems into a singlesystemand remove theoverlapping parts to obtain the complete data from the measured objects. This makes ustocombine all point clouddata acquiredbypartial measurements. This involves two steps: registration and merging.Herewewill describe some reg- istration techniques. There are two methods for registration: one is to design special mea- surement devices that can record their distance and anglewhen moving or rotating during the measurement; and the other is to use computer software to solve the problem and realize reconstruction of the original model. This approach is the most frequently used in non-contact 3D scanning measure- ment. The point cloud data acquired in measurement can be regarded as rigid bodies. Point cloud registration can be seen as a process of coordinate trans- formation. It involves aligning two partially overlapping point clouds opti- mally by coordinate transformation according to a pre-designated optimal 58 3 Digitalization ofCultural Heritage

Technical framework of largesceneinformation acquisition matching principle. The optimal matching principle most frequently used in applications is the method based on three base points. Asthreepoints can determine a plane, after specifying three corresponding points in the different cloud point data sets, we can solve the transformation parameters between data scanned from different perspectives. Usually, the more corresponding points specified, the better is the registration effect that can be achieved by the least square method. Currently, most of the point cloud registration algorithms are based on the ICP (Iterative Closest Point) method proposed by Besl (Paul and Neil, 1992). Using this algorithm, we have to preset a point set P (which has to be transformed) and a point set Q. To make P match Q,wehavetofind the minimum distance from points of P to Q and establish the mapping relationship between the points. Then we calculate an optimal coordination transformation M using the least squares method with minimum square and let P =M(P ). We do this operation iteratively until meeting the required precision. The final coordinate transformation is the combination of each transformation. For example, suppose there are two point sets S1 and S2 that overlap at some area. S1 and S2 are corresponding locations of measurement points. The procedure of iterative operation is as follows: 3.3 Information Acquisition from LargeScenes 59

(1) At the kth iteration, we use reverse projection to determine theoverlap- ping area. (2) To find the closest point in the overlapping area, let pi ∈ S1 and x ,y ,zz C pi =( p p p) be the eigenvector. pi is the covariance matrix of eigen- vectorsofpi and its k-nearest-neighbor point. qi ∈ S2 isthepoint with the minimum distance from point pi to S2. (3) For all the nearest point-pairs pi and qi in theoverlapping region, using the function below and the quaternion method, re-compute the transfor- mation matrix Tk = [Rk, tk],amongwhichRk is the orthogonal rotation matrix and tk isthetranslation vector.

T R p t − q C R p t − q min ( k i + k i) pi ( k i + k i)(3.13) i (4) Compute the registration error:     R p t − q TC R p t − q E = ( k i + k i) pi ( k i + k i) (3.14)

(5) If E < ( is the given threshold value), or iterative time k is exceeded, then the iterative operation stops. Otherwise go to step (1). Here, we choose at least three of pair points in the two sets through human observation to compute the initial rotation matrix [R0, t0] using the formula in step (3) and the quaternion method.

• Merging of Large Scene Polygon Models

With themodel pieces registered optimally, there are still some overlapping areas between different mesh models, thereforewehavetoremovetheredun- dant data and merge them as a whole. We can first project the 3D mesh model onto the camera’s imaging plane. For each triangle projected on theimaging plane, we compute its bounding box in an octree scene graph. As shown in Fig. 3.17, the red rectangle is the bounding box of the blue triangle, which is formed by each triangle grid projected onto the imaging plane of the camera. For each projected triangle, if the angle between its normal and view vec- tor is greater than 90◦, we discard it. Those triangles, whose boundingboxes have no area overlapping the bounding boxes of triangles from other mesh models, will also be skipped. If at least one triangle’s bounding box overlaps the current triangle, we should investigate therelationship between thetwo triangles. There may be three kinds of relationship, separated, intersected, or contained (Fig. 3.18). Suppose there are two triangles, T1(P1,PP2,PP3) and T2(Q1,Q2, Q3).For point P (x, y),weinvestigatetherayL from P to the left (to the infinite). If the number of intersection points of L with thetriangleisodd, P isin the triangle. Otherwise P is out of the triangle. If any of P1, P2 or P3 arein 60 3 Digitalization ofCultural Heritage

Fig. 3.17. Bounding box of triangle

Relationships between two triangles. (a) Separated; (b) Intersected; (c) Contained

T2, then T1 and T2 are overlapping (intersected or contained).Ifnoneofthe threepointsareinT2, then consider the edge relationships of T1 and T2. For symmetry,weconsider therelationship between P1P2 and Q1Q2.

(1) Let R bethe rectanglewhose diagonalline is P1P2,and S bethe rectangle whose diagonal line is Q1Q2. If the two rectangles do not intersect each other, the two edges are not intersected. Otherwise go to step (2). (2) Consider (P1Q1 × Q2Q1) · (P2Q1 × Q2Q1), if the two lines intersect each other, the value is no more than 0 and vice versa, since P1Q1 and P2Q1 areon differentsides of Q2Q1.

IfanytwoedgesofT1 and T2 do not intersect each other, then T1 and T2 are separated. We project all triangles onto a plane. If the edges of two triangles are intersected or one triangleisinsidetheother, then we compute these two triangles in space. If the distance between them is smaller than the threshold value, they are overlapping. Then we compute the dot product of their normal 3.3 Information Acquisition from LargeScenes 61 vector, the dot product of the line connecting the centroid of each triangle and the optical center of the camera’s coordinate system. The one with smaller dot product is marked as redundant and should be removed. To obtain a complete mesh model, we should seam some holes leftafter merging. Seaming is generally carried out at the seam between different grid models. Traverse the two lines, we will find two neighboring sides on the two lines (the directions of the sides are contrary), and seam them as seed sides. Firstly, we can obtain two triangles from two sides. By connecting the two seams, we can obtain three new sides; meanwhilethe seamingline is also changing. The subsequent seaming work is conducted on every seam. Then a new trianglecanbeobtainedbythe new side generated each time. Finally, we calculate the ratio among the three sides and put the triangle with the smallest ratio into the mesh. The process is shown in Fig. 3.19. By repeating this process, we can finish the seaming work between two grid models.

Fig. 3.19. The seam between two polygon model patches in (a) can be stitched up step as in (b) and (c)

3.3.3 Typical Applications

In recent years, a lot of research has been carried out on the technologies asso- ciated with acquiring site scenes using scanners. Calgary University, Canada has used a laser scanner to acquire the cloud points of a whale bone in the Thule whale house. With the help of modeling software, they have made models of whale bone and realized virtual reconstruction of the whale bone house (Levy et al., 2006). Technologies such as 3D laser scanning, scattered point 3D reconstruction, texture mapping, model format transformation and realistic rendering were applied in the project. The reconstruction procedure isshown in Fig.3.20. Zhejiang University and DunhuangAcademy have cooperated to finish the 3D reconstruction of the 158th cave of Mogao Grottoes (Fig. 3.21).We used 3D acquisition and processing technology to acquire the surface shape information from the scene, simplify, registry and mergemodelstoobtain 62 3 Digitalization ofCultural Heritage

Fig. 3.20. Whale house reconstruction procedure

Fig. 3.21. The 158th cave in the Dunhuang Mogao Grottoes. Mid-Tang Dynasty 781∼847 A.D. D: 720 cm, W: 1810 cm, H: 680 cm 3.4 Information Acquisition of LargePaintings and Murals 63

3Dmodel,and also used a digital camera to obtain high precision texture photos. Finally,wemappedandfused the textures to obtain a complete 3D model of the cave with a satisfying sense of reality.

3.4 Information Acquisition of Large Paintings and Murals

This part focuses on the acquisition of large plane heritageitems,suchas paintings and murals. The images obtained are high resolution, largearea, and high precision.

3.4.1 Process and Technical Framework of Acquisition of Large Paintings and Murals

Acquiring image information of these relics is generally the key step in relic information acquisition. However, few devices are available in the market to meet this requirement. A single scanner is too small to acquire the whole painting,whileasingle digital picture is not able to meet the precision re- quirement for its low resolution. Usually, we will divide a painting into several parts and scan one part at a time, and then merge all data together. The process is shown in Fig. 3.22. Before taking blockwise shots of alarge painting or mural, we need to calibrate the camera and make an acquisition plan. Then, geometry rectification and color correction steps need to be taken

Fig. 3.22. Information acquisition process for largepaintings and murals 64 3 Digitalization ofCultural Heritage to retain the fidelity of the photo images. After automatic mosaic and global optimization steps, we can finally obtain the complete image.

3.4.2 Key Technologies for Information Acquisition of Large Paintings and Murals

The key technologies for information acquisition of largepaintingsandmu- rals include geometry rectification, color correction, automatic image mosaic and global optimization. The technical framework is shown in Fig. 3.23.

Fig. 3.23. Technical framework for information acquisition of large paintingsand murals

• Acquisition Planning

During the digitalization of largepaintingsandmurals,image resolution loss and precision error may occur because of malformed walls or uneven paint- ings, deformed camera perspective, improper shooting spot, or incorrect auto- matic mosaic, etc. However, as the resolution and precision of images should meet certain requirements of digitalized appreciation, errors should be con- trolled in the digitalization process to obtain the desired effect. Themainstepsincontrolling image precision error are: analyze thesteps inthewhole digitalization process where digital resolution may change, an- alyze the reasons for various kinds of errors, work out the error data by 3.4 Information Acquisition of LargePaintings and Murals 65 calculation and practical experiment to identify the range of error, and fi- nally rectifytheimage based on the data obtained. Loss of image resolution precision becomes directly apparent at the mosaic step, but its mainly origi- nates in the step of photographing. Errors may be caused from the shooting location, the lens, projection distortion, unevenness in walls of murals, etc. As cameras do not work ideally according to the pinhole model, optical distortion (lens distortion) errors arise between the real image and the ideal image. Distortion errors may be divided into three types: radial, decentering, and thin-prism distortion. Radial distortion only generates radial error, while decentering and thin-prism distortion generate both radial and tangential errors. Abdel-Aziz and Karara (1971) proposed a direct linear transformation method for camera calibration. They carried out in-depth research on the re- lationship between the camera’simageand environmental objects using pho- togrammetry and established a geometrical linear model for camera imag- ing. The parameters of this kind of linear model can be solved by linear functions. Tsai (1987) proposed a method based on RAC (Radial Alignment Constraint), that makes use of radial consistency to work out the parameters of the camera (except for the translational movement of the camera’s optical axis direction). The RAC-based method works effectively as it uses lots of linear functions that can reduce the complexity of computation. After finding the distortion parameters of a camera and lens with a certain focal length, we can compute the best shooting distance. When the distance between the camera and the surface exceeds, errors will grow (Pan and Fan, 2004) We must take account of the surface incline in acquisition. If the largest θ M angle of asurface incline is , and thefinal resolution required is ppi, then M 2 2 theshooting resolution should reach cos θ /inch (Fig. 3.24).

Fig. 3.24. Effect of incline on photographing

Suppose the lens cone angle is φ, and the resolution of the camera is W × H, then the longest distance from the camera to the object surface is W cos θ φ L = cot . 2000M 2 The resolution precision of images taken by cameras can be influenced by a lot offactors, many of which cannot be measured or calculated accurately. 66 3 Digitalization ofCultural Heritage

By correcting lens distortions and eliminating the effect of uneven surfaces, we can effectively improve the acquisition resolution and fidelity of paintings or murals.

• Automatic Image Mosaic

Photos taken for different parts at different times, from different views, and with different sensors should be registered and fused. Image registration, based on the similarity between images, includes the following three types. (1) Fourier Transform Shift Theorem Based Method. Let f2(x, y) be the imagetranslatedby x0 and y0 in x and y direction of f1(x, y), namely, f2(x, y)=f1(x − x0,y − y0). The Fourier transform of f1 and f2 is F1(u, v) −j(ux +vy ) and F2(u, v) respectively. Then it satisfies F2(u, v) = F1(u, v)e 0 0 . ∗ F1 (u, v) F2 (u, v) The cross-power spectrum of f1(x,y) and f2(x, y) is ∗ = |F1 (u, v) F2 (u, v)| j(ux +vy ) ∗ e 0 0 where F2 is the conjugate complex number of F2. We generate a unit impulse function at (x0, y0) by inverse Fourier transform. The pulse distance is the sum of translation between two images that are going to be registered. For images that need to be both rotated and translated, they should be rotated first. Fourier transform shift theorem based method is sim- ple and accurate. However, it works only for images with large overlapping areas (usually 50% overlap between images). (2) Pixel Based Method. Generally, the methods based on pixels include: (a) Pixel matching. It has low precision and massive computation. (b) Piece matching. It has high precision and massive computation. (c) Grid match- ing. Using this method, it is difficult to select the initial value. Alternatively, we can use the method of minimum sum of squares of gray scale differ- 2 ence between two images. The function is E = [I (x, y) − I (u, v)] .The value that reaches the minimum value of E istheoptimal mosaic param- eter. For simplicity, we can use the minimum absolute difference function E = |I (x, y) − I (u, v)|. (3) Feature-Based Method. The feature-based method includes four steps: feature extraction, feature matching,image interpolation, and re-sampling. The commonly used interpolation methods include closest neighbor, spline and bilinear interpolation. If the image registration method is identified, the imagematching method will also be identified correspondingly. Currently, the most frequently matching method is the method based on characters, while the most effective method is to combine the method based on pixels with the one based on characters.

• ImageFusion

After image mosaic, images of different parts may differ in color, thus need to be fused. Fusion can be based on spatial domain or transform domain. Spa- 3.4 Information Acquisition of LargePaintings and Murals 67 tial domain methods include mainly the logic filter method, weighted average method, morphology method, and simulated annealing method. Transform domain methods include mainly the pyramid method, wavelet transforma- tion method, color space method, false color method and the Kalman filter method. The weighted average method is an effective method used for fusing im- ages. But it is sensitive to noises, and will cause a largeamountof noises. The simulated annealing method treats pixels and their neighbours as atoms or molecules in the physical system. It uses an energy function to describe the system and to determine the Gibbs distribution (Geman et al., 1994). The Gibbs distribution is equivalent to the Markov random field (MRF)(Li, 1995). If an image can be expressed by MRF, the image model can be determined by the energy function. When the energy function is reduced toaminimumvalue, thephysical system annealsthe global minimum value too. Then, the preliminary image with noise is turned into the maximum a posteriori estimation of the real image. The multi-resolution pyramid algorithm is a popular image fusing method. It can be divided into the Guass-Laplace pyramid, gradient pyramid, ra- tio low-pass pyramid, and morphology pyramid, based on different pyramid structures. The wavelet transformation based method splits the source imageintodif- ferent character domains with different frequencies, then fuses them to build a new wavelet pyramid, and finally synthesizes images by inverse wavelet transformation. Compared with the multi-resolution pyramid method, this method has lots of advantages: (1) It is more compact than the Laplace pyramid. (2) It provides direction information; pyramid methods do not introduce the space direction into thesplitting process. (3) There are redundancies in the Laplace pyramid method but none in the wavelet method. It is difficult to judge whether redundancies or charac- teristics of the image itself cause the image similarities. (4) The pyramid reconstruction process is sometimes unstable; the wavelet method is more stable. The color space method makes use of different advantages of RGB and HIS in display and calculation. We convert RGBintoHIS and separately compute three independent components, as the HIS color spaceismoresimilartothe human visual system. After computing, we convert it back into RGB, as RGB spaceismoreeffective in display.

3.4.3 Typical Devices and Applications Dunhuang Academy and Zhejiang University, China have cooperated in re- search on the acquisition of information from valuable ancient large-size im- ages and have developed a special specification for acquiring information of 68 3 Digitalization ofCultural Heritage murals. They have also acquired information of murals from more than ten caves, with the lowest resolution of 72 ppi (pixel per inch). The results of the murals on the west wall of the 196th cave of the Dunhuang Mogao Grottoes are shown in Fig.3.25.

Fig. 3.25. Digitalized acquisition results of murals in the 196th cave, the Dun- huang Mogao Grottoes, China. Arrows indicate the zoom-in process involved in the selection of rectangle areas

In 2002, Zhejiang University and Dunhuang Academy cooperated to un- dertake the digitalization and replication of the Map of the Ming Dynasty which is preserved by the First Historical Archive of China. The map is 4.5 mlong and 4.2 m wideand its resolution reaches 300 ppi. Theacquiring scene and detail of the map are shown in Fig. 3.26.

Fig. 3.26. Acquiring scene and detail of the map of the Ming Dynasty. (a) Acquir- ing scene; (b) The results

Based on information acquisition from murals and large-size maps, Our group has developed a set of devices which can scan ultra large paintings. Each device can scan paintingsaslargeas7.5mlong and 3.5 m wide with maximum resolution reaching 600 ppi. The device is shown in Fig. 3.27. 3.5 Summary and Prospects 69

Fig. 3.27. Ultra-scale painting acquisition device developed by Zhejiang University. The main parts of this device include a platform, movingframe box, motor control system, camera and light system, etc. 3.5 Summary and Prospects

In this chapter, we have introduced thetechnologies applied in acquiring information from archaeological sites, museum preserved sculptures and ar- tifacts, large archaeological site scenes, and large paintings and murals. We have also described information acquisition processes with their technical frameworks and some relevant applications. The technologies introduced have come through the preliminary techni- cal trial phase and have been applied in many fields. However, despite their constant development and maturation, they still have some deficiencies. Tech- nologies such as the acquisition of on-site 3D information and the unearthing locations of relics have been applied in excavation sites, but methods to ac- quire overall time-continuous 3D process information are still being explored. As for museum preserved sculptures and artifacts, althoughwehavetheabil- ity to acquire high precision 3D information and texture information, acquir- ing information of surface materials is still a problem. In respect of acquisition of large scenes, the scene reconstruction process based on 3D information ac- quisition devices is quite mature, but we still face with the problem of how to process enormous amounts of data. For large paintings and murals, some special information acquisition devices have already been created, but we still cannot meet the needs of retaining color fidelity and acquiring high precision data from concavo-convex surfaces. There are many examples of the successful application of digital technol- ogy in recording a variety of culture heritage items in many countries. Japan, 70 3 Digitalization of Cultural Heritage

USA, and the European Union have many competitive advantages in apply- ing relevant technologies and in project implementation. The application of digital acquisition technology to the culture heritage of China is also devel- opingfrom basic digital photography, digital scanning,andfile databases towards 3D information recording, multimedia recording and data base on grid. Digitalization and recording technologies will become more rapid, effective and be able to acquire more typesofinformationinamoredetailedway. These technologies not only provide a strong and better support for recording, conservation, research,exhibition and relevant applications but also promote new applications such as remote disease diagnosis and Internet exhibition.

References

Abdel-Aziz Y, Karara H (1971) Direct linear transformation from comparator coordinates into object space coordinates in close-rangephotogrammetry. In: Proceedingsof the Symposium on Close-RangePhotogrammetry, Falls Church, VA, American Society of Photogrammetry 1-18 Allen P, Feiner S, Troccoli A, et al. (2004) Seeing into the past: creating a 3D modeling pipeline for archaeological visualization. In: Proceedingsof the 2nd International Symposium on 3D Data Processing, Visualization and Transmis- sion, Thessaloniki, Greece 751-758 Boehler W, Marbs A (2002) 3D scanning instruments. In: Proceedings of the CIPA WG 6 International Workshop on Scanning for Cultural Heritage Recording, Corfu, Greece 9-12 Diao CY, Lu DM (2007) Interactive high resolution texture mapping for the 3D models of cultural heritage. In: Proceedings of Virtual Systems and MultiMedia, Springer, Brisbane, Australia 191-202 Geman S, Geman D (1984) Stochastic relaxation, Gibbs distribution and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(6):721-741 Giovanna S, Matteo C, Roberto R (1999) Three-dimensional vision based on a combination of gray-code and phase-shift light projection: analysis and com- pensation of the systematic errors. Applied Optics 36(23):4463-4472 Levy R, Dawson P (2006) Reconstructing a thule whalebone house using 3D imaging. IEEE Multimedia 13(2):78-83 Li SZ (1995) Markov Random Field Modeling in Image Analysis. Springer, 2001, 2nd Edition Liu Y (2005) Development of a Structured Light 3D Scanner Prototype System. MA, Zhejiang University (in Chinese) Pan YH, Fan JS (2004) Dunhuang, Virtual and Reality. Zhejiang University Press, Hangzhou Paul J, Neil D (1992) A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2):239-256 Tsai R (1987) A versatile camera calibration technique for high-accuracy 3D ma- chine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation 3(4):323-344 4 Archaeological Research Aiding Technologies

Archaeology is the science that studies ancient human society and history through remains and other environmental data left by ancient human be- ings. The development of archaeology depends more and more on natural science and engineering technologies. The introduction of computer technol- ogytoarchaeology will definitely bring revolutionary changes in at least two aspects, by improving work efficiency and meanwhile developing a new re- search method for archaeology. Inthis chapter, we will discuss some issues related to theprediction and detection of archaeological sites, remote sensing of archaeological sites, com- puter aidedline drawing generation, anddate aiding.

4.1 Digital Technology and Archaeological Research

Digital technology can assist thearchaeological work andhelpimproveits work efficiency. Writing reports is an important part of archaeology. How- ever, it is time-consuming to draw different types of line drawings. Digital technology can help not only generate line drawings of relics and archaeologi- cal sites but also help conduct digital measurement of relics or sites, based on the information obtained, thus reducing the measurement time which would be spent on material objects and avoiding potential damage to relics or sites causedby unnecessary improper contact. During archaeological research, a lot of process information concerning cultural heritages needs to be analyzed, such as the construction of build- ings and the manufacturing of objects. Besides, the original appearances of damaged cultural heritages also need to be identified. Digital technologyhas provided a visual simulation approach for assisting archaeological research. In addition, digital detection technology can help detect the distribution of underground archaeological sites accurately and avoid penetrating the sites directly during the planning of a construction, which may result in unrecov- erable damages. 72 4 Archaeological Research Aiding Technologies

The application of computer technology to archaeology started in the 1950s when scholars in the USA did research on ethnic tribes. In the late 1970s, Chinese scholars also began to apply computer technology to archae- ology as well. Hence, the application of the computer became more and more popular, while the field of application keeps expanding from excavation of relics to the detection of the unearthing location of relics, prediction of loca- tions of important archaeological remains and sites, computer aided genera- tion of archaeological reports as well as virtual evaluation of archaeological research hypothesis and virtual reproduction of archaeological research con- clusions.

4.2 Process and Technical Framework ofArchaeological Research Aiding

Archaeological research mainly includes three processes, investigation, field excavation and indoor work.

4.2.1 Process of Archaeological Research Aiding Technologies

The main purpose of archaeological investigation is to find the sites to be ex- cavated,including prediction anddetection. Prediction, evaluating the pos- sible scope of the sites, is the initial selection of sites. Detection, based on prediction, is to check whether the relic exists or not. Traditional detection is generallymade byconsulting the literature, which cannot beapplied to prehistoric sites. Traditionaldetection is done byusing Luoyang shovelsand analyzing remote sensing materials visually. It is ineffective and can only cover quite a small area using manpower. At the investigation stage, the 3S (GPS, RS, GIS) technology can offer much assistance. GPS can be used to locate the position of sites precisely while RS can utilize electromagnetic waves in different wave bands to de- tect sites and relics scattered inside. Nowadays, military ground-penetrating radar of some foreign countries can detect clear 3D images of objects buried underground at a depth of over 100 m which, technically, can also be ap- pliedinthedetectionof archaeological sites. Finally, GIS can demonstrate the distribution status obtained through investigations at the former stage to visually help make excavation plans for the next step. Viewshed analy- sis, surface cost analysis and remote sensing technology are always used in archaeological investigation. Several tasks should be undertaken during field excavation, including ex- cavation unit map drawing, culturallayer map drawing, relic distribution map drawing and measurements (Feng, 2003). Digital techniques can help to generate line drawings which can be used in further archaeological research re- ports. Digital measurement and mapping is also used in field excavation, hav- 4.2 Process and Technical Framework of Archaeological Research Aiding 73 ing the advantage of high speed and accuracy, mainly including close-range photogrammetry and measurement based on a 3D model and GIS/GPS. Indoor work includes classifying the materials obtained from archaeolog- ical excavation and investigation systemically, finding out the corresponding relation between relics and their dates by stratigraphy and typology, ana- lyzing the usage and manufacturing techniques of various kinds of relics and conducting other basic research. It is the basis for preparing archaeological research reports. It also includes identification of the relics and measurement of some key sizes. Digital techniques can be used for evaluation, dating and other laboratory research.In addition, computers can also help to simulate the archaeological research results. For example, based on theexcavated woods, we can simulate how thearchitectures were constructedbythose woods. The whole process is shown in Fig. 4.1.

Fig. 4.1. Process of archaeological research aiding

4.2.2 Technical Framework of Archaeological Research Aiding Technologies

During the whole process of archaeological research, three technologies can be used, that is computer aided investigation of archaeological sites, excava- tion aiding and computer aided quantitative analysis and research (Fig. 4.2). Some technologies are detailed. 74 4 Archaeological Research Aiding Technologies

Fig. 4.2. Technical framework of archaeological research aiding

• Scope Prediction Modeling Technology

Scope prediction modeling technology can predict the locations of sites or relics in one area, based on the models observed or assumption of human behavior. It mainly contains gradient analysis and direction analysis. In gradient analysis, an area is considered to be suitable for habitation onlywith an appropriate gradient. Hence, it is much more possibletodiscover sites in an area with a proper gradient than in other areas. For those areas that have been excavated, the gradient analysis model generated by GIS software shows that there are few sites at the places with a gradient of less than 1◦ or more than 3◦. Hence, the selection of archaeological sites can be done with the assistance of gradient analysis. As many places have most of their rainfall intherainy season, ancient peoples had to reserve sufficient water in the rainy season in order to maintain their life in the dry season. Therefore, they had to build dams on nearby rivers so as to create a reservoir upper reaches of the dam and collect a certain amount of water during rainy seasons with heavy precipitation, so as to meet their long-term needs. However, since a large gradient with a large drop in height is not suitable for building dams or reservoirs, people may not have settled down there. An area with a small gradientisalsonotagood choice because rivers there are not stable and might change their channels when floods break out. Only an area with a medium gradient is suitable for habitation. In direction analysis, it is generally assumed that ancient peoplechose a house facing south to obtain more light and to avoid cold weather brought 4.2 Process and Technical Framework of Archaeological Research Aiding 75 by northwesterlywinds, which is also recorded in ancient literatures.

• Viewshed Analysis

The viewshed of archaeological sites or encampments and whether a group of sites are in sight of each other are very important factors in archaeologi- cal structural research (Wikipedia, 2009) and is a common function of most GIS software. For example, if aplacecanonly be discovered from few per- spectives, it is less likely to be under the threat of wild animal attacks or climatic hazards and is suitable for habitation. On the contrary, if a place can be viewed from many perspectives, then it can be the location of many buildings such as tombs, statues of gods and towers. Similarly, a place can possibly be used for monitoring for being convenient for the observation of other areas, such as an ancient fortress. Hence Viewshed analysis plays an important roleinarchaeological analysis. For many cultures and tribes, the visual influence of an area is much more important than other information receivedbythe sensory organs.

• Surface Cost Analysis

Surface cost analysis analyzes the costs from some fixed point to other points and finds the optimal path (with least cost) from one point to the other on the rasterized digital map. Most GIS software can fulfill this function. It helps a lot to analyze the activities and environment of ancient people. For example, water resources are always supposed to be located within half an hour walk and the scope of human activities is always within a distance of a 4 hour walk. An analysis of the optimal path can be made based on the surface cost analysis, which can offer very helpful information for archaeologists, such as providing clues for discovering the trading path or other paths for particular uses.

• RS Technology

Historical relics are the remains or outcome leftby ancient peoples involved in certain activities with a specific purpose during a period of time which typically follows soil perturbation and landform modification theories. Thesoil perturbation theory indicates that in a construction project, peo- ple need to gothrough certain processes including excavating, tamping and building. Generally, when building a tomb, people need to clean and move the original soil and then backfill the soil. The soil for backfilling is always transported from other places and is mixed with other materials such as lime, sand and stone to ensure the stability and imporosity of the tomb. These ac- tivities change the original nature of the soil and destroy its capillarity and porosity, whichleadstoachangeinthe components, structure and moisture 76 4 Archaeological Research Aiding Technologies property of the soil. Hence, from RS photos, it can be seen that the soil has different image characteristics from the surrounding original soil. The landform modification theory claims that the construction of large- size archaeological sites, such as ancient towns, tombsand other architectures are closely related to the landform of their locations. Atthebeginning of construction, people make a plan, including how to take advantage of the location and arrange buildingsbasedontheterrainfeatures. Besides, in the process of construction, people would modify the terrain including expanding itaswell as leveling and tamping it, causing changeanddamagetothe original. Furthermore, the buildings would also derive new landform features. The change in the terrain can be demonstrated directly in the RS images. RS technology can be classified into six types, optical detection, electro- magnetic detection, gravity detection, nuclear magnetic resonance (NMR) detection, radioactivity detection and sonar detection.

• Digital Measurement

Digital measurement and mapping,replacing traditional measuring tape and rope by digital technology, has the advantages of highspeedandhigh accu- racy, mainly including close-range photogrammetry and measurement based onthe3DmodelandGIS/GPS. Close-range photogrammetry determines the distance between two points in real space by calculating two corresponding points on the photo through camera parameters which, just like a measuring scale, decide the transformation relation of the above mentioned two dis- tances. Measurement based on the3Dmodel calculates the distance between two points by using a Euclid distance formula for 3D models. Measurement based on GIS/GPS calculates the distance between interest points by geo- graphical coordinate data of the points acquired by GIS measuring tools.

• Drawing Generation

A fieldwork excavation map and indoor artifact drawing are also generated based on digital technology with high efficiency and high accuracy, instead of manual drawing by rough visual estimation. A contour draft of unearthed relics can be made by contour generation technology based on the acquired 3D model. An excavation site draft is generated by setting up a virtual camera and its parameters including the location, direction, focus length and aspect ratio. However, as what is excavated in fieldwork are mostly excavation units which keep expanding downward from one layer to the other, the target sur- face of the drafting is similar to a plane. For those drafts not requiring high accuracy, satisfying drafts can be generated by assuming that the target face is a plane. For the cultural layers that need to be marked, edge extracting technology can detect the contour and mark it, as different cultural layers have clear edges between them. In conclusion, in generatingfieldwork excava- 4.2 Process and Technical Framework of Archaeological Research Aiding 77 tion maps and indoor artifact drawings, edge extracting, contour generating and projection transformation technologies are mainly employed. Take the contour drawing generation of unearthed relics as an exam- ple. The contour drawinggeneration of relics is an important description of relics in archaeological reports. Asarchaeological reports are often printed in monochrome due to cost, the contour drawings are often used rather than photos for they can describe relics more clearly. The contour drawing,namely the section map of relics, is a closed circle formed by the intersection of a plane and the relic. In computers, the 3D model of relicsisoften repre- sented by triangles. Hence, the contour can be described as a set of segments formed by intersections of one plane and each triangle in the 3D model. Assume that the origin is on the plane (move the origin to the plane by co- ordinate transformation if the origin is not on the plane), then the formula f(x,y,z)=ax+by+cz=0 generates a triangle. And segments can be generated by the algorithm below: for each triangle P1P2P3 {

Assumecoordinates of three vertices are P1(x1,y1,z1), P2(x2, y2, z2), P3(x3, y3,z3) let mi = f(xi,yi,zi),if mi  0,si =1; else si = −1(i = 1, 2, 3). if (s1 = s2 = s3), go to the next triangle, since the triangle has no intersection with theplane. Otherwise find si that isdifferent. Assume s1 = s2 = s3,due to rotational symmetry. Determine two ends of the segment, i.e. Q1 =(P1 ∗|m3| + P3 ∗|m1|)/(|m1| + |m3|) Q2 =(P2 ∗|m3| + P3 ∗|m2|)/(|m2| + |m3|) draw segment Q1Q2 by graphics engine } The set of all such segments is the intersection of the relic surface and the fixed plane, i.e. the contour line. Fig. 4.3 shows the line drawing example for the Jinsha relics, generated by the system developed by Zhejiang University. A 2D image can also be used for line drawing generation. The edge ex- traction, edge detection techniques are used to generate lines, and projection transform can calibrate the drawing by camera parameters. These drawings are not always the final result needed in research reports, but a simple manuscript which can be further updated manually.

• Computer Aided Quantitative Analysis and Research

Computer aided quantitative analysis and research has led to a combination of quantitative archaeology and computer technology, the main research topic 78 4 Archaeological Research Aiding Technologies

Line drawing of Jade Dagger. (a) A normal perspective view of the Jade Dagger; (b) A vertical view of the Jade Dagger; (c) Line drawing of the Jade Dagger of which is an emerging subject, called computational archaeology, or archae- ological informatics (Burenhult, 2002; Huggett and Ross, 2004) or archaeoin- formatics. Computational archaeology is a subject which utilizes specially de- signed computer programs to process and analyze traditional archaeological data so as to study the behavior of ancient human beingsandtheevolutionof human behavior. It mainly includes statistical or mathematical modeling and simulation of the behavior and living conditions of ancient human beings. Ar- chaeological informatics try to uncover, discuss and represent quantitatively the characteristics and properties of archaeological information. To meet the need of basic research on how to process data, many types of software are produced to helpcomprehend and solve archaeological problems byquantita- tive analysis. Containing most of the theories and methods created since the 1960s, this technology has surpassed the scope of quantitative archaeological research and provided a solution for expressing archaeological information and problem structures byusing computer algorithms anddata structures. The main purpose of introducing computer technologyintoarchaeology is to solve the bottleneck problems such as low precision and low working 4.2 Process and Technical Framework of Archaeological Research Aiding 79 efficiency. Meanwhile, it has also contributed to the conservation of precious relics, as real contact with the relics has thus been greatly reduced. Inthepast decade, cooperation between computer scientists and archae- ologists has become increasingly close, and computer aided analysis of ar- chaeological materials has also become applicable due to the maturing of the technologies of computer graphics, image processing and artificial intelli- gence, the popularization of 2D or 3D acquisition equipment, and especially the enormous amount of research on the acquisition, representation and anal- ysis of 3D shapes based on computer geometric modeling technologyand3D laser scanning equipment.

• DatingBasedonTypology

Dating based on typology is implemented by searching the library of existing relics as shown in Fig.4.4.Featuresof relics are extracted from their images by object segmentation, edge detection and eigenvector based description. Then the features are sorted and compared with those of existing relics whose date have been determined. From the figure, we can see there are two key techni- cal problems involved in this process-feature extraction and expression and comparison of extracted features (pattern matching), which are introduced as follows.

Existing Eigenvector Digital Object Edge Based Image for Segmentation Detection Relics Description

Search and Eigenvector Digital Object Edge Match Based Image Segmentation Detection Description

Results

Fig. 4.4. Basic process of searching similar relics

For the draft, a set of contour points can be acquired by directly sampling since the contour is representedbyasegmental Bezier curve. For theobjects in database, Canny edge detection technology for colorful images can be utilized to acquire the set of contour points. Toextractthefeatures which are irrelevant with transformation, such as rotation or amplification, eigen-decomposition is utilized to express shape information on an orthogonal principal components basis. The adjacency ma- trix is defined to establish the corresponding relationship between the sets 80 4 Archaeological Research Aiding Technologies of contour points of the draft and the sets of contour points in the database under eigen-decomposition. The adjacency matrix is defined by the corre- sponding relationship of the position above and is orthogonal. The eigenvec- tor of the matrix contains the shape information of 2D objects. The adjacency matrix H is defined as follows: 2 2 Hij =exp(dij/2σ ),i,j = 1, 2, ...,m (4.1) 2 2 where m is the number of points on the contour, dij = ||xi −xj|| is the Euclid distance between xi and xj onthe contour. Parameter σ controls the Gauss width, which reflects the degree of inter-influence among points. When σ is small, the influence is local and, when σ is large, more points will influence each other. The eigenvalue and eigenvector is calculated according to the next formula after acquiring adjacency matrix H.

Hei = λei, i, j = 1, 2, ...,m (4.2) Since H is orthogonal, all the eigenvalues are positive. After eigenvec- torshavebeenacquired, normalization is required to ensure size-independent matching. Normalized eigenvector ei is called the pattern or feature shape. The pattern matrix is defined as follows:

T T T T V = [v1 v2 . ..vm] =[e1|e2 ... em](4.3) The eigenvector correspondingtothe greater eigenvalue is called a low frequency pattern which contains major shape information, while a smaller eigenvalue is called a high frequency pattern which contains transformation information of the local shape. Since the pattern matrix is orthogonal, m feature patterns construct an orthogonal basis which defines an orthogonal coordinate system space with m dimensions. The ith row vi of pattern matrix V is marked as the eigenvector of point i in the set of contour points, whose value correspondstoacoordinate in m-dimension space. In order to improve the search speed, the following two methods are em- ployed: Decrease the number of contour points set m. As for shape information ofheritage objects in the database, unify all the sizes to 200×200 and utilize a Canny algorithm to operate edge detection. Discard some eigenvectors with small eigenvalues. The information con- tained in the eigenvector is proportional to the corresponding eigenvalue. Firstly, sort eigenvalues in descending order and then fix the percentage P representing the shape information contained. The number of eigenvalues required can be calculated as follows:

t λi i=1  P m (4.4) λi i=1 4.2 Process and Technical Framework of Archaeological Research Aiding 81

Preset a search object e withfeature informationmatrix Ve anda database object l withinformation Vl, respectively containing me and ml points. To establish the corresponding relation between the above two ob- jects, a dissimilarity matrix Z(me × ml) is used to realize the purpose, which isdefined as follows:

e l zij = ||vi − vj||,i= 1, 2,...,me,j= 1, 2,...,ml (4.5) e l where vi is the ith column vector of Ve and vj is the j th column vector of Vl. To know the distance between them, they need to be redefined to t × l dimensions. By discarding some eigenvectors with small eigenvalue, it is proposed to decrease the dimension of ve and vl to ml × t and me × t. The value of zij is between 0 and 2. Through the following algorithm we can find the corresponding eigenvalues: Define the matching number k=0, thematching vector S for i=0to me {

Find the minimum value zij of the ithrow. If the minimum value of the jth column is also zij a matching pair is discovered. S[k]=zij, k++ } Define kmax =min(me,ml). Apparently k  kmax. Define the shape sim- ilarity measurement as follows. Obviously, the measurement satisfies the following basic properties: Non-negativity: d(sel)  0 Self similarity: (see) = d(sll) Symmetry: d(sel) = d(sle) Triangleinequality: d(sep)+d(spt)  d(sel)

• Relic Identification Based on Fuzzy Reasoning

Knowledge representation is one of the key problems of expert systems. Knowledge representation refers to the study of how to store a great number of facts and rules in specificprofessional fields on computer for processing.It is the foundation of the expert system and determines whether the system can simulate the thinking process of experts by reasoning, which has a great impact on how to solve the problems and the efficiency in solving those prob- lems. The form of proper knowledge expression can help organize knowledge about the objective world efficiently on the computer and maximize the func- tion of the knowledge acquired. In the field of relic identification, most proofs and knowledgearefuzzy. For instance, in describing the foot features ofa round tripod, fuzzy words like “thin” or “long” are often used. Hence, when designing a heritage identification aid system, knowledge representation is 82 4 Archaeological Research Aiding Technologies always in line with rules concerning the threshold and weighted fuzzy logic, which are shown as follows (Dong, 2006). If E1(W1) ∧ E2(W2) ∧ . ..∧ En(Wn) Then H(CF (H, E),λ) Ei represents the sub proof in the assumption of rules and Wi is the weighted factor of proof Ei, 0  Wi  1, i = 1, 2, ...,n. The normalized n condition should be satisfied, i.e. Wi =1. i=1 T (Ei) is the truth value of proof Ei and0 T (Ei)  1. If T (Ei)=0, Ei is false. Otherwise Ei is true. When T (Ei) is between 0 and 1, it indicates that Ei is close to the truth. λ is the threshold of rules. Only when T (E)  λ,can the rules be adopted; and the value scopeofλ is: 0 <λ1. H is theconclusion set of rules, which can be one or more than one. CF (H,E) is the reliability of the rules, also called rule intensity. It reflects the reliability degree in the case that H istruebasedonthepremise that E istrue, with 0 < CF(H, E) 1. The reliability degree of the fact that the conclusion H istrue based on the premise that E istruebecomeshigher, when the CF(H, E) is larger, and vice versa. During relic identification, the importance of different sub proofs to the conclusion always varies, only being the same in rare cases. A reasoning method based on weighted fuzzy logic can solve the problem in typical cases whereallthesubproofs share the same status to the conclusion. In the expert system of relic identification, the simplest method is to use the same formula to calculate the truth value of proofsregardless of the logic relation among various sub proofs, namely whether the logic relation is the conjunction or disjunction between sub proofs. Generally, the sum of proofs is the weighted mean of sub proofs, i.e. n [WiT (Ei)] T E i=1 ( ) = n (4.6) Wi i=1 Now, the truth value of conclusion H is T (H) = CF (H, E) × T (E).

4.3 Typical Applications

We will give some typical archaeological research aiding applications home and abroad in this section.

4.3.1 Utilization of RS

Archaeological research in Burgundy, France, mainly focuses on local settle- ment during a period spanning over 2000 years from the Celtic Iron Agetothe present. In the above research, RS technology and GIS have been employed for over 20 years. The project, named “Analysis on the Long-term Regional 4.3 Typical Applications 83

Archaeology Settlement Modes by Applying Geographic Information”,isthe task ofScott Madry, a doctor of anthropology from the University of North Carolina, Chapel Hill, USA. It is a subproject of a larger French project, initiated by Carole Crumley, a doctor of anthropology and ecology from the same university. It is a cooperative project involving a variety of subjects and conductedby several research institutes. Researchers deemed that theroads in that locality ought to be straight. But through visible scope analysis, it turned out to be wrong as the roads built by the ancient Celts should be in sight of the fortresses. Assume that the height of a fortress is 3 m and the eyes of the person standing on it are 2 m higher, the total height should be 5 m. The research results showed that the roads were in sight of the fortresses despite their being winding or undulating. Researchers have obtained more clues for further study through surface cost analysis on the distances to the roads, water sources and fortresses (Crumley, 1987). In the past decade, many RS archaeology research centers have been es- tablished in China and a lot of work has been done. For instance, in the remote sensing excavation of the Ruins of Yin in Anyang,Henan,bythe Chinese Academy of Social Sciences, computer image processing technology was utilized to make the analysis. During the analysis, TM images with low resolution and rich spectral characteristics from a U.S. land satellite and aerospace images with stable geometric relations were processed together. As a result, the quality of RS images was improved greatly and some new tombs and sites of the Yin Dynasty were also discovered. By using satellite RS image analysis, technologists from China have discovered many archaeological sites along the middle and lower reaches of the Changjiang River. In the middle of the Taklimakan Desert, the Jingjue, a country that had disappeared for almost 2000 years, has been rediscovered.They have also rediscovered the remains of a section of the Beijing-Hangzhou Grand Canal (which had been silted up for over 1000 years) and city walls of over 1000 km, built by Genghis Khan.

4.3.2 Digital Measurement of Large-size Archaeological Sites

In the excavation ofShouchun City, capital ofChu, computer image process- ing technology is utilized to process monochrome aerophotographs, infrared color aerophotographs and satellite photographs of various sizes, taken in the period between 1954 and 1980 and to make primary analysis of them. Through careful comparison and analysis as well as on-sight investigation, itwasfinally determined that the scope surrounded by a dark linear object on a remote sensing photo was the area of the ruins ofShouchun City and the inconsecutive furrows were the remains of city walls with a water chan- nel around them as a moat. Through investigation conductedby symmetry quadripole resistivity method and archaeological shovel exploration in the field, as well as a small area excavation of part of the city walls shown in the photo, it turns out that the analysis results from the remote sensing photo 84 4 Archaeological Research Aiding Technologies match the actual distribution. The result of remote sensing technology proves to be accurate. Finally, a remote sensing map ofShouchun City, with a scale of 1:1, is generated by integrating analysis and verification results, which clearly shows the appearance of the city (Gong, 2001).

4.3.3 Computer Aided Bronze Ware Identification Expert System

Identification of bronze ware is made to determine its age, identity and other properties. A computer aided bronze ware identification expert system, de- signed by Prof. GengGuohua from the Visualization Institute of the North- west University of China, is the expert system in this field which helps im- prove the efficiency of identification and promotes the communication and co-sharing of knowledge. China has a long history of bronze casting, which started at the end of the Neolithic Age, came to a peak in the Shang and Zhou Dynasties, and still flourished in the Warring States Period, the Qin Dynasty and the Han Dynasty. Thereafter, bronze wares were also cast during all the dynasties for practical use, worship of the ancient culture or for making profits as counterfeits. Bronze wares mentioned here mainly refer to those produced in the Xia, Shang and Zhou Dynasties and the Spring and Autumn period. Bronze wares cast during different periods are distinguished with the special characteristic of that period, Bronze ware cast in different periods has its own unique shape, pattern, epigraph, chemical components and casting techniques, which can be used to determine independently the date of the bronze ware. If there are no contradictions between the intersections of the period determined by the above-mentioned, then the time of the intersection can be judged to be the period when the bronzewarewascast.The bronzewarewill be determined tobeacounterfeit if there is any contradiction. A computer aided bronze ware identification expert system has been de- veloped and now put into use. It determines the date of bronze wares from the above five factors. The reasoning process is introduced as follows by taking the ancient bronze cooking vessel, the “Tripod”, as an example. Assume that a weighted rule exists in the system rule database as follows: If the ear of the tripod is erect (0.3); And the abdomen is shallow (0.2); Or the abdomen is high (0.2); And the bottom is circular (0.15); And the shape of the feet is a beast with curly tail (0.15) Then this tripod, called flat-footed tripod, is cast in the late Shang Dy- nasty (0.95, 0.8). If there are proofs existing in the fact database as follows: Erect ear T (E1) =1; Shallow abdomen T (E2) =1; 4.4 Summary and Prospects 85

High abdomen T (E3) =1; Circular bottom T (E4) =0.8; The shape of the feet is a beast with curly tail T (E5) =0.6. Bytheformula above, weights satisfy the normalization condition and the truth value of the combination can be calculated as follows: n [WiT (Ei)] T E i=1 . × . × . × . × . . × . . . ( ) = n =03 1+0 2 1+0 2 1+0 15 0 8+0 15 0 6 =091 Wi i=1 Since T (E) >λ, the conclusion is accepted. The truth value of conclusion H is T (H) = CF (H, E) × T (E) =0.95 × 0.91 = 0.86. That is to say, the truth value of the conclusion that the bronze ware is a flat-footed tripod of the late Shang Dynasty is 0.86 (Dong, 2006).

4.3.4 Reconstruction Simulation of Stilt Style Buildingsof the Hemudu Site No complete buildings were excavatedduring the two archaeological exca- vations in the Hemudu Site, while many piece of wood are excavated. From these pieces of wood, the archaeologists can draw lots of conclusions. For ex- ample, tenon and mortisejointswereused in the buildings bytheHemudu people. And the direction of the stilt style buildings of the Hemudu Site is not north and south orientated, but northeast and southwest orientated. The buildings faced at the south by east by 10 degrees, which can greatly avoid the hot weather of Zhejiang Province in summer. Research also indicates that the stilt style buildings are not directly based on the ground, but based on some vertical wood. The ancients employed such designtoavoid theanimals’ attacksand the high humidity causedbythewet ground. We have used 3DS Max to reconstruct the buildings to simulate the con- struction process and the original appearance of them according to the ar- chaeological research results. Further more, some physical principles may be taken into consideration in simulation. For example, we can test whether the angle of the roof is stable by physical analysis. But now such physical simulations are not yet employed. Fig. 4.5 shows some simulation results of stilt style buildings of the Hemudu Site.

4.4 Summary and Prospects

This chapter mainly focuses on digitalized archaeological work and researches into cultural heritages, including computer aided prediction and detection 86 4 Archaeological Research Aiding Technologies

Fig. 4.5. Simulation of the reconstruction of stilt style buildings. (a) and (b) are the stilt style buildings under construction; (c) and (d) are the final results. of archaeological sites, excavation aiding and computer aided quantitative archaeological research and analysis. Digital technology helps conduct infor- mational prediction and detection of archaeological sites and thus improves working efficiency. It makes drawing in fieldwork more accurate and reduces omissions and mistakes, laying a sound basis for the future recovery of sites and future archaeological researches. Quantitative research and analysis helps to uncover theregularity hidden behind complicated archaeological phenom- ena. Digital technology for cultural heritage research has provided archaeolo- gists with hi-tech methods and generates better results. Currently, computer and information systems are widely used in archaeology, but are still limited to the management of documents and materials in database and webpage applications for publicity and so on. Some scientists offoresight have started toapply computer technology to assist archaeological investigation and ex- cavation of sites, as well as quantitative archaeological research and analysis, obtaining great achievements and making a profound influence. Without com- puter technology, it is hard to process the increasinglyexpanding complicated archaeological data. References 87

The application of computer technology in archaeology is untypical and has always been misunderstood, in spite of the great role it has played. What is more, only a small number of scientists are able to apply complicated com- puter technologytoarchaeologyandbring the potential of computer and information technology into full play (Pelfer, 2004). Therefore, more efforts are needed to promote the application of computer technology in archae- ological researches. In respect of computer aided prediction and detection, hi-tech remote sensing equipments like ground penetrating radar and com- puter judgment technology are still not widely used. As far as computer aided archaeological excavation is concerned, we are still at the initial stage of data inputting and the drawing of fieldwork excavation mapping. Although there are textbooks introducing how to utilize Photoshop to make various kinds of excavation maps, generating excavation maps by photos automatically is still at thepreliminary stage. For quantitative research and analysis, there are al- ready some mature theories and many successful cases using SPSS (Statistic Package for Social Science). There is also a special system developed for au- thenticating bronze wares. However, these applications mainly focus on only certain points, general archaeology software used to solve common problems such as AutoCADhavestillnotcomeintobeing so far. The intersection of computer technology and other science is believed to provide new advanced approaches for archaeological research. In respect of location prediction and detection, the maturing and application of the new technologyofground penetrating radar will help achieve more discoveries of sites. As for computer aided excavation, new technologies will be developed to improve the efficiency and quality of generated maps. And as far as quan- titative analysis and research is concerned, more sophisticated archaeological research and analysis methods will form a software system to assist the ar- chaeologists to conduct their researches. In a word, with the exemplary role of scientists who apply computer technology in their work and obtain satisfying results, the computer will play a more important role in future archaeological work and research.

References

Burenhult G (2002) Archaeological informatics: pushing the envelope. CAA2001. Computer Applications and Quantitative Methods in Archaeology. BAR Inter- national Series 1016, Archaeopress, Oxford Crumley C (1987) Regional dynamics: Burgundian landscapes in historical per- spective. Academic Press, San Diego Dong LH (2006) An expert system of bronze identity based on fuzzy reasoning. Journal of Northwest University (Natural Science Edition) 36(2):197-200 (in Chinese) Feng EX (2003) Fieldwork Archaeology. Jilin University Press, Changchun (in Chinese) 88 4 Archaeological Research Aiding Technologies

Gong XC (2001) Practice of remote sensing archaeology in Anhui. Geology of Anhui 11(4):292-296 (in Chinese) Huggett J, Ross S (2004) Archaeological informatics: beyond technology. Internet Archaeology15 Pelfer PG, Barcelo JA, McDonaill C, etal. (2004) ArchaeoGRID, a grid for ar- chaeology. IEEE Nuclear Science Symposium Conference Record 4:2095-2099 Wikipedia (2009) Viewshed. http://en.wikipedia.org/wiki/Viewshed 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Throughout the ages, there have been so many cultural heritages, suffering from natural damage caused by wind, rain, thunder, fire, earthquake, insects and mildew, and also from man-made damage resulting from inadequate con- servation and improper handling.Therefore, cultural heritages are in urgent need of conservation and restoration. The conservation of cultural heritages refers to creating a safe and appropriate environment and taking positive mea- sures to increase their life spans and avoid deterioration and damage. Thus, the status of cultural heritages and their surrounding environment, such as temperature, humidity,CO2 density and the state of microorganisms and vegetation should be monitored in real-time. In addition, the storeroom, the exhibition hall and even transportation should also be safeguarded. Restora- tion aims to artificially restore damaged cultural heritages and improve their conservation status and prolong their “life”. Traditional conservation methods mainly rely on the manual operations of conservators based on their knowledge of the conservation field, which re- quires that the conservators should possess a solid basic knowledgeof human- ities, history, art, archaeology, as well as sophisticated handicrafts. However, there is a great shortage of personnel to handle the rich cultural resources. Therefore, a variety of subjects including physics, chemistry, art, information technology, are all widely used in conservation and restoration, among which digital technologies play an important role. This chapter focuses on three types of research in the field. Section 5.1 covers the digitally aided investigation of cultural heritages, including dig- itally aided mapping investigation, photographic investigation and disease investigation, aimed at helping the heritage investigation work, improving its efficiency and effect. Section 5.2 discusses environmental data monitoring is- sues, which helps real-time estimation and analysis of a heritage site’s status. Section 5.3 describes the digitally aided technology used in the constitution and implementation of cultural heritage conservation plans, such as a digi- 90 5 Digitally Aided Conservation and Restoration of Cultural Heritages tally aided mosaic of broken relics, virtual restoration of color and texture deteriorations of paintings.

5.1 Digitally Aided Investigation

Digitally aided current situation investigation serves as assistance for the above-mentioned investigation processes of cultural heritages. It helps to ac- quire information related to the environment, mapping, status and deteriora- tion of cultural heritages and manages the investigation information through databases, thus providing detailed information for developing a conservation plan forthenextstage.

5.1.1 Current Situation Investigation by Digitally Aided Technologies

Traditionally, an investigation of aculturalheritagemainlyreliesonthe conservator’s access to relevant information, on-site field investigation and hand-drawn maps of the archaeological site or cultural heritages, made by the conservators in person. Digital technology can be applied in cultural heritage investigation using two methods: one is by using digital technology as an aid to the traditional in- vestigation method, to improve work efficiency and handle the problems that cannot besolvedbytraditional methods. For example, in the investigation of the Qin Shihuang Mausoleum, researchers applied aerophotogrammetry to draw the map and the tomb army figurines position the map precisely. Meanwhile, close-rangephotogrammetry was applied to makeanelevation drawing and a contour map of various types of terracotta warriors, horses and weapons. Researchers have also used this method to measure the scope of the walls of thedoublecitywallof the Qin Mausoleun. During the en- vironmental monitoring and conservation at the excavation of thesiteof the terracotta warriors and horses, computer technology was also applied to analyze data concerning the air and soil. And microscopic analysis of the microbiology was also used to discover the hazards of microorganisms on the heritage and the remains. The second method provides a completely new method of investigation, based on the digital information of cultural her- itages. Theall-digital photogrammetry technology, whichgenerates heritage mapping through scanning the 3D models of heritages, is an example.

5.1.2 Process and Technical Framework of Digitally Aided Current Situation Investigation

In heritage investigation, there exist several phases, including elementary in- vestigation, detailed investigation and investigation report generation. And 5.1 Digitally Aided Investigation 91 detailed investigation also includes several different tasks, including envi- ronmental investigation, mapping investigation, photographic investigation, diseases investigation and so on. Digitally aided investigation aims to assist every phase of the traditional current situation investigation (Fig. 5.1).

Fig. 5.1. Process of digitally aided current situation investigation

• Environmental Investigation

Environmental investigation mainlyaimstocollect meteorologicaldata closely related to the objects in need of conservation through collecting literatures and conducting on-site monitoring, including collecting the temperature, hu- midity and density of CO2 of the surrounding environment. For large-size sites in the wild, data such as rainfall and wind speed should also be collected. Currently, environmental data concerning cultural relics can be acquired in two ways. One way is to conduct regular monitoring over a period of time with environmental sensors and then export environmental data from the sensors after a continual monitoring period. The other way is to construct a relic environment monitoring system, link the sensors to the network and conduct on-line monitoring.Inthis way, conservators can easily get access to all environment data from the dynamic monitoring system. In Section 5.2, 92 5 Digitally Aided Conservation and Restoration of Cultural Heritages we will discuss relevant dynamic environmental monitoring technology and systems. • Mapping Investigation

Mapping is also an important means for current situation investigation and the basis of conservation works. It provides basic drawings for subsequent disease investigation and the making of conservation plans. Cultural heritage mapping, unlike terrain mapping, has its unique characteristics. (1) Cultural heritages have many forms. They can be divided into traditional architecture, archaeological sites, cave temples, special memorial buildings, museum pre- served artifacts and so on. And heritages in the same form may differ in their structures. (2) There are a variety of mapping drawings for heritage items, such as planar graphs, sectionaldrawings, elevation drawings, contour maps, regional heritage topographic maps and thematic maps. (3) There are large differences in the surveying and mapping conditions. For example, under- water heritages are different to map. There are two forms of digital mapping investigations. The first form is a digitally aided method based on traditional methods, including geodetic methods, photogrammetry and remote sensing methods. Geodetic methods, which aim to identify precisely the terrestrial reference system, the geoid surface and the earth’s gravity field through a mathematical earth model, is mainly applicable to planar graph mapping and terrain mapping of ar- chaeological sites, ancient tombs, cave temples and ancient buildings. Pho- togrammetry and remote sensing methodsmakeitpossibletomovehigh precise image mapping data into laboratories, helping to improve work effi- ciency and precision. Close-range photogrammetry can be used to map the cultural relic’s facade, contour line and sectional drawing. Satellite remote sensing methods are mainly used for wide range qualitative analysis of cul- tural relics. The other form is a purely digital mapping method, which makes use of digital models of cultural relics, including 3D models and 2D imagedatato map automatically. The digital mapping method makes use of the digital- ized data, which is convenient for storage, fast for retrieval and flexible for output and input. To store drawingsof cultural relics in digital forms can facilitate their use. For example, digital measurement of a cultural relic can be conducted based on its model acquired through 3D scanning. All-digital photogrammetry was adopted in the conservation of the cliffface of the Dun- huang Mogao Grottoes. • PhotographicInvestigation

The aim of photographic investigation is to get a panoramic photographic record of the cultural relics’ status. At an early stage, records of photographic investigation were always made by film cameras. With the emergence of dig- ital photographingdevices and the improvement in their precision, digital 5.1 Digitally Aided Investigation 93 photography has become the mainstream technology of cultural heritage pho- tographic investigation. Photographic investigation has also become a neces- sary part of the digitalization of cultural relics. In Chapter 3 we discussed the application of digital photography to large-scale archaeological sites, museum preserved artifacts, large-size murals and so on. • Disease Investigation

Cultural relics are plaguedwithavarietyof diseases. Disease investigation is an important part of current situation investigation. For example, for metal relics, corrosion is the main disease. Key conservation methods including de- rusting, cleaning,surface passivation, corrosion inhibition and surface pro- tection are used. Ancient paintings, especially murals, suffer from dozens of diseases, such as cracks, partial paint exfoliations, deep loss, efflorescence, sootiness, hollow protuberance, fissure and mildew. As for cultural relics made of stone, problems such as structural cracks, bedding plane cracks, weather- ing cracks, water erosion, biological wind erosion, exfoliation, flake peeling and diseases caused by humans always exist. Therefore, to conduct disease investigation and find out the categories, locations and coverageof diseases is the basis of the subsequent conservation work of cultural relics. In disease investigation, it is of great importance to detect and separate the diseased area, and then mark it. Currently, this work is done manually. To assist dis- ease investigation, the major objective is to make use of image processing and pattern recognition methods to detect and separate the diseased area af- ter the cultural relics have been digitalized. Generally, diseased objects have special color, texture and shape. As an example, Fig. 5.2 shows some diseases from the Dunhuang murals. • Investigation Reports Generation

The content of a current situation investigation report includes: basic infor- mation (such as the impact of nature and of humans, the situation at opening- up and management, the investigation of historical conservation records), value evaluation, environmental investigation results, disease investigation results, and research results on materials or techniques and so on. Infor- mation management and editing techniques can provide quick access to the above investigation data by information retrieval. This contributes to the re- alization of a quick and convenient digitalized editing and management of the investigation report. When the investigation report is finished, it can be kept in digital form or be printed for filing. The digitally aided investigation of culturalrelicsneedstobesupported by many technologies, as illustrated in Fig. 5.3. Digital photography is mainly used to record the current situation of cul- tural relics for filing.Digital photography has been introduced in Chapter 2. Disease detection and marking mainly refers to the automatic identifica- tion and marking of cultural relics’ diseases. For a long time, the identification 94 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Fig. 5.2. Typical diseases of the Dunhuang murals. (a) Crack; (b) Partial paint exfoliations; (c) Mildew; (d) Efflorescence and marking of cultural relics’ diseases has been conducted manually. With the popularization of the digitalization of cultural relics, it becomes possible to identify and diagnose the diseases of cultural relics with the help of digital technology. The general process is: firstly, use intellectual image processing technologytoget the features of the disease; secondly, use pattern recogni- tion algorithms to conduct detection and identification and, finally, isolate the disease locations and mark them with corresponding tags. Digitally aided environmental investigation involves the choice of sensor and the transmission and storageof environmental data. Environmental sen- sors and sensor networks are the primary means for obtaining environmental data of cultural relics, such as temperature and humidity. There is only one cacheinearlier environmental sensors, and its data has to be retrieved man- ually. With the development of networks and sensor networks, long-distance automatic data acquisition and transmission has become the mainstream. 5.1 Digitally Aided Investigation 95

Fig. 5.3. Technical framework of digitally aided investigation

Digitally aided surveying and mapping is an important means of record- ing cultural relics’ spatial form. Normal digital mapping techniques include GPS, digital topographic mapping (total station and microcomputer) and close-range photogrammetry. With the development of digital technology, on the one hand, digital photogrammetry and laser scanners now can enable people to make high precision and high-resolution 3D observation. On the other hand, as the application of the digitalization of cultural relics spread and 3-D digital photogrammetry becomesmorepopular, peoplecandopho- togrammetry and draw maps directly on the digitalized cultural relics. What is mainly involved in the generation of the current situation inves- tigation report is the management and editing of digital information. And the most important content is the expression of metadata.

5.1.3 Key Technologies of Digitally Aided Investigation

Digital mapping and disease detection and marking will be detailed here. 96 5 Digitally Aided Conservation and Restoration of Cultural Heritages

• Digital Mapping

Mapping is an important part of recording the spatial information for her- itages. General mapping technologies consist ofGPS-based fixed-point mea- surement, total station based digital terrain mapping and close-range pho- togrammetry based relics mapping. GPS-based fixed-point measurement of cultural relics (Wang, 2002), (Feng et al., 2008) benefits much from GPS features, such as 24-h, high precision and full earth coverage. To conduct fixed-point measurement of cultural relics by GPS, one just needs to set up the GPS receivers respectively at the location of theculturalrelicandatthebasestations.After dozensofminutes of the reception of field data and dozens of minutes of lab computing, high precision coordinate data can beobtained conveniently. Digital terrain mapping based on a total station is used more frequently (Shasby and Carneggie, 1986). It mainly depends on the use of a total sta- tion and a microcomputer. Total station is a new measurement instrument composed of an electronic theodolite, an infrared range finder (or laser range finder) and microprocessor. It can measure angle and distance at the same time at an observation station and can also automatically calculate the co- ordinates and the elevation. Meanwhile, it can store a certain amount of data. For the mapping of ancient towns and ancient architectures, the gen- eral approach is to collect the data in accordance with the basic principles of cultural relic mapping and then process the data by using a mapping gener- ation system to finally produce a digital topographic map conforming to the requirements of conservation. The mapping technology based on close-range photogrammetry (Atkin- son, 1996) can measure the relics’ shape and geometric location precisely. It uses a variety of sensors to measure the surface imageof objects by short- distance acquisition approach based on RS technology. The size, shape, loca- tion, geometric location and spatial relationship of objects can be identified after the above data has been processed. And a variety of physical parame- ters can beobtained,such as a planar graph, a stereographic map, a sectional drawing, a perspective map, a structure map and contour map of the objects, as well as the coordinates of the objects. • Disease Detection and Marking

The main goal of disease detection and marking is to find out and mark the disease distribution of cultural relics. Some existing disease patterns should be firstly taken as samples. And pattern recognition algorithms are used for detection and segmentation. The features of cultural relics’ diseases mainly includecolor, texture and shape. For diseases having obvious color features, a color model evenly com- bined with a classification algorithm can be constructed to make classifica- tion. Frequently-used models include a single Gaussian model, a Gaussian 5.1 Digitally Aided Investigation 97 mixture model and color space histogram.

(1) Color Space Histogram:

disease[c] p(c/disease) = (5.1) Norm where, p(c/disease) is the probability density of disease color c. Norm is the total pixel number of color samples. disease[c] is the pixel number of diseases whose coloris c. A color histogram is usually used together with Bayesian classification algorithm.

(2) Single Gaussian Model:

1 T −1 1 − (c − μs) ΣS (c − μs) p(c/disease)= 1 · e 2 (5.2) 2π |ΣS| 2 where ΣS is the covariance matrix. Parameters μs and ΣS can be estimated like this: n n 1 1 T μs = cj; ΣS = (cj − μs)(cj − μs) (5.3) n n − j=1 1 j=1

(3)Gaussian Mixture Model(GMM):

k p(c/disease) = πi · pi(c/disease)(5.4) i=1 where k isthenumberofGauss and pi(c/disease) is the probability density of each Gaussian model. We can estimate the parameters of GMM using the Expectation-Maximization (EM) algorithm (Bilmes, 1998). Using Bayes’ theorem, we can get the posterior probabilities of a given pixel belonging to the disease and non-disease class as follows:

p(disease/c) p(disease)p(c/disease) = p(disease)p(c/disease) + p(non-disease)p(cp/non-disease) 98 5 Digitally Aided Conservation and Restoration of Cultural Heritages p(non-disease/c) p(non-disease)p(c/non-disease) = p(disease)p(c/disease) + p(non-disease)p(cp/non-disease) (5.5) The ratio of p(disease/c) to p(non-disease/c) can be used as a decision rule to detect the diseased region of the mural. As for diseases with obvious texture features, the texture feature extrac- tion method can be used to extract the features and then identifyandseg- ment them by using classification algorithms. Common texture features in- cludethose based on a gray level co-occurrence matrix, local gray statistics, fractional mode and that based on a Gabor filter. And as for diseases with obvious features in their shape, shape descriptors can be used to classify them. Shape, as one of the important visual contents of the image, has very intuitive features in 2D space. Generally speaking,the object in the image has a counterpart in the real world. Their projection in 2D space seems like a continuous region. Shapes are usually considered to be a region surrounded by a closed contour curve. Therefore, the description of the shapes involves the description of the shapes’ closed border and the region surrounded by the border. Currently, shape based object retrieval uses contour and region features of shapes. The above three features can be used separately to detect diseases, or be used together to improve the effectiveness of disease detection.

5.1.4 Typical Digitally Aided Current Situation Investigation System

A typical example of digitally aided mapping is the mapping of the caves of the Yungang Grottoes and Longmen Grottoes, the mapping of the stone carv- ing on the Dazu Beishan Mountain and Baoding Mountain. As the exquisite sculptures and paintingsintheYungangGrottoes have already become very fragile as a result of weather and because a slight contact with the surface may cause exfoliations, non-contact close-range photogrammetry mapping is in fact the best choice among a variety of solutions. From 1985 to 1988, in the practical research work contained in “The Application Research of Close- range Photogrammetry Mapping in Grottoes”, researchers have finished the mapping of the caves of the YungangGrottoes and Longmen Grottoes and that of the stone carvings on the Dazu Beishan Mountain and the Baoding Mountain. During the process, they have conducted a systematic studyona set of technical methods regarding the application of close-range photogram- metry mapping in grottoes, including the adaptability of the technology, the technical process, precision assessment, the form in which results are ex- pressed and quality control. The results of their research have passed inspec- tion at ministerial-level and won the second prize of the ScientificandTech- nological Advances Medal of the Construction Ministry and the third prize 5.2 Dynamic Environmental Monitoring ofCultural Heritages 99 of the National Scientific and Technological Advance Medal. Afterwards, the results have been widely applied in many famous grottoes in the country, as is illustrated in Fig. 5.4.

Fig. 5.4. Digital maps of buddhists of the YungangGrottoes

Murals disease investigation is also a typical investigation method.Zhe- jiang University and DunhuangAcademy have co-developed several disease detection systems. The conservation and environment monitoring system can provide current andhistoricaldata which is probed in the3Dmodel.The system makes it easier to find the location of the probe in the murals. It provides evidence for researchers to decide whether to take further measures or not. The investigation and resource management system is used to help the investigation, management and filing of murals and observe the trends of diseases. The currently developeddiseases investigation system has achieved a high degree of automation in murals disease detection. Fig. 5.5 is a sample mural and its disease detection results.

5.2 Dynamic Environmental Monitoring of Cultural Heritages

Dynamic environmental monitoring refers to real-time and continuous collec- tion of environmental information of cultural heritages. By analyzing these data, researchers can make conservation plans. Generally, environmental pa- rameters include temperature, humidity, lightandtheconcentrationofair pollutants. Research indicates that the surrounding environment of cultural relics has a great impact on their long-term conservation (Camuffo, 1998). For example, certain elements of cultural relics cannot be preserved for a long time in improper temperature, and great changes in temperature will easily lead to disintegration and exfoliation of relics. Relative humidity has a greater impact on relics also. Low humidity may cause wood artifacts to become dry and crack, while high humidity may lead to a variety of diseases such as warp, distortion, rusting and mildew. Air pollutants such as SO2 can generate a new compound with water vapor called sulfuric acid and corrode 100 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Fig. 5.5. A disease distribution marking map of the Dunhuang Mural. (a) The mural in the 260th cave, the Dunhuang Grottoes, China; (b) The marked diseases, A: Sootiness, B: Flaking disease cultural relics. Microbial enzymes, a powerful biological catalyst, can cause bad damage to organic relics. Ultraviolet light has a strong destructive effect on various kinds of relics, such as reducing the strength of organic relics and causing the fading of paint color. Therefore, it is very important to monitor the environment of cultural relics. Compared to traditional environmental monitoring approaches, dynamic environmental investigation can provide a real-time report of environmental information, so that conservators can get the latest data conveniently and make responses more rapidly.

5.2.1 Process and Technical Framework of Dynamic Environmental Monitoring

Dynamic Environmental Monitoring includes three main steps, information collection, analysis anddocumentation. Inthefirst step, we obtain the environmental information of cultural relics by using environmental auto-acquisition techniques. According to the environmental characteristics and the goal of conservation, we transmit the information to thedatacenter after formalization. In the second step, some digitally aided methods are adopted to analyze the environmental information data and give some forecasts and warnings. In the third step, we keep the environmental information of cultural relics in a safe place, for example, saving the information in the database on a 5.2 Dynamic Environmental Monitoring ofCultural Heritages 101 computer, or documenting the data on paper files. In addition, we analyze and process the environmental information and, if necessary, build an exhibition system to display these data. Theabove mentioned process is illustrated in Fig.5.6.

Fig. 5.6. Technical process of dynamic environmental monitoring

There are mainly three kinds of technologiesinvolvedinthedynamicenvi- ronmental monitoring system, sensing,transmission,analysis and exhibition. The sensing technology of environmental information deals with sensing techniques that can probe the environmental information of the relics. There are various types of sensors, including temperature, humidity, gas, light and so on. The transmission technologyof environmental information handles trans- mitting issues. Off-line methods are usually used traditionally. However, in dynamic monitoring systems, the information needs to be transmitted on-line and in real-time. Wireless transmission is a good choice for transmission in heritage sites, as wiring will probably affect or damage the sites. Low energy transmission is also an important issue in transmission, as the sensor nodes in sites probably have no power supply but a battery. Data formalization should be done for transmission, and is achieved by metadata expression. 102 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Theanalysis and exhibition technology dealswith storage, analysis and forecasting, and information exhibition issues. Based on statistical methods, the environmental data can be used for analysis and warning for heritage sites. Another important usage is information exhibition, e.g. temperature curve exhibition. The technical framework is illustrated in Fig. 5.7.

Fig. 5.7. Technical framework of dynamic environmental monitoring technology

5.2.2 Key Technologies of Dynamic Environmental Monitoring

Key technologies mainlyinclude dynamic environmental sensing, environ- mental information transmission, and environmental data analysis and dis- play. • Dynamic Environmental Sensing Sensors are the devices sensing the environment. Digital sensors are more convenient than traditional analog sensors. It is important to choose an ap- propriate sensor for the environmental monitoring system. To meet the needs 5.2 Dynamic Environmental Monitoring ofCultural Heritages 103 of dynamic monitoring of different cultural relics, it is necessary to analyze the applicability of sensors (Sheng et al., 2005). The range: Different sensors have a different measurement range. For example, in terms of the carbon dioxide sensor, the range varies from 0 ∼ 2×103 ml/m3, 0 ∼ 5×103 ml/m3, to 0 ∼ 5×104 ml/m3.Therangeof the sensors for the environment to be monitored should be considered carefully in the dynamic investigation of the cultural relics. In order to get the best measurement precision, a sensor of appropriate measurement range should be chosen in the dynamic investigation. The precision: Different sensors have different characteristics in respect of their precision. For example, the Sensirion SHT series of temperature and humidity sensors have only an error of 1.8% to 3% in humidity, and 0.3 ◦C to 0.4 ◦C in temperature. Meanwhile, it is worth noting that the rangeandthe precision sometimes are closely related. For example, the Telaire 6004 series carbon dioxide sensor’s precision will reduce as its range increases. The working conditions: Sensors generally have their particular working environment. The Telaire 6004 series carbon dioxide sensors work in an en- vironment with a temperature from 0 ◦C to 50 ◦C and humidity from 0% to 95%. Even within the scope of the requirements, some of the characteristics of sensors, such as precision, may also change with the working environment. For example, the humidity precision of the Sensirion SHT75 sensors will drop directly when the humidity is higher than 90% or lower than 10%. Therefore, in choosing or designing the sensors, we should consider whether the sensors canworkinthespecific environment and whether sensors can meet some particular requirements of a specificsite. • Environmental Information Transmission Real-time environmental information in cultural heritagesitesisfrequently quite difficult to obtain, as many of them are located in the wild where power and communication infrastructure are often unavailable. The lack of power supplyand network access at culturalheritagesitesdoes not only come from the difficulty of deploying wire connections in the wild, but also comes from the constraints of heritage conservation policy, since deploying wire connections may destroy the original scene at heritage sites. Therefore, Wireless Sensor Network (WSN) technology, in which all sensing devices use batteries as their power supply and use low power wireless communication to relay data, are very attractive in heritage environment monitoring. In order toprolong battery life, energyefficiency, which mainly involves routing and Media Access Control (MAC), is one of the most important considerations in system development. (1) Routing: Energy-efficient routing mainly aims at finding an appropri- ate path for data delivery while maximizing the battery lifetime of devices involved in data relaying. The energy-aware multi-path routing protocol for wireless sensor networks (Shar, 2002) is a typical energy-efficient routing pro- tocol,which establishes multiplepaths between source nodes anddestination 104 5 Digitally Aided Conservation and Restoration of Cultural Heritages nodes. Theprotocol assigns a certain probability to a certain path accord- ing to the energy consumption and remaining energyof the nodes, so as to prolong the lifetime of the entire network through balancing the energy consumption of all nodes involved in data relaying.Thefollowingformula denotes the cost of each communication path in the protocol.

CNj ,Ni = Cost(Ni) +Metric(Nj,NNi) α β (5.6) Metric(Nj,NNi)=eijRi C N where Nj ,Ni is thecostwhen thedataistransferred from source node j to destination node Ni. Cost(Ni) is the cost of communication of the transfer α fromnode i to the destination. eij is the energy consumption of the direct β communication from node Nj tonode Ni. Ri is the residual energy of node Ni. The node will remove paths with extra high cost, the condition of the node Nj inserting Ni into its routing table FTTj is:

FTTj = {i|CN ,N  α(min(CN ,N ))}, α > 1 (5.7) j i k j k

The probability of the node Nj selecting thenode Ni asthenextjump node is

1/CN ,N P j i Nj ,Ni = (5.8) 1/CNj ,Ni k∈FTTj

Then, node Nj recalculates the Cost(Ni) of itself. N P C Cost( j)= Nj ,Ni Nj ,Nk (5.9)

k∈FTTj (2) MAC. Because the size of environmental data is usually very small, the data transmission time of environmental monitoring devices will be quite short. Energy-efficient MACprotocolsgreatly reduce the power consumption of the devices through putting the device into sleep mode when there are no datatotransmit.These protocolscanberoughlycategorized into two types: synchronized and asynchronous. For synchronized protocols, theclock of all devices is synchronized periodically and they will wake up or sleep synchronously for information exchange, as shown in Fig. 5.8(a). For asyn- chronous protocols, the receiver wakes up periodically and checkswhether any one will deliver data to it. If there are no data available, it will sleep immediately, or it will start receiving data, as shown in Fig. 5.8(b).Readers can refer to a survey of energy-efficient MAC protocols in wireless sensor networks for further information (Demirkol, 2006). The unstable feature of wireless communication greatly lowers the reli- ability of environmental data delivery in cultural heritage sites. However, environmental data are very important for studying the role of environment 5.2 Dynamic Environmental Monitoring ofCultural Heritages 105

Fig. 5.8. Working mechanism of (a) synchronized and (b) asynchronous MAC protocol data in the deterioration of aculturalheritage. Therefore, reliable data trans- mission mechanisms should be employed to avoid data loss in transmitting. Traditional reliable transmission protocols, such as TCP/IP, do not take energy-efficiency into consideration. They require the source and destina- tion nodes to do end-to-end re-transmission, thus wasting a lot of energy. Therefore, hop-by-hop re-transmission is frequently employed to ensure data transmission reliability while conserving energy. Suppose node A transmits data to C by B. In traditional end-to-end re-transmission, if an error occurs in B to C transmission, A should re-transmit the data again to B, then B to C, as is illustrated in Fig. 5.9(a). In hop-by-hop re-transmission, the data will be pushed into a cache. Once there is a data transmission error, B will not order A to re-transmit the data, but will retrieve the data from the cache, and directly re-transmit the data to C, as illustrated in Fig. 5.9(b). • Environmental Data Analysis and Display Technology

After the environmental data are collected and transmitted, they should be analyzed and exhibited. For storage, data volume is usually very large due to the precision re- quirement. Therefore, a high-performance database should be used to store data. Meanwhile, in order to reduce database maintenance costs and improve speed, we should consider storing the data in separate tables. In addition, we should pay more attention to the portability and scalability of the database inthe design. 106 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Data retransmission protocols. (a) Traditional end-to-end re- transmission; (b) Hop-by-hop re-transmission

In order to meet the requirements of different users and different applica- tions of cultural relics, the information display interface should be carefully designed to facilitate data browsing. B/S (Browser/Server) architecture is widely used because of its flexibility, as we do not need to install any dedi- cated program in the client computer. At the same time, displaying data in a chart is helpful for users to un- derstand and analyze the data. We can use charting class libraries such as ZedGraph (ZedGraph, 2009) to dynamically plot the real-time environmental data in the browser.

5.2.3 Typical Dynamic Environmental Monitoring System

Presently,alotof conservation institutions and museums have taken mea- sures to carry out environment monitoring of cultural relics. For instance, to observe the environmental changes, the Dunhuang Academy has set up a variety of sensors such as air temperature and humidity probes, rock temper- ature and humidity probes and a carbon dioxide probe in some of the caves of the Mogao Grottoes. And the Jiayuguan Great Wall Museum, Gansu, China also collects information about the temperature and humidity of the relics storehouse twice a week and has made a preliminary analysis of the rules of temperature and humidity changes in different seasons. 5.3 Digitally Aided Restoration of Cultural Heritages 107

Themicroclimate environmental monitoring system developedbyZhe- jiang University integrates a wireless sensor network with a long-distance wireless communications network and wired LAN, thus greatly expanding the sensing and data transmission rangeof the wireless sensor network and effectively avoiding the problem of deploying lines in the Dunhuang Grottoes. As the system supports an intermittent working mode, it has a long life-time and can work for more than one yearonthepoweroftwoAA batteries’. Since the Mogao Grottoes have a large and complex terrain, the wireless sen- sor network is used in caves and the long-distance wireless communications network is used outside caves. Finally,they are connected to a wired LAN. The system will be introduced in more detail in Chapter 8.

5.3 Digitally Aided Restoration of Cultural Heritages

The restoration of cultural heritages mainly includes two aspects. One is to remove impurities in the relics. The other is to splice the fragments, repair the defects and restore the relic to its original state, including color, texture or shape. It must be ensured that the desired effect can be reached and the restoration can be achieved successfully, for the restoration is usually irreversible. Digitallyaided restoration is blessed with inherent advantages. It can simulate expected results with different restoration methods. For example, the Virtual Restoration System of Excavated Cultural Relics funded by the Japanese Asahi Glass Co. in 2000 has assisted the mosaic of a large number offragments ofQin terracotta warriors and horses with digital technology and improved working efficiency (Zheng, 2009; Zheng and Zhang, 1999).

5.3.1 Process and Technical Framework of Digitally Aided Restoration

Digitally aided restoration of cultural heritages mainly includes the formal- ized expression of the domain knowledge, virtual restoration planning, virtual restoration, restoration effect exhibition and evaluation, and physical restora- tion. Firstly, we should express the restoration knowledgeinaformalized form. Digitally aided restoration is a technology closely integrated with its appli- cation, which should make full use of the knowledge of cultural heritages. For example, to restore the fading color of murals, the domain knowledge of murals, such as the content of them and the age of the drawing are all required. Secondly, on the basis of the domain knowledgeof cultural heritages, a corresponding restoration plan, including different restoration strategies for different types of heritages, should be studied. For example, in order to repair 108 5 Digitally Aided Conservation and Restoration of Cultural Heritages the cracks in ancient paintings, we should classify the types of crack, since the restorationmethod will differ as the cause of thecracksvaries. Thirdly, virtual restoration is taken in the computer. After the effect is exhibited and evaluated, the experts can judgetheeffectiveness of the restoration, or it can serve as guidance for physical restoration. As is noted above, the process of digitally aided restoration of cultural heritages is illustrated in Fig.5.10.

Fig. 5.10. Process of digitally aided restoration of cultural heritages

There are three aspects of digital restoration of cultural heritages, includ- ing color restoration, texture restoration and shape restoration. Color is the tint information, texture is the structural property of the surface, and shape is the outline model information. Take murals as an example: to splice the broken muralsand restore their original shape belongstoshape restoration, while to restore the color of faded murals belongs to color restoration. The murals texture restoration mainly refers to the restoration of the diseases which cause damagetothesurface of murals, such as partial pigment loss, sootiness and mildew. Color restoration focuses mainly on repair to the fading and surface dirt removal. And the color processing and imagesegmentation techniques are used for color restoration. In 5.3.2 we will describe color restoration in detail. Texture restoration focuses mainly on virtual restoration of surface de- tachment and virtual restoration of cracks. Texture synthesis and inpainting 5.3 Digitally Aided Restoration of Cultural Heritages 109 are basic technologies used for texture restoration. Texture synthesis is widely used in texture restoration. Inpainting technologyoriginally referred to the restoration work on damaged old paintings carried out by museum restoration experts. Bertalmio et al. (2000) firstly presented the concepts and methods of digitalized inpainting. Subsequently, the algorithms and technology in respect of inpainting, photo repair and removal of objects have been created. Shape restoration focused mainly on relic fragment mosaic and surface shaperepair. Shapeexpression provides a formalized expression method for the shape surfaces in digital form. And surface mosaic techniques are the basic technologies for relic fragment mosaic. The technical framework is illustrated in Fig.5.11.

Fig. 5.11. Technical framework of digitally aided restoration

5.3.2 Key Technologies in Digitally Aided Restoration

Digitally aided virtual restoration of surface color, virtual restoration for cracks and exfoliations, and mosaic of relic fragments will be covered here. • Digitally Aided Virtual Restoration of Surface Color

Color fading is a common disease in relics, especially in old paintings. Coun- tries like Greece, Italy, Japan and China have done a lot of work in this field. 110 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Zheng et al. (1999) has made a lot of research on the color restoration tech- nologyof thearmorsof the Qin terracotta warriors and horses. Michail et al. (2000), from Aristotle University, have proposed a series of methods for the restoration offaded paintings. Mauro et al. (2000) from the University of Florence, Italy, have presented a virtual restoration method for art images suffering oxidative discoloration. Basically, these methods are based on the same assumptions as illustrated in the following: Assume that all disposals are dealt with in the CIELAB color space and some patches have been repaired manually. si (i = 1, 2, ...,n) are the repaired patches, the number of which is N; xi (i = 1, 2, ..., n) are the corresponding un-repaired patches. mˆ si and mˆ xi respectively stand for the average value of si and xi in the CIELAB color space. mˆ s and mˆ x stand for the average color value matrix of all patches; then define Δmˆ = mˆ s − mˆ x and sˆ isdefined as the result of restoration. (1) Sample Mean Match: The method classifies each pixel of the paintingsandfinds mˆ xi, which is nearest to it in the color space, and then obtains the repair result sˆ through the following formula:

sˆ = x + Δmˆ i, (5.10) where Δmˆ i is column i of Δmˆ . Direct and simple, this approach is, in fact, a linear interpolation algorithm and it can obtain better results when the patch number is large. (2) Linear Transform: s, standingfor each pixel in the restored image sˆ, can be expressed by the following formula:

s = f(x) = (A + I)x (5.11) I stands for the 3×3 unit matrix. A isa3×3 coefficient matrix. The off- set vector is d = Ax and A is worked out bythePolynomial Regression Calculation Method. (3) Iterative Closest Point (ICP) Transform: ICPisaneffective algorithm for the registration of two 3D data sets. Assume that there are two 3D data sets si (i = 1, 2, ..., n) and xi (i = 1, 2, ..., n) and si corresponds to xi,thenevery pixel of the restoration result can be obtained by the following formula:

s = f(x)=Rx + d (5.12) R isa3×3 rotation matrix. d is a 3×1 offset vector. The complexity of the ICPalgorithm is O(N 2). Therefore, the main problem of this algorithm is how to obtain the ICP algorithm more rapidly, especially when N is large. 5.3 Digitally Aided Restoration of Cultural Heritages 111

(4) White-point Transform. This method assumes that color-changed ancient paintings can be approx- imately regarded as normal paintings under a dark light. And the features of this dark light are decided by the reference white vector, namely wXY Z . Therefore, the visual differences between color-changed and unchanged works are decided independently by the difference of white spots generated through the conversion from CIEXYZ color space to CIELAB color space. Assume sLAB is unchanged painting in the CIELAB color space, then the pixel of corresponding discolored paintings is represented by xXY Z . The estimated restoration results can be obtained throughthefollowingformula:

s T { } ˆLAB = xXY Z ; xXY Z (5.13) Here, T {.; .} stands for the nonlinear transformation from CIEXYZ to CIELAB. • Virtual Restoration for Cracks and Exfoliations

Cracks and exfoliations of paint layers is another common problem plaguing culturalheritages, such as painting. To repair cracksand paint losses is a regular work of cultural heritage conservation. Due to the rapid development of image processing technology, a great amount of research work regarding the restoration of cracks and exfoliations has already been proposed. (1) Interactive Repair of Cracks. There are two steps in the virtual repair of cracks. The first step is to detect and divide the crack region. The second step is to carry out vir- tual repair using acrackfilling algorithm. The crack region can be detected semi-automatically or fully automatically. For the semi-automatic detection method, users need to select a seed point in cracks and then use the track- ing algorithm to split the current cracks. For the fully automatic detection method, cracks can be detected through a suitable filter. As the cracks are similar to small strokes in their texture, they may be detected at the same time. Therefore, information like shape and color should be used to separate strokes. Barni et al. (1998) have presented a technique for repairing the cracks of ancient paintings based on the interpolation value. By his method, the crack region is selected through studying the relation of the crack pixels and its surrounding pixels, and then it can be eliminated. As the system is unable to accurately identify the cracks and the dark lines and strokes, users firstly need to select a point in thecrack region, and then thesystemwill automatically track and remove cracks. As the cracks of art paintings are usually darker than their background, and they are mostly identical in the gray-level, therefore cracks can be tracked by two features: the identicalness of the absolute gray value and the gray-level similarity of cracks. 112 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Firstly, specify a point on the cracks manually, and then use an eight neighborhoods algorithm to iterate in accordance with the following condi- tions.

|f (A) − f (Bi)|  T (5.14) where f(A) is the gray-level of the manually designated point A; Bi(i = 1, 2, ..., 8) are 8 neighbors; T istheadjustablethreshold,and the resultob- tained is a crack. In case of meeting bifurcation or gaps, the pixels generated by the algorithm are no longer adjacent, namely a bifurcation is detected. As for gaps, a search of a certain distance along the edge in the current direction may be made. And as for the tracked cracks, they can be deleted by the method of interpolation. Here, the Shepard interpolation is used. The interpolation surface is defined as:  μ rj (x,y) 1 − R f (xj,yj) j∈P u (x, y)=  μ (5.15) rj (x,y) 1 − R j∈P Among which:

⎧ ⎨⎪ 2 2 2 2 (x − xj) + (y − yj) , if (x − xj) +(y − yj)  R; rj (x, y) = ⎩⎪ 2 2 R, if (x − xj) +(y − yj)  R. (5.16) For the radius R, a large value should be used for the central part of the cracks (to ensure that there are enough undamaged pixels used for restoration pixels), and a small value for cracks at the border (to ensure that there are large differences between the crack’s pixels and its surrounding pixels). (2) Automatic Detection and Restoration. Although the interactive cracks repair method can locate cracks accurately, its efficiency is quite low. Researchers from the Aristotle University of Thes- saloniki, such as Ioannis Giakoumis, have presented the automatic crack de- tection and repair algorithm (Giakoumis and Pitas, 1998). Firstly, it extracts the local minimum value byusing mathematical morphology, thetop-hat op- erator (it mainly aims at cracks and small black strokes) and then separate cracks and black strokes through the MRBEF neural network in the HVS space. Finally, fill the cracks and strokes throughamediafilter, trimmed filter and controlled anisotropic diffusion technologies. The morphological operation, top-hat, is used to detect theturtlecracks. After the top-hat transformation, a threshold is used to isolate cracks which are to be refined later. In order to extract cracks as much as possible, users can modifythefollowing two parameters. 5.3 Digitally Aided Restoration of Cultural Heritages 113

a) Type and size of structural element b. Generally, it is a square or a circular structural element. Size is set according to the width of detected cracks. b) Numbers of corrosion and expansion operations. After the enhancement through top-hat transformation, it is necessary to separate cracks from the background image. As there are differences between cracks and the background image after the top-hat transformation, a threshold can de- fined to separate them. The threshold can be set throughtheOtsu method or a certain simple image statistics algorithm. In order to isolate the cracks more effectively and reduce the impact of noise, the images are often divided into several small pieces, with each one separately using the above method to calculate its threshold. However, the choice of the size of small regions often has a great impact on the result. (3) Restoration Technique for Exfoliation Blanks. As for blanks caused by exfoliations, it is generally proposed that one should separate the blank regions first and then fill them according to empirical knowledge. Rosa et al. (2001) has presented a better way to automatically segment the blank regions of ancient oil paintings. They use different restora- tion technologies to repair them and then display the results. Thetraditional image segmentation algorithm, based on regional growth, firstly designates a pixel in the damageregion chosen by human recognition and then determines whether this pixel p should be added to the region in accordance with the pixel gray level and the differences in the average gray values of pixels.

p ∈ (Sm − Δ, Sm + Δ)(5.17) However, this approach is not suitable for the division of ancient paintings, as the Human Vision System (HVS) information of color should be taken into consideration, for which human recognition is more suitable and is able to obtain better results. Definition:

A1 A2 A3 r = 2 + 2 + 2 + A4Gm (5.18) σH σV σS 2 2 2 where A1,A2,A3 are the inspiration constant values and σH σV σS are the HVS mean square errors of each exfoliation region, respectively. Gm isthe gradient information of consideration pixel p. This consideration pixel p will be added to the exfoliation region when r increases. Attheverybeginning, as pixels in the same region have the same features, the value of r increases and tends to reach the border. However, for the sake of the gradient, when thevalue of r continues to increase and cross the boundary, it will decrease and the region will thus stop growing. After the broken regions are separated, it can be restored by using color replacement technology discussed in the last chapter. It can also be repaired by traditional methods, namely interactive repair by artists. 114 5 Digitally Aided Conservation and Restoration of Cultural Heritages

• Mosaic of Relic Fragments

Due to natural and human factors, such as earthquakes, improper excavation, etc., many cultural relics are broken. How to mosaic and restore these broken cultural relics is an important issue faced by conservators. The Northwest University of China has done a lot of work on the mosaic of relic fragments. (Ru, 2004). Advantages include: (1) A virtual mosaic does not change the physical status of the relics, whichisalsorecoverableifsomemistakesaremade. (2) The computer can easily simulate the mosaic result, which is needed formosaictasks. (3) For large-size fragments, it is inconvenient to mosaic them. However, in the computer it is easier to move or rotate the digitalized fragments. Although field excavation and research on the relics is irreplaceable, vir- tual restoration greatly improves the efficiency. Therefore, it is a beneficial supplement to traditional restoration work. Mosaic software displays relic fragments on the monitor, and then the user may use interactive devices, such as mouse, keyboard, data glove,tracker,to move fragments to observe whether the fragments can be spliced together, and then adjust (mainly translation and rotation) the relative position of relic fragments, show the mosaic result on the monitor, and finally make a human judgement about whether the mosaic is appropriate or not. Software mainly includes RapidForm, PolyWorks, GeoMagic, and a specially developed soft- ware, Silicon Graphics Inventor, from the Asahi Glass Foundation. Recently, some automatic software was also developed for relic fragments mosaic, which will compute the matching degree between fragments automatically. Based on their shape, mosaic software can also be divided into that ori- entedat2Drelicsorat3Drelics.A 2D mosaic of cultural relics is mainly used for flat objects’ mosaic such as paintings, while a 3D mosaic is mainly for objects with 3D shapes, such as pots and statues. 2D mosaic is based on 2Dmatching techniques, which can be classified according to different stan- dards, for example whether shape matching is based on the border or the internal part of the shape. The shape matching approach can be classified into two types. The first type searches invariants in different transformations. (a) Similarity invariants: distance, moment, angle, roundness and Fourier descriptors. (b) Affine invari- ant: simpleratio,length, area surrounded and improved Fourier descriptor. (c) Perspective invariant: cross-ratios and its extension. The second type can deal with more complex shape transformation. It minimizes the matching errors by searching the corresponding local features between the objective and the model. In order to obtain the minimum value, people use many methods, such as a generalized Hough transform, dynamic programming, neural networks, a deformation template, a genetic algorithm, as well as analytical methods. The mosaic algorithm of 3D objects is much 5.3 Digitally Aided Restoration of Cultural Heritages 115 more complicated. Ding et al. (2001) has put forward a typical mosaic algo- rithm for 3D objects, the specific process of which is as follows. (1) Fragments digitalization. Obtain shape data of fragments and store them. (2) Preprocessing of fragments data. Extract some matching features as different algorithms require different feature data. (3) Fragments matching. Search the matching fragments. (4) Identify the results. In the matching result set, identify the correct subset and make necessary amendments to ensure the correctness. (5) Mosaic. Mosaic the matching fragments into a larger one. (6) Repeat step (2)∼(6), until no more fragment needs to be matched or the user stopstheprocess. (7) Output the restoration result.

5.3.3 An Introduction to Typical Application of Digitally Aided Conservation and Virtual Restoration

The digitally aided conservation system for cultural heritages has already been put into wideuseinthewholeworld. Japan, the European Union and China have done a lot of work in this field. These systems can be divided into three categories. (1) Information management. It inputs cultural heritages’ information into the computer for retrieval and sharing. (2) Digitally aided restoration. It is mainly applied to finish various kinds of virtual restorationv in the computer, such as a virtual restoration system for archaeological findings, a computer-aided cultural relic restoration system, a computer-aided murals restoration system, and so on. (3) Diseases mechanism visualization and digitally aided research systems. It is mainly applied to displaydiseases and their trends, and provide support for appropriate conservation measures, such as the computer-aided mural conservation and evolution simulation system. • Virtual Restoration System for Unearthed Relics

This project was funded by the Japanese Asahi Glass Co., Ltd. in 2000 (Zheng, 2009). The team included archaeologists, computer scientists and museum artists. A computer graphics interface, multimedia and virtual real- ity were used to research the probability of restoring some archaeological find- ings in the Lintong terracotta warriors and horses museum in Xi’an, China. In 1970 a farmer found a lot offragments of terracotta warriors and horses when he was digging a well. Since then, a world famous cultural heritagesite has been found and the museums at the site are still under excavation. A group consisting of 80 people has spent more than 20 years on the site and excavated about 3000 statues (The total number is estimated to be 8000). 116 5 Digitally Aided Conservation and Restoration of Cultural Heritages

It is believed that the work efficiency and productivity would have been im- proved by using computer technologies. In this project, three objectives are expected to be achieved in particular. Digitalize therelics: Theteamhas useddigital images and a laser range finder to take photos of each object and determined the location of each object in its excavation unit. They have also established a database including information about all relics discovered. In this way, the images of the relics and the site can be seen from various angles and perspectives permanently, while the original artifacts still remain in the position in which they were discovered, without being disturbed or destroyed. Use virtual restoration techniques to display and assemble fragments in the virtual space as well as providing technical guidance for professionals. The interface of software should be able to display a virtual fragment from all angles andbeable to reconstruct. During this process, common points on different fragments should be found and then brought together by the soft- ware. The restoration also includes taking colors sample from the fragment (or from its surrounding soil) for texture mapping. Virtual restoration, without the problems of weight and binder, can be repeated (to reduce the possibility of damage and creating an incorrect mosaic) andbeoperatedremotely. Display virtual restoration results on the Internet or in a museum. Real objects are still retained in their original status, while high-resolution 3D data of these objects are displayed without moving those relics. In this way, virtual relics have replaced the real ones and images have become the preferred method for display. And audiences can observe and appreciate them more carefully and freely. • EROS (European Research Open System)

The aim of developing EROS (Genevi``eve, 2003) is to manage various kinds of digital documentation. It is designed for dealing with the analysis of mu- seum collections. These analysis data come from laboratories, or conservation and restoration departments of museums. The information, mainly technical data, includes search terms, research reports, restoration reports and digital data (such as quantitative analysis, spectroscopy, drawings, chemical formula, ultraviolet/infrared/ramp-rays photos and scanning electron microscope pho- tos). The database also includes management data, such as the tracking of artwork catalogues, restoration history and regular investigations of collec- tions. Recent developments of the system involve automatic content recogni- tion of targets, display ofgeographical locations, panorama display, as well as the exhibition of multi-spectral and 3D models. In 1990, C2RMF built a multilingual database called Narcisse, to manage all the scientificdocumentsgathered since the establishment of the conserva- tion and restoration department of the Louvre in 1931. In 2001, with the help ofHP(Hewlett-Packard Development Company), theopensourcedatabase EROS was built. Thishasmade the museum’s research laboratories and 5.3 Digitally Aided Restoration of Cultural Heritages 117 conservation center able to effectively handle huge records, multi-language metadata (the latest languages include Estonian and Greek) and ultra-high- resolution color and multi-spectral images. In addition, it can also be used for processing, such as automatic image content identification and new re- trieval methods. Presently, a total of 56,000 artworks, 300,000 photos and X-ray pictures, 10,000 copies of technical reports and 5,000 3D objects, are available there online. By using the XML standard, one can exchangedata with different platforms and use them easily. • Computer-aided Relics Restoration System

The system was developed under the instruction of Professor Zhou Mingquan from Northwest University, China. It focuses on the complementary match of broken objects, the smooth automatic stitching of broken objects and the automatic integration of broken objects. It is applied in the restoration pro- cess of relics, assisting the conservators to finish their work. By digitalization of a 3D scan, the system applies the shape mosaic and matching technique for relic restoration. It studies several techniques, e.g. spatial surface con- tour extraction, feature points extraction and matching. These researches have provided a scientific basis for the restoration and imitation of cultural heritages. With the help of the terracotta warriors and horses museum, it succeeded in the virtual restoration of broken terracotta warriors and horses. Restoration sample resultisillustrated in Fig. 5.12.

Fig. 5.12. Process of computer-aided relics restoration system (With permission from Mingquan Zhou) 118 5 Digitally Aided Conservation and Restoration of Cultural Heritages

• Computer-aided Mural Restoration and Evolution Simulation System

The computer-aided mural restoration and evolution simulation system (Shi and Lu, 2005) is used to assist the artist to imitate the Dunhuang murals. Using the knowledge and experience of the artist and Dunhuang experts, judgments have been made on whether the physical and chemical fading evolution is right. The virtual restoration experiments include: the fading evolution simulation of a supporter in the 205th cave, the fading evolution simulation of musicians on the northern of east cliff, local diseases simulation on the south wall in the 260th cave, the color restoration in the 205th cave, the restoring results of musicians on the northern of east cliff in the 205th cave, which have provided significant materials for further discussion and research by experts. The fading evolution simulation of a supporter in the 205th cave is illustrated in Fig. 5.13.

Fig. 5.13. Fading evolution simulation of supporter in the 205th cave

5.4 Summary and Prospects

In this chapter, we discussed digital conservation and restoration technolo- gies of cultural heritages, including the digitally aided current situation in- vestigation, the dynamic environmental monitoring,and the digitallyaided restoration of cultural heritages. The application of these technologies im- proves working efficiency greatly. Through these technologies, a huge amount of data of different types can be stored for the convenience of retrieval and identification of some typical diseases. What’s more, the environment of the relics can be monitored dynamically, so as to provide data and references for the choosing and implementation of conservation measures, to assist the restoration of relics in the real environment (the mosaic and restoration of broken relics) as well as the virtual restoration of broken relics. Digitallyaided conservation and restoration technology has achieved pre- liminary results but is not yet systematized. And digital technologyhasshown its advantages in the conservation and restoration of cultural relics, although it is just at the beginning stage. In the assessment of cultural relics, with the References 119 digitalization of cultural relics and the development of networks, remote value assessment of cultural relics will become the main development trend. Asfor the investigation of cultural relics, status investigations on digital artifacts will be widely applied and the level of automation technologies will also be improved. For example, digitally aided mapping will be developed towards real non-contact mapping, anddisease investigation will be automated to a greater degree. Dynamic environmental monitoring will also become wireless, smaller and self-organized. In the formulation and implementation of cultural relics’ conservation plans, digital technologies will also play a greater role, and the problems which remain unsolved will also be solved gradually.

References

Atkinson KB (1996) Close Range Photogrammetry and Machine Vision. Whittles, USA Barni M, Bartolini F, Cappellini V (2000) Image Processing for Virtual Restora- tion of Artworks. IEEE Multimedia 2:34–37 Bertalmio M, Sapiro G, Caselles V, et al. (2000) Image in-painting. In: Pro- ceedingsof International Conference on Computer Graphics and Interactive Techniques, New Orleans, Louisiana, USA 417–424 Bilmes JA (1998) A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models. Inter- national Computer Science Institute. UC Berkeley, Tech. Rep. TR-97-021 Camuffo D (1998) Microclimate for cultural heritage (ndevelopments in atmo- spheric science). European Commission, Environment and Climate Research Programme 23. Elsevier, Amsterdam Demirkol I, Ersoy C, and Alagoz F (2006) MAC protocols for wireless sensor networks: A Survey. IEEE Communications Magazine 44(6):115–121 Ding XF, Wu H, Zhang HJ (2001) Review on shape matching. Acta Automatica Sinica 27(5):678–694 (in Chinese) Feng M, Ze L, Wensheng Z, et al. (2008) Research and application of spatial information technology on grand canal of China. In: Proceedings of 2008 IEEE International Geoscience and Remote Sensing Symposium (3):7–11 Giakoumis I, Pitas I (1998) Digital Restoration of Painting Cracks. In: Proceed- ings of the IEEE International Symposium on Circuits and Signals, IEEE, Beijing 269–272 Genevi`eve A, Christian L (2003) EROS: European Research Open System. In: Proceedings of International Cultural Heritage Informatics Meeting. Ecole´ du Louvre, Paris, France Pappas M, Pitas I (2000) Digital Color Restoration of Old Paintings. IEEE Trans- actions on Image Processing 9(2):291–294 Rosa AD, Bonacchi A, Cappellini V, Barni M (2001) Image Segmentation and Region Filling for Virtual Restoration of Art-Works. In: ProceedingsofICIP 2001, IEEE, Thessaloniki 1:562–565 Ru SF (2004) Re-assembly of Fragmented Cultural Relics, Phd, XiBei Univer- sity(in Chinese) 120 5 Digitally Aided Conservation and Restoration of Cultural Heritages

Shar RC, Rabaey JM (2002) Energy aware routing for low energy Ad Hoc sensor networks. In: Proceedings of IEEE Wireless Communications and Networking 2002, Orlando, USA 1:17–21 Shasby M, Carneggie D (1986) Vegetation and terrain mapping in Alaska us- ing Landsat MSS and digital terrain data. Photogrammetric engineering and remote sensing 52(6):779–786 Sheng LM, Li JZ, Chen Y, Zhou HS (2005) Wireless Sensor Network. Qinghua University Press, Beijing (in Chinese) Shi XF, Lu DM (2005) Colorimetric and Chemical Modeling Based Aging Sim- ulation of Dunhuang Murals. The 5th International Conference on Computer and Information Technology 1:570–574 Wang ZJ (2002) Application of 3S-technology and its integration in protection of cultural relics. Sciences of Conservation and Archaeology 14(2):52–58 (in Chinese) ZedGraph (2009) http://zedgraph.org/ Zheng JY (1998) Virtual Recovery and Exhibition of Heritage. IEEE Multimedia 7(2):7–10 Zheng JY, Zhang ZL (1999) Virtual Recovery of Excavated Relics. IEEE Com- puter Graphics and Applications 19(3):6–11 Zheng JY (2009) A virtual heritage on net. http://www.cs.iupui.edu/∼jzheng/bing- mayong/e-index.html 6 The Impact of Digital Technologies on the Exhibition of Cultural Heritages

The ancestors created a brilliant ancient civilization and leftagreat amount of precious cultural heritages. The exhibition of these cultural heritages is an important way of appreciating their historical, artistic, and scientific value. Cultural heritages could mainly be divided into two categories, immov- able and movable heritages. For immovable heritages, such as ancient build- ings, grottoes and cultural sites, visitors always visit the heritage site, while movable heritages are often exhibited in museums, combining some auxiliary tools, such as showcases, tags, display boardsand photographs, to enhance the understanding of these heritages. The above mentioned methods present some problems. First of all, the exhibition of cultural heritages has a highcostandtheflexibility is relatively limited; usually it costs a lot of manpower, material and financial resources to set up or maintain a heritage scene or arrange an exhibition. Once the arrangement is completed, it is hard to modify the contents, so the flexibility is not there. Secondly, scattered heritages are difficult to organize. Some of them may have an important association as they were parts of an entirety. Thirdly, most of the exhibitions at present are static and cannot embody dynamic information, such as the process of production or use, while this dynamic information is important for showing the value of the heritages. Furthermore, exhibitions are constrained by time and space. Opening times and the location of exhibitions are both fixed, which is prohibitive in mod- ern life and limits the opportunities for visiting. In addition, visitors usually are confined to the exhibits as they are presented, which cannot serve visi- tors’ personal interests in breadth or depth. Finally, a great deal of cultural heritages were lost or ruined for many reasons. It’s difficult to exhibit these heritages in their original appearance. However, the exhibition of heritages intheir entirety is very essential. Information technologies provide the exhibition of cultural heritages with a novel platform, which can reduce the cost of arranging the exhibition, in- crease the flexibility, break the restriction of distribution and reorganize these 122 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages cultural heritages scattered all over the world. Furthermore, information tech- nologies can leave a deeper impression on visitors by displaying all kinds of dynamic information of cultural heritages vividly, allowing tourists to visit culturalheritages whenever and wherever they are. Most importantly, it can let visitors appreciate various heritages as if they were present at the original scene. The visitors can interact with the content, customizedbythemselves, based on their own interests and concerns. In a word, information technolo- gies make it possible to convert the exhibition concept from object-orientated to visitor-orientated. This chapter is concerned with key technologies used in digital exhibition and the interaction of cultural heritages. Exhibition contents include shape, color and material information acquired or reconstructed, and heritages’ in- herent historical, artistic and scientific values. During the exhibition, the public can join theexhibition using the interactive means provided. Digital exhibition can help to increase the flexibility of the exhibition, break the con- straints of time and space, restore historic appearance and display abstract traditional culture. At present, there are many exhibition techniques for displaying cultural heritages. We will divide the techniques into three categories, according to the different problems they solve. Section 6.1 describes the technology of online heritageexhibition, which includes management, organization, trans- mission and publication of digitized heritages through networks. This kind of technique aims to break through the constraints of time and space, and establish the so-called “museum of anytime and anywhere” or the “24 hour online museum”. Section 6.2 describes the digital reconstruction exhibition technology of remains, which reconstructs the ruins by computer technology, e.g. computer animation, computer vision and multimedia. Section 6.3 de- scribes the interactive techniques in theexhibition hall,which makes visitors interact with the scene, by virtual environment, natural interaction, or mobile devices.

6.1 Online Heritage Exhibition

As mentioned earlier, cultural heritages exhibition is divided into two cate- gories: exhibition on site and exhibition in the museum exhibition hall. Both of them require that visitors should reach the venue during opening hours. Modern busy lifemakesitdifficult to realize this. In addition, geographical conditions limit people’s ability to visit, as most museums are usually dis- tributed in various places around theglobe. Thus,itisanimportantissueto be resolved as to “how to break through the constraints of time and space in theexhibitions”. 6.1 Online Heritage Exhibition 123

6.1.1 Online Exhibitions Breaking Constraints of Time and Space

Online heritage exhibition refers to arrangements of digitized heritagedatain accordance with certain themes or sequences, and 24 hour exhibitions around the world using the Internet. It includes three aspects in connotation. First of all,theexhibition is “never ending” and opens to theworld 24 hours daily. It gives the user enough time to visit the exhibition and browse the heritages. It becomes an exhibition that can be visited at the visitors’ convenience. Secondly, it can also be visited wherever the Internet exists. Visitors can visit exhibitions all around the world conveniently instead of going to the sites or museums in person. Finally, theexhibition is customized;theuser can arrangeand organize thecontentsdisplayeddepending on his or her personal interests. Online heritage exhibition techniques can break the time and space con- straints of real exhibitions effectively. The user views the online heritages via the Internet whenever andwhereverhecan surfthe Internet.Thecontents of online exhibitions are capable of being organized by the users according to their owninterests.

6.1.2 Process and Technical Framework of Online Heritage Exhibitions

Online heritage exhibition includes several processes, including heritagedig- italization, exhibition data processing, customized arrangement, data trans- mission, rendering anddisplay, and so on. Digitalization is the main method for obtaining data for online heritage exhibition. In order to preserve the heritage details, the accuracy is always as high as possible, thus the models or data are always of the highest preci- sion after digitalization. While in online exhibition, they should be processed firstly, e.g.simplified or compressed, as the contents of an online exhibition need to be transferred via the Internet, whose speed is relatively slow. Other- wise the transmission process will be time-consuming, which greatlyreduces the effects of the exhibition. Various types of data, such as 3D models, video, audio and text information, need to be linked or organized. Thus it enables us to provide a variety of digital information to be exhibited synthetically. Furthermore, formal expression of various exhibits’ data is necessary in order to form a unified data format; for example, the unearthed relics need to mark the excavation time, position, material, weight and other relevant informa- tion. Metadata expression of the cultural heritages is an important method for organizing and expressing the cultural heritages information. Customized arrangement is an important part of online heritage exhibi- tion, by which users can organize the digitalized exhibits following certain themes, sequences or forms. For example, according to the composition of materials, theexhibition can be divided into pottery, jadeware, stoneware and so on. According to their use, the exhibition could be divided into farm 124 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages tools, burial articles, ritual vessels and so on. Similar to traditional exhibi- tion arrangements in museums, customized arrangement refers to the means of arranging and organizing various exhibits digitally. Not only the organizer of adigital exhibition but also the participants and users can be the driving force behind the digitized exhibition arrangement. Data transmission is defined as the transmission of the content of the arranged exhibition content bynetworks, especiallythe Internet. During this process, users send the requests for the exhibition contents and the systems transmit the contents requested back to the client. Rendering and display includes a variety of media outputs, such as a 3D model, video, audio, text, or a combination of them. It is worth noting that arrangement and exhibition is an interactive process, which means that we could makeadigital re-arrangement and update the content repeatedly during the exhibition, in order to achieve a more satisfying result. The online heritage exhibition process is shown in Fig. 6.1.

Fig. 6.1. Online heritage exhibition process

The main technologies for online heritage exhibition include data prepro- cessing, customized arrangement, data transmission, heritage data rendering anddisplay etc. Data preprocessing organizes and converts the high precision data ac- quired in a format that is suitable for online transmission and display. Her- itage metadata provides formal data expression for an online heritage exhibi- tion system. It records not only their common characteristics but also partic- ularities. The common characteristics of the relics include periods, materials, uses etc. Since there are many different forms of relics, different recording information needs to be applied based on their contents. For example, sur- rounding environmental information is important to record for unearthed cul- tural relics, while historical background information is important to record for heritable relics. The diameter of round objects needs to be recorded, while it is necessary to record the length, the width, and the height of cuboid ob- jects. On the other hand, due to the access requirements of the data, the 6.1 Online Heritage Exhibition 125 access to the heritage data needs a unified interface; it makes efficient access toheritageinformation and sharing it via the Internet possible. Therefore, we need a means of expressing and organizing data, which not only provides a unified access interface for the public, but also makes a detailed description of various heritages. Multi-level precision heritage data could provide different resolutions of data for the visitors, which adapts to different hardware and software environments. For example, low resolution images are sufficient for PDA display. Multi-level precision heritage data includes various forms, such as video, audio, 3D models etc. This chapter will focus on the multi-level precision generation of 3D models. Customized arrangement of the heritage exhibition increases the flexibility of online exhibitions. Digital theme modeling can arrange various exhibition themes dynamically, and satisfy an individualized service of an online her- itage exhibition. Different viewers have different interests and habits, so it is necessary that the viewers’ visiting habits, age, identity and other infor- mation need to be recorded in order to supply individualized services and to arrange the exhibition accordingly. In addition, allowing users to view cus- tomized content is also necessary. For example, when a viewer is interested in porcelain of the Ming and Qing Dynasties, he could collect all of the dig- itized porcelain data available on the Internet and arrange his own theme. Heritage information retrieval helps viewers inquire about the digitized ex- hibits needed. A keyword-based search is the most widely used method, while content-based retrieval is a more effective method following human thought habits for mass multimedia information. For example, searching multimedia data about jade cong with its shape or sample image is an application of content-based retrieval. However, content-based retrieval is under research, thus not widely accepted for heritage applications. Data transmission techniques can convert the user’s viewing requests to computer languages, and transmit them via the Internet. Most online heritage exhibitions employ browser/server technology and transmit data with an HTTP protocol. Progressive data transmission technology provides a means of letting the user observe the digitized exhibits “gradually” in a low-bandwidth environment. In such a case, the user cannot receive mass high-precision data immediately, so the system transfers lower precision data first,andthentransfers higher precision data. With this scheme, users can view more and more precise content during thewhole transmission. This kind of technique has been applied in 3D models and image transmission. Heritage data rendering and display techniques are responsible for ren- dering anddisplay the arranged and transmitteddata. Remote rendering provides a means of viewing high-resolution heritage models from a long dis- tance. Ordinarily, models are downloaded completely and rendered on the client side. However, heritage models are usually of high precision, client side computers may have insufficient computational and rendering capabili- ties. In addition, transmission of high precision heritage models is also time- 126 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages consuming. Therefore, servers should do some rendering tasks and then trans- mit rendering results, e.g.images, to the client. In addition, remote rendering could also meet the demand for the protection of high precision heritage data. It prevents the model from being downloaded for improper purpose, such as illegal publication. Nevertheless, local rendering of a heritage model is still an important means of displaying heritages. Thetechnologies mentioned above are shown as Fig.6.2.

Fig. 6.2. Technical framework of online heritage exhibition

6.1.3 Key Technologies for Online Heritage Browsing

We will focus on several key technologies, such as heritage metadata expres- sion, digital theme modeling,multi-precision heritage data generation and remote rendering of high-resolution heritage models.

• Heritage Metadata Expression

In order to display heritage data on the Internet, heritage information must be organized and described formally and effectively so as to be used by upper level services. To put it simply, metadata is the data about data. Metadata 6.1 Online Heritage Exhibition 127 provides a standardized and consistent description of different information and also becomes a tool and link which integrates and organizes this data. The role of metadata mainly includes three aspects. Firstly, metadata gives a detailed and comprehensive description and expression of heritageinfor- mation (MARC, GILS and FGDC/CSDGM are typical ones),suchasname, excavation time, period etc., namely description and expression; secondly, metadata helps people find and search the data needed (Dublin Core is a typical one), namely discovery and search; thirdly, metadata provides a uni- fied interface for upper level services and management, namely management. Different applications have different metadata standards, such as Dublin Core, CDF and Web Collections in the network resource field, MARC and Dublin Core in the document field, TEI Header in the humanities field and CIMI, CDWA and RLG for museums and artworks. It should be said that metadata management is used widely in the field of literature application. The online exhibition of cultural relics has its particularity: There are many different types of heritages and each of them needs different data. Description data usuallyislarger and more complicated so that dozens or even hundreds of items of data are needed for integral description. A heritage metadata expression model was proposed by Liu et al. (2002) of Shandong University in China. Heritage metadata expression needs to ad- dress the problem of combining complexity and simplicity. The expression model constructs simple metadata expression according to basic informa- tion about the heritage, which is frequently used during online exhibition. Those expendable metadata, seldom used involving complete information of the relics, are utilized for integral storage, exchange and management of the heritage data with a tree structure. Simple and complex metadata expression models together construct an integral description of heritageinformation. Commonly used information in the description and retrieval of a heritage includes name, period,material, excavation or acquisition position, usage, shape description, source, artifacts ID and collectors. Several necessary items of metadata have been added, to be retrieved and associated with the other multimedia information, such as keywords, digital information sources, re- lated resources, language anddescription.

• Digital Theme Modeling

In a traditional heritage exhibition, the museum sets up the exhibition themes, and selects and arranges heritages based on the themes. In the digital heritage exhibition, users become the people in charge, instead of museums. The user could set up the exhibition theme and make a customized arrange- ment based on his or her interests or templates, supplied by the museum. In short, an online digital exhibition incarnates customization and participation by the users; it is the essence of what the Internet can offer. 128 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

The digital theme model is the abstract of all kinds of online heritage exhibition components. Three elements, exhibits, theme and classification method, are included in the digital theme model. The digital theme model draws on the experience of the exhibition arrangement in the museum. Cul- tural relics are classified under different classification methods, such as ma- terial, period and uses of the relic. A certain classification method can also include many themes; for example, the classification of heritagematerialin- cludes themes such as jade, pottery, and stone. Each relic belongs to a certain theme while a relic may have a variety of attributes. Thus, a certain relic can be classified into various themes. In summary, theme is a name or an attribute of the heritages combination and each combination represented by the theme is composed of different heritage entities. The basis for dividing relics into a series of topics is theme classification, which refers to the name or attribute of a division. As shown in Fig. 6.3(a), each theme contains a number of cultural relics, each relic could also belong to several classifications, and each theme is de- termined by a certain classification method. The online heritage exhibition system needs to provide certain classification methods and a number of relics. The user defines new themes according to classification methods, and selects

Fig. 6.3. Digital theme modeling technology. (a) Relations of classification method, themes and relics in digital theme modeling; (b) An instance of digital theme mod- eling 6.1 Online Heritage Exhibition 129 corresponding relic exhibits to arrange the exhibition. Fig. 6.3(b) gives an example of setting up themes. When classified according to their usage, relics could be divided into sacrificial vessels, tools, adornments etc. jade cong, jade beads, copper halberds and stone axes can be classified as sacrificial vessels, adornments, sacrificial vessels and tools respectively, according to their us- age. The same relics can also be classified as jadeware, jadeware, bronzeware and stoneware respectively, according to the material they are composed of. Digital theme modeling provides the possibility for personalized online heritage exhibition and customization for visitors.

• Multi-level Precision Heritage Data Generation

In digital exhibitions, a variety of data is needed, such as video, images, 3D models and so on. In this section, the focus will be on multi-level precision data generation of aheritage3Dmodel. The shape information of the heritage is represented in the form ofa polygon, which is the most fundamental data element. Ordinarily, the high precision model of a relic contains millions of polygons. In the online heritage exhibition system, such a large amount of data is difficult to transmit on the Internet and the client’s graphics processing ability may not handle the mass data effectively. The idea ofgenerating heritage data with multi-level precision is that the server keeps a series of models of differing precision. A high precision model will only be processed when thenetwork conditions and the user’s graphics processing capability are adequate. Otherwise, lower precision data is used. Multi-level precision data is generated by simplifying the high precision model. Typical simplification includes vertex removal, triangle removal, edge collapse and trianglecollapse, as showninFig.6.4.

Fig. 6.4. Typical geometrical simplification. (a) Vertex removal; (b) Triangle re- moval; (c) Edge collapse; (d) Triangle collapse 130 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

In the process of simplification, the importance of vertexes and triangles needs to be evaluated. Those vertexes and triangles, which are less important, could be collapsed in order to simplify the model. The importance can be de- terminedby measuring thevariationindistance, curvature and characteristic angle after being simplified. For example, take the distance as the measuring method: Define a center point (e.g. center of gravity) in the model, and then select n directions Di equally. Then the importance could be determinedby measuring the value as shown in Eq. (6.1): n  |OAi| − OAi (6.1) i=1 where Ai is the intersection point of a line from point Oi and the model in the  direction of Di before being simplified and let Ai be the intersection point of the line and the model after being simplified. The smaller the difference is, the better the effect of the model simplification. Also, in the simplification of ruins’ terrain data, the importance could be measured with the curvature of the terrain. For a surface determined by function H(x, y),let x and y be the abscissa and ordinate of that surface respectively, let H(x, y) be the height of surface, and then the curvature can be expressed as Eq. (6.2):

2 2 2 (∂ H/∂x2)2 +2(∂ H/∂x∂y)2 + (∂ H/∂y2)2 (6.2) The vertex with the smallest curvature value should be selected to be removed.

• Remote Rendering of High Precision Heritage Model

Shape is one of themostimportantitemsof information in an online heritage exhibition, which is expressed in the form of a polygon and texture in the computer. In order to express heritage information truly, the quantity of data is always tremendous, which requires a long time and a highbandwidth to transfer on the Internet. Although multi-level precision data generation could reduce the consumption of time and bandwidth, the simplification of the model is contradictory to the quality of heritage expression. In other words, model simplification only solves the problem partially. To build a life exhibition of heritages on the Internet, current data transmission and display method must be modified. A remote rendering system is composed of a server and a client. The client is responsible for receiving user operations, such as translation and rotation for 3D model viewing parameters, where the server is responsible for receiving the operation data from the client and then rendering corresponding images. After the images are generated, the server delivers them to the client. This method takes full advantageof the high-end rendering capacity of the servers and reduces hardware requirements of the clients. It could also secure the 6.1 Online Heritage Exhibition 131 high-precision heritage model since the high-precision heritage model will not be sent to the client. Sometimes, a low-precision model will be delivered to the client for interaction purposes. As the user drags the model using a mouse, the low-precision model will show the approximate view effects and after the drag process stops, the server will render a high precision model image and then send it to theclient. With increasing client visits, a higher hardware capacity at the server will be required. Thus, high-performance graphics workstations or clusters are usually used as servers. Multi-level precision 3D heritagemodelsshouldbegenerated first. The server is composed of amodelmanagement unit, image rendering engine and encoding unit. And the client has an interaction prediction unit, decoding and caching unit, and a data processing unit. Networkdata includealow- precision model, interaction events, and rendering results. The arrows means the data directions. The framework of remote rendering is illustrated in Fig. 6.5.

Fig. 6.5. Remote rendering of heritage model 132 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

6.1.4 Typical Online Heritage Exhibition Applications Some typical online heritage exhibition applications will be briefly introduced.

• Online Heritage Exhibition System of Zhejiang University

The Computer College of Zhejiang University has addressed the need for ex- hibitingheritages on theInternetanddeveloped an online heritageexhibition system. The system has implemented upload, retrieval and exhibition of her- itage data, such as text, 3D model, video and image. Heritage information includes name, ID, material,weight, shape description, excavation location, the collector’s name and so on. The system has also implemented a customized exhibition theme arrange- ment of heritages. User can manually select the interested relic exhibits and arrange them in the digital exhibition hall provided by the system. The visi- tors can explore in the virtualhall and choose any visiting route. Thesystem interface is shown in Fig.6.6.

Fig. 6.6. In the exhibition system developed by the Computer College of Zhejiang University,thethemeof pottery wares is exhibited in the system as an example. These pottery wares were excavated at the Liangzhu Site, Zhejiang, China

After visitors click on the virtual relic, the system will link to a pagewith remote rendering plug-in embedded, in which they can translate, rotate and scale the virtual exhibits. The remote rendering system is shown in Fig. 6.7. In addition, thousands of relics have been put into the database, such as a gold mask and stone tiger from the Jinsha Site, jade cong and jade bi from the Liangzhu Site, and a bird-shaped kettle from the Hemudu Site, as shown in Fig. 6.8. 6.1 Online Heritage Exhibition 133

Fig. 6.7. In the remote rendering system, a multi-view of relics can be rendered according to interaction events generated by the user. In this figure, a jade cong from the Liangzhu Site, Zhejiang, China is rendered in multi-view

Fig. 6.8. Many relics are put into the database of the online heritage exhibition system, most of them are from the Jinsha Site, Sichuan, China, the Liangzhu Site, Zhejiang, China, and the Hemudu Site, Zhejiang, China 134 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

• The University Digital Museum Plan

The University Digital Museum Plan, funded by the Ministry of Education, China, has built 18 university digital museums, embracingfields ofgeo- sciences, engineering, biosciences, archaeology, etc. (Xu and Dong, 2004). The portal of the plan has collected more than one hundred thousand pieces of digital exhibits. Most of these university digital museums employ the form and technology of an online exhibition. Beijing University of Aeronautics and Astronautics takes part in the research and the application of a 3D model promulgating technology based on the Internet. Jin etal. (2006) have developed a 3D rendering system based on the Internet. High precision modelsarekept on rendering servers, whilerough simplified models are owned by the client. Users could translate, rotate and scale the rough model on the client side and select a better viewpoint. When theoperationiscompletedbythe users, a rendering request is sent to the server that contains the current viewpoint, orientation andlightconditions. The servers perform the rendering task and return images of a high preci- sion model according to the client requests. Since the sensitive high precision models are kept on the servers, the system has an inherent safety advantage. The amount of data transferred is about 30 KB each time a new imageis generated and sent to the client, hence the system runs well in the low band- width environment. In addition, therough model that theclient needsto render is so simplethat theclient does not need a high graphics capacity.

• Digital Michelangelo Project

The Digital Michelangelo Project (Levoy, 2006) was performed by Stanford University, the University of Washington, and the Cyberware Corporation. The project officially began in January of 1997, and ended in 2000. It uti- lizes laser scanning technology to digitize sculptures and architectures from the Renaissance in Italy. The most famous digitalized sculpture is of David, sculpted by Michelangelo. In the process of this project, many academics took an interest in the scanned data. The team had developed an online remote rendering system named ScanView, which was introduced by Koller et al. (2004), to share the models scanned, which prevents these sculptures from being replicated. ScanView is a rendering system based on C/S architecture, which could be used to view high precision models of the Digital Michelangelo Project. Thesystem, on theonehand,could let the researchers who are interested in the sculptures view the results of the project. On the other hand, it could ensure the safety of the high precision models. Fig. 6.9 shows the client view of ScanView system. In (a), the system renders the image using locally stored low precision model, while in (b), it renders the image using remotely stored high precision model. 6.2 Digital Exhibitions of Reconstructed Archaeological Sites 135

Fig. 6.9. The client of the ScanView system. (a) Low precision model; (b) High precision model

6.2 Digital Exhibitions of Reconstructed Archaeological Sites

Archaeological sites are vulnerable to variations in climate, to natural dis- asters and other man-made damage, so we can only see the ruins remaining at present. However, in many cases we would prefer to see the original ap- pearance. For example, visitors wish to see the original appearance of the Winter Palace, China, before it was destroyed by fire, by the invaders. Since reconstruction is usually not allowed and is also a hard task, the sites are, in reality, rarely reconstructed. Therefore, the digital reconstruction of archae- ological sites has considerably practical significance, which is a pressing issue to address.

6.2.1 Archaeological Sites Exhibition of Reconstructed Original Appearance

A digital reconstruction exhibition of archaeological sites refers to digital reconstruction of the lost and damaged parts, on the basis of digital acquisi- tion of the site. Then its original appearance is displayed by means of digital exhibition. In the exhibition, the digitally reconstructed site could even be combined with the real site, which resolves the problem that the sites could hardlyinreality be reconstructed.Inthis way it becomes possible to re-create the original styles and features. People can make a digital reconstruction of the site, and avoid incalculable mistakes and losses caused by improper re- construction. In addition, use of computer animation will be helpful in vividly expressing the stories implicated. While reconstructing thesiteitself,people can also reconstruct the dynamic scene in ancient times. For example, seeing the way of life and working environment in an ancient village, makes visitors feel as if they were immersed in that environment. Therefore, reconstruction of archaeological sites has great significance. The rest of this chapter will 136 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages introduce the process, technological framework, key technologies and typical applications of the digital reconstruction of archaeological sites.

6.2.2 Process and Technical Framework of aDigital Reconstruction Exhibition A digital reconstruction exhibition refers to reconstructing the sites’ original appearance on the basis of the present situation, and exhibiting it by various means with some dynamic information, which mainly includes some character and scene animations, etc. The present site model is a 3D model acquiredbythetechnologies men- tioned in Chapter 3. A reconstructed site model is a digitally reconstructed model of the damaged or lost part of the site, according to the archaeological researches. Dynamic modeling of site information refers to modeling of dynamic in- formation on the basis of the present site situation model and reconstructed model. Dynamic information includes various activities of people, working motions with tools, and dynamic information regarding the scene, such as a flowing stream. Character animation and scene animation techniques can be used in dynamic information modeling. The dynamic information really helps a great deal in showing theoriginal appearance vividly. There are two types of digital reconstruction exhibition, an in-site exhibi- tion and a virtual exhibition. The former displays the reconstructed contents atthe present site, and uses augmented reality technologies to register the two parts together. This method can let visitors view not only the remains on the spot but also the part which was reconstructed, as if they were on the scene. The later displaysthewhole reconstructed content with the virtual acquired present site contents only on the computer. The process of a site reconstruction exhibition is shown in Fig. 6.10.

Fig. 6.10. Technical process of digital reconstruction exhibition of site

Many technologies are used in a digital reconstruction exhibition of an archaeological site. Along with modeling of the site dynamic information, in-site exhibition of the site and virtual exhibition are three main ones. 6.2 Digital Exhibitions of Reconstructed Archaeological Sites 137

Modeling of site dynamic information refers to the simulative exhibition of the environment and human activities based on the technologyof scene animation and character animation in the process of reconstruction. It is used to demonstrate scenes from ancient times more vividly. The display achieves better results only if the site becomes what it was in ancient times. In-site exhibition refers to the exhibition that integrates the site recon- structed into thesiteonthespot.Viewpointtracking is used to track the position and orientation of the visitor’s viewpoint, which makes the computer generate images based on the variation of the data. Registration of the virtual and real scene, registering thereal scene on thespot,with the virtual scene createdby computer, ensures their seamless integration. A virtual reconstruction exhibition of the site refers to the reappearance of the site after being reconstructed and combined with the dynamic information in the computer. Rendering and multimedia display technology provides technical support for the site reconstruction exhibition on the spot and a virtual reconstruction exhibition. Rendering technologyrenders the present situation model and the reconstructed model,whilemultimedia display technologyprovides vivid multimedia information for the reconstruction exhibition. Site virtual reconstruction exhibition refers to the technology that brings back the beauty of the history of the site, according to archaeological research findings. Virtual reconstruction of the heritage refers to the digital restoration of site scenes and relics on the computer. The technical framework of adigital reconstruction exhibition of an ar- chaeological site is illustrated in Fig.6.11.

Technical framework of digital reconstruction exhibition 138 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

6.2.3 Key Technologies of Digital Reconstruction Exhibition

We will focus on two key technologies, tracking and registration in virtual reconstruction exhibition, and skeleton animation of human bodies.

• Tracking and Registration inVirtual Reconstruction Exhibition

In the process of visiting an in-site reconstruction exhibition, visitors usu- ally should wear or use see-through display devices, which are composed of cameras, trackers, monitors, etc. Cameras are used to take real-time images of the real world on site, based on which computer generated images will be integrated. Trackers are used to track the motion of the head and output 6 degrees of freedom (DOF), x, y, z, α, β and γ, x, y, z aretranslation values in thedirectionsofthecoordinate axes X, Y , Z respectively,while α, β and γ are rotation angles around the coordinate axes X, Y , and Z. Images of a virtual archaeological site are then generated, using the virtually reconstructed scene model and 6 DOFs. These images are displayed on a monitor and users can see thereal scene in see- through monitors. Thus the virtual and real scene will be combined. Users can view not onlythe scene at thesitebut also thereal-time generated virtual scene. In order to achieve a better effect, a pair of images corresponding to the left and right eye, respectively, is generated. The scene could be viewed with a 3D effect on the basis of stereo vision theory introduced in Chapter 2. Thesystemisshown in Fig.6.12.

Fig. 6.12. Illustration of devices for in-site exhibition

Registration of the real and virtual scene is mainly accomplished by three means, including registration based on marks, registration based on track- ers and registration based on visual images. Registration based on marksis widely used indoors. Since the marks are influenced by light and pedestri- ans outdoors, it is seldom used outdoors, The site is usually one of outdoor scenes, so registration based on marks is not widely used outdoors. 6.2 Digital Exhibitions of Reconstructed Archaeological Sites 139

Trackers are composed of its sensors and the reference frame. The sensor and the user head are relatively stationary. The output results of the trackers are 6 DOFs of rotation and translation corresponding to the reference frame. Assume that the coordinates of the left eye and right eye in the reference frame whose center is the sensor are left, right, respectively,andlet x, y, z, α, β and γ be the output DOFs of the sensor. Let M be the matrix defined as below. Then the coordinate of the left eye rotated corresponding to the sensor is M× left, and the coordinate of the right eye rotated corresponding to the sensor is M× right, where

M = ⎡ ⎤ cos β ∗ cos γ cos β sin γ − sin β ⎣ − cos α sin γ − sin α sin β cos γ cos α cos γ + sin α sin β sin γ sin α cos β ⎦ sin α sin γ + cos α sin β cos γ − sin α cos γ + cos α sin β sin γ cos α cos β (6.3) So⎡ the⎤ coordinates of the⎡ left⎤ eye and the right eye in⎡ the⎤ reference frame x x x are ⎣ y ⎦ + M × left and ⎣ y ⎦ + M × right, where ⎣ y ⎦ is the sensor z z z coordinate in the world coordinates. Finally, the world coordinates can be obtained by multiplying the transformation matrix TM between the world and the reference coordinates, which can be acquired offline. Registration techniques based on visual images have been developed rapidly and applied successfully. Several scene images are taken and stored as reference images beforehand. The system compares the imageof the cur- rent frame with the reference images,andthentransforms the generated images based on match information and the information of pre-stored im- ages. Finally, the system makes a combination of the virtual images and the photo-captured images. As an example, a scene registration method based on visual images which could be used outdoors is introduced by Lin et al. (2005).

• Skeleton Animation of Human Bodies

Cultural remains reflect the situation of people’s work, living and communi- cation in ancient times. In order to reflect the real situation, the activities at that time must be simulated and recreated. How to integrate human factors into theexhibition andhow to make visitors better understand thecircum- stances at that time have become a hot topic in research relating to cultural heritagedigital exhibition. Simulation of character animation is being used widely in the reconstruction exhibition of an archaeological site. Here we will give a brief introduction to 3D human body skeleton animations, that are commonly used in reconstruction exhibitions. A 3D human body is regarded as a set of rigid bodies assembled by joints. For example, the upper arm and the forearm are two rigid bodies assembled 140 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages bytheelbow joint. By representing a rigidbodywith a line, human motion can be simplified as skeleton movement. The human skeleton model is shown in Fig. 6.13.

Fig. 6.13. Human skeletonmodel

The motion data captured on the basis of the human model mentioned above is a set of coordinates in 3D space, which correspond to joint motion varying with time. Considering the model as a tree, the node of the pelvis joint is regarded as the root node and the child joint moves with the parent joint. The entire human motion could be considered as two kinds of motion, translation and rotation, which refers to the translation of therootnodeand the rotation of every child node around the respective parent nodes. Based on the corresponding joint, every bone can rotate with a certain freedom and guides with constraint the movement of the bones that are followed. Two coordinate systems, the global coordinate system and the local coordinate system based on each joint, are used to describe the human motion. The position of each joint can be calculated with the rotation vector and the length of the bone. For example, the relation between position and rotation vector of the right ankle joint can be determined by Eq. (6.4). 6.2 Digital Exhibitions of Reconstructed Archaeological Sites 141

PR Ankle (x,y,z) = TRootRRootTR HipRR HipTR KneeRR KneeTR AnkleRR AnkleP0 (x, y, z) (6.4) PR Ankle (x, y, z) refers to the global coordinate of the right ankle, refers to the coordinate of the right ankle in the local coordinate system, which takes the rightkneeastheoriginwhenthesystemisinitialized.Ti refers to the trans- lation vector whennode i points away from the current coordinate system to the coordinate system of the parent node, while Ri refers to the rotation vector when node i rotates around its parent node. The rotation vector of each joint and the translation vector of the root node can be derived according to the method mentioned above. Sothehuman motion can be indicated with a function of discrete time vectors; in which T (t) refers to the translation of the root node and Ri(t) refers to the rotation of node i.

M(t) = [T (t), R1(t), R2(t),...,Rn(t)](1  t  T )(6.5) In reconstruction exhibitions, skeleton animation should becombined with skin meshes to simulate human beings.

6.2.4 Typical Applications for Digitized Reconstruction and Exhibition of Sites

Some typical site reconstruction exhibition applications will be introduced. • Reconstruction exhibition for the Hemudu Site

The Hemudu Site is a Neolithic site in the Yangtze River valley. The total area of the site is about 40,000 m2, and about 4 m in depth. It consists of a stack offour cultural layers. By using the method of carbon-14, it has been determined that the period of the 4th cultural layer was about six or seven thousand years ago. Zhejiang University, in cooperation with Hemudu Site Museum, exhibits the virtual reconstruction according to the layout of the village, constructions, the working situation of the local people, using the research findings. Results are shown in Fig.6.14. The material for this system was collected according to research on the site, and the content of the scene was modeled using the software of 3DS Max to digitize and reconstruct thesite.The joints animations were captured using a motion capture device of the Motion Analysis Corporation. On this basis, the animations of characters’ skeleton and skin were produced. Finally, the animations relating to theoriginal inhabitants work were completed with examples of tree felling, cooking and rice milling,etc.Fig.6.15showssome reconstruction effects. 142 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

Fig. 6.14. Reconstruction exhibition for the Hemudu Site, Zhejiang, China. The Hemudu Site is famous for its stilt style architecture

Fig. 6.15. Digital reconstruction of the Hemudu Site, Zhejiang, China. (a) Mod- eling of pile dwelling; (b) Animation of rice milling

• The Exhibition of the Reconstruction of Characters’ Animation in a Virtual Church

CruzNeira (2003) presented a concept of computational humanities. He indi- cated that apart from exhibiting aheritageusing the technologyof Virtual Reality (VR), it is also necessary to rehabilitate the humanities, such as the action, the scene and the story,asthey were at the time. The Virtual Re- ality Application Center of Iowa State University utilizes VR technologies inteaching. It simulates traditional rituals and the activities of priests in a virtual Indian church, so that students of religious studies can appreciate therelevant contents in theclassroom, without theneed to visit thesitein person. 6.2 Digital Exhibitions of Reconstructed Archaeological Sites 143

• Field Museum in Waseda University, Japan

Morikawa et al. (2004) of Waseda University in Japan, implemented a Field Museum such that we can understand the concept of being in the real field. Usually, archaeologists carry out archaeological excavations in the wild and place the excavated relics in a museum. In the Field Museum, the exhibited heritages are digitized and placed in the wild and then the technology of augmented reality is applied, to let visitors to observe in the field and be immersed in the real field. Equipment for this system is designed to let the visitors observe in the wild. The user can not only observe the heritageof Heijiyoukyou, but also can enjoy it by zooming, fading in, fading out, and using other control mech- anisms.

• Reconstruction Exhibition of Digital Winter Palace

The project of the Digital Winter Palace, Beijing, China by Wang et al. (2006) of the Beijing Institute of Technology has implemented three differ- ent programs for the outdoor augmented reality of digital reconstruction and exhibition. This includes the fixed-point observation augmented reality sys- tem, the hand-held portable augmented reality system and the head-mounted augmented reality system. These programs and devices make tourists feel the vicissitudes of the Winter Palace. Through analyzing the images of the scene, a fixed-point ARsystemim- plements registration and orientation by photoelectric encoders. The infor- mation about registration will beused to render thepre-built3Dmodel,and generate the virtual image data. Then, through simulated sunlight and other illumination models, and by combining these with images of the real scene, the system outputs the final result to the user. The fixed-point observation device is shown in Fig. 6.16. The project has also implemented themobileaugmented reality system which uses PDA to implement exhibition of Winter Palace augmented reality. This system compares the digital images taken by PDA with the reference images that are stored in the computer, and calculates the external param- eters of the camera. Virtual images generated according to the parameters are superimposed on the original images taken, and then transmitted to the PDA and displayed through a wireless network. With the system, users can conveniently carry mobile handheld devices and experience the exhibition of digital reconstruction during the process of visiting. In addition, the project of the Digital Winter Palace also implements head-mounted equipment for exhibiting in the field. A head mounted display (HMD) is an important augmented reality device. A reconstruction example of the Digital Winter Palace is shown in Fig. 6.17. 144 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

Fig. 6.16. Reconstruction exhibition of the Winter Palace, Beijing, China (With permission from Yongtian Wang)

Fig. 6.17. Reconstruction exhibition of the Winter Palace, Beijing, China. (a) Original field of site; (b) Exhibition of reconstruction after virtual scene is registered (With permission from Yongtian Wang)

6.3 Interactive Experience in the Exhibition Hall

Interactive experience means that a museum makes exhibitions more vivid with the help of digital techniques. During the process of visiting,themu- 6.3 Interactive Experience in theExhibition Hall 145 seum provides various interactive means for the visitors, in order to stimulate visitors’interests andattract their attention. At present, several methods are used to assist the exhibition in the mu- seum, such as sand tables to imitate the original appearance of the ancient scene, narrating information about specific uses of the exhibits and their backgrounds, using notions or pictures to enhance theembedded cultural in- formation. Overall, there is still something to be further improved in order to exhibit more vividly the exhibition in the museum. First of all, the static display of the exhibition, the most common method used now, does not interpret well dynamic information, such as the rules, the phenomenon, the process, the activity, etc. Thus the essence cannot be recognized well unless it is analyzed in a dynamic way, which requires some dynamic exhibiting methods for a vivid expression of the exhibits. Secondly, as the exhibited subject is set in a fixed sequence, the visitors will have to fol- low the sequence when visiting. Considering that different visitors may have different interests and focus on different things, the visit should be made more customized, which is hard to realize at the present time. Finally, the exhibits in the display case cannot be shown from all directions. The char- acters description and pictures alone cannot express the embedded cultural values behind the exhibits, such as the style and the polytechnic techniques. Thereisnodoubtthat an auxiliary exhibiting method is urgentlyneeded. At present, museums have applied a variety of technologies for interactive display that stimulate the visitors’ initiative. For example, Shanghai Mu- seum, China, provides digital narrators for the exhibits. Visitors can enter the number of an exhibit in the process of visiting, and then the exhibit’s detailed information will be given automatically. In Jinsha Site Museum, Sichuan, China, there is an interactive system that allows the visitors to play the chime (a kind of ancient instrument) by touching a wide screen monitor. When visitors touch different positions, the system will produce a different sound,which is shown in Fig. 6.18.

6.3.1 Interactive Experience that Enhances a Sense of Participation

Digital technology can offer effective support for the interactive exhibition in the museum, which can show all the dynamic information about the remains of ancient cultures and make the visit more personal, based on the visitor’s particular interests. The display can also offer the corresponding backgrounds and stories to achieve an interactive exhibition. In addition, digital technology can show us thecultural values of theexhibits. First, the technologycangive a vivid simulation of the ancient panorama with information about the environment in all its aspects. With the addi- tion of sound and vision effects, it enables the visitors to have an intuitional understanding of the historical remains. Second, the visitors can participate in the ancient activities personally by sightseeing in the ancient environment 146 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

Fig. 6.18. Interaction of digitally playing ancient instrument in Jinsha Site Mu- seum, Sichuan, China and interacting with the characters, the scenes and the objects. Some ex- hibits, which are hard or impossibletodisplay to visitors, such as production processes in ancient times and the use of tools, are not a problem anymore. It can totally alter the current situation that focuses on static exhibiting and which is inadequate. Lastly, the display of the exhibits viewed from all di- rections and from all aspects can be targeted, or individualized, according to visitors’ interests and focus.

6.3.2 Process and Technical Framework of the Interactive Experience in the Exhibition Hall

As mentioned above, a museum will use a variety of auxiliary means to make the exhibition more attractive. A museum will display the ancient scene with a sand table imitation, which makes visitors understand the background to those times better. A museum can display various aspects of the exhibits with an additional introduction, such as by using labels, notes and figures. Moreover, mirrors or magnifiers can make visitors notice more details, and guideboardscanhelpthe visitors. Narrators in a museum can give a detailed commentary about a part of the exhibits, which makes visitors learn the background stories. Interactive experience technology can beused to assist theexhibition by adding auxiliary devices. Watching ancient movies of the scene by high reso- lution multi-projector devices, visitors can immerse themselves in the ancient environment. They can even interact with the 3D objects, for example expe- rience fishing and hunting as ancient people did. Visitors who take advantage of a multimedia display, route planning and interactive narration can get all- directional and more detailed information about the exhibits. In the process 6.3 Interactive Experience in theExhibition Hall 147 of visiting, visitors can get personalized information on the basis of position information and can make requests with output devices (hand-held devices, interactive displays and earphones). All these techniques mentioned above are helpful in guiding the visitors. The process mentioned is shown in Fig. 6.19.

Fig. 6.19. Process of interactive experience in exhibition hall

Interactive experience techniques include a digital guide and an immersive interactive exhibition. Digitalguidetechnologyneeds to position the user, and then providethe user with personalized narration and a guide based on the position infor- mation. It includes various means, such as the position based on a WIFI (Wireless Fidelity, 802.11 b based wireless local area network) signal, and position based on RFID (Radio Frequency Identification). The digital guide provides multimedia commentary for users, which is transferred by wireless transmission technique. The multimedia system provides a variety of vivid materialsfor theexhibition. Immersive interaction allows visitors to experience the ancient scene as if they were on the scene. Stereo image and stereo sound provide a basis for the exhibition. Multi-projector calibration provides a more lifelike output effect for an immersive interactive exhibition. Collaborative rendering drives themulti-projector system and calculates thecontentdisplayed.Inorder to interact in the exhibition hall, the interactive information needs to be 148 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages recognized and processed; for example, voice recognition, gesture recognition and visual tracking are all applied. The technical framework mentioned above is shown in Fig. 6.20.

Fig. 6.20. Technical framework of interactive experience in exhibition hall

6.3.3 Key Technologies of Interactive Experience in Exhibition Hall

Technologies on multi-projector calibration, multi-computer collaborative rendering protocol,anddigital guideinexhibition hall will be discussedhere, which can helptoenhance interactive experience in exhibition hall.

• Multi-projector Calibration for Immersive Exhibition

An immersive exhibition is an important means of displaying cultural relics in a museum. A high resolution exhibition environment that has a strong visual appeal is a crucial part of the immersion system. At present, high resolution multi-projector systems are more and more widely used in the exhibition. In the process of constructing the environment, since the images projected may be distorted by the geometric distortion of the projection screen, they should be corrected geometrically. Images projected are correct only if the projection screen is completely planar and the postures of the projectors are also correct. Image distortion 6.3 Interactive Experience in theExhibition Hall 149 is the phenomenon where projected images differ from the expected images on account of anon-planarprojection screen or the incorrect posture of a projector. Image distortion correction handles the problem of the projected images and the expected images. In addition, it cannot be certain that images projected by different projectors are well aligned, and this is dealt with by multi-projector alignment techniques. Distortion correction and multi-project alignment together are called multi-projector calibration. Images projected on different screens may have varying degrees of distor- tion. The images projected are correct only if the plane screen is vertical to the optical axis of the projector, otherwise the images may have distortions, suchaskeystonedistortion.Today,thecorrectionhardwareisintegrated into the projector for flat screen distortion. The projector corrects images by con- trolling the optical axis of the lens automatically, but this method works effectively onlyfor certain kinds of distortion. When the projection screen is not flat, more complicated image distortion will require a more complex correction. The distortion needs to be corrected further in the multi-projector sys- tem because of the deficiency of automatic correction. The projector pro- vides manual correction features for users, so engineers can correct the image distortions. The multi-projector system is available today on the basis of available machinery and manual correction features. This method requires a lot of manpower and is not adequate for a non-planar screen. The effect also depends on the technical skill of the engineers, so current systems adopt correction with software. Software correction obtains projected images with cameras and other equipment, and determines the distortion parameters of the projectors ac- cording to imageanalysis results. Thesystemthen distorts theimages in an opposite manner to remove the distortions. We call such process pre-warping, asshown in Fig. 6.21. According to the process of pre-warping, there are two distortion correc- tion techniques commonly used today. The first is to analyze the images and obtain the coordinate transformation matrix, which relates to the projected images and the image acquisition equipment. The second is to select several key points, and then achieve one’s objective by bilinear interpolation within the images between the key points. The first method is only suitable for planar screens, while the second is suitable for screens of all shapes. For the first method, the relation between the points in the projected image and the points in the projector image can be described as Eq. (6.6). ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ xp a11 a12 a13 xi ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ w yp = a21 a22 a23 yi (6.6) 1 a31 a32 a33 1 150 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

Fig. 6.21. Sketch of projector screen correction ⎡ ⎤ xp ⎣ ⎦ IntheEq.(6.6), yp is the homogeneous coordinate of apointinthe ⎡ ⎤1 xi ⎣ ⎦ projector image, yi is the homogeneous coordinate of a point in the ob- 1 ⎡ ⎤ a11 a12 a13 ⎣ ⎦ tained image. So we only need to calculate a21 a22 a23 , named the ho- a31 a32 a33 mography matrix. These parameters can be estimatedby sampling several pairs of corresponding pixels in the projector image and camera image, by the least square method. In order to improve the accuracy further, more pairs of points are usually adopted. Raskar et al. (2002) proposed a method to ac- quire the transformation relation with cameras, which utilizes the marks on the projected screen or information of the screen border. The transformation relation could be calculated within a few seconds by using this method, and the resolution of the sub pixel can be achieved. As the field of view of a single camera is limited, some researches are focused on scalability issues for achieving multi-projector alignment. Chen et al. (2000) proposed adding an attachment under the camera, which 6.3 Interactive Experience in theExhibition Hall 151 could be used to adjust the posture of the camera. The attachment makes the camera take images from different viewpoints and justifies the projectors, which are visible to a single camera, one by one. Eventually, all of the projec- tors can be justified. Chen et al. (2002) proposed a system that carries out a wide range of shooting using an array of cameras, and then splices these images together on the screen of a virtual “super perspective” camera. Both of the methods mentioned above have resolved the scalability problem of the multi-projector system. There are also many implementations of the second method mentioned above. Asanexample, Yuan et al. (2007) of Zhejiang University proposed a calibration method, which can achieve inner-projector distortion correction and multi-projector alignment in a single process (Yuan et al., 2007; Yuan and Lu, 2008). The method lays the camera on a high precision numerical control (NC) turn plate. The camera can take photos of a larger and more integral region by controlling the orientation of the NC turn plate, which has ahigh accuracy. The inner-projector distortion parameters are calculatedby analyzing the structured light projected by the projectors using bilinear in- terpolation. And the multi-projector alignment parameters can be obtained byturnplate angle parameters, which are used to adjust the viewpoint and the viewing orientation of the camera. Then the pre-transformed imageis rendered and projected on the screen and aligned to the adjacent image, as shown in Fig. 6.22. We obtain the external parameters of the physical camera by reading out the turn plate angle. And then we apply it in frustum and viewpoint setting in the graphics application. After the images are rendered using these parameters, we then pre-warp these images as textures and finally project them.

Fig. 6.22. Large screen alignment method based on parameters of cameras. (a) Rendered image; (b) Eye observed image 152 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

• Multi-computer Collaborative Rendering Protocol

In order to construct a large exhibition environment, several projectors are necessary for building ahigh resolution exhibition interface. The display con- tent needs to be collaboratively rendered under the control of several com- puters.

Fig. 6.23. Multi-computer collaborative rendering protocol for multi-projector sys- tem

Fig. 6.23 shows the synchronization process of different modules. When the system is initialized,theremotecontrol moduleand therendering mod- ule are started (the number of the rendering modules and the projectors is 6.3 Interactive Experience in theExhibition Hall 153 the same, however, for we only draw one rendering module for simplification in expression). Each rendering module creates a TCP (Transmission Control Protocol) socket, and then binds and listens on one port, while the remote control module sets up various parameters and then connects to theren- dering module. When all of the connections have been completed (Fence 1, waiting for all the connections to be completed), the control module loads the application and sends it to the rendering module. Afterwards the control module notifies all of the rendering modules to call the initialization interface app::Init in the application module. When all of the modules are initialized and returned back (Fence 2, waiting for all the nodes to be initialized),a main rendering module calls the interface app::Frameupdate in the applica- tion module and updates the next frame. After app::Frameupdate is com- pleted (Fence 3), all of the rendering modules call the interface app::Display. After all the rendering modules call the interface app::Display (Fence 4),the system goes to the next loop, rendering the next frame. (Yuan et al., 2005). Table 6.1 shows the communication primitives and their functions in the synchronization. Primitives setHostCfg, loadApp, unloadApp, runApp,and stopApp handle the preparation work before rendering, including setting the configuration of each rendering node, to send application (.dll) to rendering nodes, to send the command of start and stop. Primitives Init, frameUpdate, and Display correspond to the three basic operations in the framework of OpenGL: Initialize the scene, update the next frame and render the next frame.Primitives FUFinished andDFinished are sent bytherendering node, whichis used to return itselftotheactive state.

Table 6.1. Communication primitives of multi-computer synchronous rendering Primitive Function Description setHostCfg Set configuration of each rendering node loadApp Send application (.dll) to rendering nodes unloadApp Load and unload application runApp Run the loaded application stopApp Stop the running application Init Initial the virtual scene frameUpdate Update the next frame Display Render the next frame FUFinished The next frame has been updated DFinished The next frame has been rendered

• Digital Guide in Exhibition Hall

Along with the development of wireless communication and PDA,adigital guide in the museum has become a hotspot. First of all, since its resolution and display effects have been remarkably improved in recent years, PDA is 154 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages widely used as a display terminal in the field of exhibition in the museum. Secondly, PDA can be used as a wireless receiving device which is capable of receiving data, such as multimedia information, from the server. Finally, PDA can also be considered as the locating equipment which can position itself by measuring and processing the signal strength from wireless access points (APs), and then the adjacent information can be provided based on the position results. Based on the position information, Mei-hsuan LU of CMU designed a wireless video service system (Lu and Chen, 2006), CMUSEUM. Media ser- vices based on position are realizedbyemplacing thewireless sensors in the museum. The Zigbee (IEEE 802.15.4 Standard) nodes emplaced are used to transfer the user’s position, while the WIFI nodes are used for service data transmission. TheZigbee nodes can communicate with each other, and the user’s position is determined by the triangulation method and the signal strength between the Zigbee node and the user node. Finally, the position information is sent to the positioning server. The video server receives an event of position variation, and then provides a video service on the basis of the new position. In addition, CMUSEUM also provides the broader service information, such as the current position, the visiting path and the distribution of visitors. The system can show the service that displaysthelocationof the visitor to avoid him being lost in the museum, and provides a guide for a visiting path.

6.3.4 Typical Application of Interactive Experience System

Some typical interactive experience systems are briefly introduced. • Digital Excavation Experience of the Jinsha Site, Sichuan, China

The Computer College of Zhejiang University built a three-channel multi- projector system, as shown in Fig. 6.24. It is composed of six computers and six projectors, in which each projector is driven by a computer. The two projectors, upper and lower in each group, display the left-eye imageand the right-eye image respectively (The stereo image pare generation principle is introduced in Chap. 2). There are three imaging channels in total. The system is calibratedbythemethod introduced in 6.3.3. The system makes an interactive display of two groups of trial pits (the first group includes IXT9641,T9642,T9741, and T9742, and the second group includes IXT7128,T7129,T7228, and T7229). The viewpoints can be freely controlled by the users. The system effect is shown in Fig. 6.25. • Interactive Exhibition of Ancient Appian Way by Bologna Uni- versity, Italy

CINECA of Bologna University of Italy has implemented a multi-user in- teractive exhibition system for an archaeological site (Farella etal., 2005). 6.3 Interactive Experience in theExhibition Hall 155

Fig. 6.24. Stereo displaying system with triple channel

Fig. 6.25. Interactive excavation experience of the Jinsha Site, Sichuan, China

Users can select and explore in the virtual scene with the hand-held PDA de- vices, and they can also adjust the speed and granularity of these operations. Visitors are no longer the passive recipients of information; they can really appreciate the interactive effect on the spot. People can view not only the 3D models but also the words of the commentary and multimedia informa- tion, and they can control the rendering models, wire frame or full rendering. The details of the scene can be viewed through PDA in the process of this immersion visit, and the roaming process can be controlled too. 156 6 The Impact of Digital Technologies on the Exhibition of Cultural Heritages

6.4 Summary and Prospects

This chapter describes the technology of an online heritage exhibition, digi- tal reconstruction exhibitions of archaeological sites, and the interactive ex- perience in the exhibition hall. These technologies meet people’s thirst for knowledge in the area of culture heritage, and are effective in assisting and expanding such functions in the exhibition in the real museum. In addi- tion, because of the excessive development of tourism and other industries, it sounds an alarm to the world. With the availability of adigital exhibition of our cultural heritage, it can help effectively to solve the problem. Digital technology provides a new platform and means for a digital cul- tural relics exhibition. It saves costs in the layout of the exhibition, increases the flexibility of the exhibition, breaks through limitations in time and space and, in distributing the exhibition, enhances the vitality of an interactive exhibition. Many aspects of a cultural relics exhibition have been reinforced. The technologyof an online heritage exhibition further spreads the impact of the heritage exhibition. The technology reconstruction exhibition of ar- chaeological sites can solve the present difficulties in recreating the original heritage on site, and the technologyof interactive experience in the exhibition hall can improve the feeling of participation and immersion for visitors. However, there are still many shortcomings in digitized cultural relics ex- hibitions. It is too restrictive in content, and most digitized exhibitions are still just a display of the cultural heritage itself. The extent to which the exhibition is lifelike is still far from that of a real cultural relics exhibition. The greatest advantage of the digitized technology for a cultural relics exhi- bition, which breaks through the limitations of entity, is to manifest traits of vividness and imagination, but this has yet to be realized. It can be predicted that a digitized cultural relics exhibition will be more humane, people-oriented, emphasizing the sense of participation and indi- vidual expectations. The equipment for the exhibition, stressing mobility, portability and easy operation, will beemphasized with multi-channel inter- active technology, so that visitors, as when visiting in thereal world,can see images, hear voices and even touch the heritage. Finally, the archaeolog- ical findings will be reconstructed and the ancient documents can be more vividly displayed by computer animation, graphics and other technologies. In short, digital technology will change the present situation in which thepub- lic is a passive spectator. It will let the public become an important part of the exhibition and participate in it. According to people’s interests, hobbies, knowledge and backgrounds, it will become a personalized exhibition by stim- ulating their thinking,encouraging their participation, even attracting them to become immersed in an interactive process. As part of a new exhibition References 157 technology, the digital experience has already enjoyed popular support and acceptance.

References

Chen H, Sukthankar R, Wallace G (2002) Scalable Alignment of Large-format Multi-projector Displays UsingCamera Homography Trees. In: Proceedingsof IEEE Visualization, IEEE, New York 339-346 Chen YQ, Clark DW, Finkelstein A, Housel TC, Li K (2000) Automatic Align- ment of High Resolution Multiprojector Displays Using an Uncalibrated Cam- era. In: Proceedings of IEEE Visualization, IEEE, New York 125-130 CruzNeira C (2003) Computational Humanities: the New Challenge for VR. Projects in VR. IEEE Computer Graphics and Applications 23(3):10-13 Farella E, Brunelli D, Benini L, Ricco B, Bonfigli ME (2005) Pervasive Computing for Interactive Virtual Heritage. IEEE Multimedia 12(3):46-58 James DL and Twigg CD (2005) Skinning Mesh Animations. In: ACM Transac- tions on Graphics 24(3):399-407 Jin P, Zhang HD, Qi Y, Sheng XK (2006) Remote-rendering Based 3D Model Publishing System. Journal of Beijing University of Aeronautics and Astro- nautics 32(3):337-341 (in Chinese) Koller D, Turitzin M, Levoy M, Tarini M, Croccia G, Cignoni P, Scopigno R (2004) Protected Interactive 3D Graphics Via Remote Rendering. In: Proceedingsof ACM SIGGRAPH, ACM, New York 695-703 Levoy M (2006) The Digital Michelangelo Project. http://graphics.stanford.edu/pro- jects/mich/ Lin L, Wang YT, Liu Y, Zheng W (2005) Outdoor Registration Method Based on Image Matching. Journal of Image and Graphics 10(9):1146-1151 (in Chinese) Liu SJ, Meng XX, Xiang H (2002) Research on the Data Integration of Archaeo- logical Digital Museum with XML. Journal of System Simulation 14(12):1624- 1627 (in Chinese) Lu MH, Chen T (2006) CMUseum: A Location-aware Wireless Video Streaming System. In: Proceedings of IEEE Conference on Multimedia and Expo 2006, IEEE, New York 2129-2132 Morikawa H, Kawaguchi M, Kawai T, Ohya J (2004) Stereoscopic Displays and Virtual Reality Systems XI. In: Proceedingsof the SPIE 5291:415-422 Raskar R, Van-Baar C, Chai J (2002) A Low Cost Projector Mosaic with Fast Reg- istration. In: Proceedings of the Fifth Asian Conference on Computer Vision, Asian Federation of Computer Vision Soc, Clayton, Vic., Australia 164-169 Wang YT, Lin L, Liu Y (2006) Outdoor Augmented Reality and Its Application in Digital Reconstruction of Yuanmingyuan. Bulletin of National Natural Science Foundation of China 20(2):76-80, 86 (in Chinese) Xu SJ, Dong SC (2004) Digital Museum-Recent Efforts of the Universities in China. Science 56(6):17-19 (in Chinese) Yuan QS, Lu DM, Chen WD, Pan YH (2005) MultiPro: A Platform for PC Cluster Based Active Stereo Display System. In: Proceedings of Computational Science and its Applications 2005 Part I, Springer-Verlag, Berlin, Germany 865-847 158 6 The Impact of Digital Technologies on the Exhibition ofCultural Heritages

Yuan QS, Lu DM, He YM (2007) Scalable Arbitrary Surrounded Surface Cali- bration for Multi-projector Rendering Application. In: Proceedings of Virtual Systems and Multimedia, Brisbane, Australia YuanQS,LuDM(2008) Multi-projector Calibration and Alignment Using Flat- ness Analysis for Irregular-shape Surfaces. In: Proceedings of the 9th Pacific Rim Conference on Multimedia, Taiwan, China 436-445 7 Digital Development and Utilization of Cultural Heritages’ Information

Cultural heritages involve rich historic, artistic and scientificinformation, which can not only help people to understand implicit heritage values but also can create new values. As the meaning of development and utilization varies in different fields, we specifically define here the meaning of “devel- opment and utilization” as used in this chapter. Development refers to the extraction, mining and expression of cultural heritages’ information, includ- ing both the noumenal information and the valuable implicit information. Utilization refers to re-creation based on the extracted and mined valuable information about cultural heritages, such as the copying of ancient drawings or the creation of patterns based on murals’ style learning. Itisofgreat significance to carry out research into the development and utilization of cultural heritages’ information. The existing research into the values of our cultural heritages is restricted to the world of academia and the results are hard to exhibit or re-utilize as they are mostly presented textually. For example, if we can extract the architectural style and dress style in the Dunhuang murals by information technology and formalize the method of ex- pression, it will be very convenient for us to apply this to education, science and technology, culture and other fields. In addition, with the technology, we can extract and represent the implicit information about heritages, and by combining this information with the noumenal information, we can provide rich data for the overall recognition of cultural heritages, facilitate their uti- lization and the propagation of cultural heritages and meanwhile fulfill the task of heritage preservation. This chapter mainly focuses on key technologies and examples of the de- velopment and utilization of cultural heritages’ information. In Section 7.1, we discuss the advantages of the development and utilization of our heritages. In Section 7.2, we introduce the process and technical framework of the de- velopment and utilization of cultural heritages. In Section 7.3, we mainly introduce the key technologies, such as pattern element extraction, expres- sion and extraction of ancient murals’ artistic style, and so on. In Section 7.4, 160 7 Digital Development and Utilization ofCultural Heritages’ Information we introduce some typical systems and applications. Finally, in Section 7.5, we summarize the prospects for the development and utilization of heritages and introduce possible research directions for the future. Different cultural heritages differ in their cultural information, also in their technologies. Therefore, we mainly take the development and utilization of ancient murals and buildings as examples.

7.1 Culture Heritages’ Value

Cultural heritages have two types of information. One is the noumenal infor- mation, including their shape, color,texture,etc.Forexample, in a mural, the shape of human beings, painting styles, color features, basic pattern el- ements (clothes, people, mountains, trees, buildings, decorations, etc.) and the historical scenes implied are all contained in the images of murals which can be acquired directly from the murals themselves. As shown in Fig. 7.1 (a) is the mural at the top of a shrine of the 328th cave of the Dunhuang Mogao Grottoes, and Fig. 7.1 (b) shows the basic elements extracted from the murals, including a flyingfairy, flowers, and so on.

Fig. 7.1. Extractions of the Dunhuang murals’ elements. (a) Original picture (Pat- tern at the top of a shrine); (b) Extracted elements

The other is implicit information, such as historical and cultural stories, material processing techniques, etc. Unlike noumenal information, implicit information cannot be directly obtained for the heritages themselves, but derivedbyarchaeological researches, backgroundknowledge andhistorical documentation. Fig 7.2 is a mural from the 285th cave of the Dunhuang Mogao Grottoes, which depicts the Buddhist Story of a nine-color deer script. The nine-color deer king walked along the water, and saved the drowning man Diaoda. Afterwards, the queen ordered the king to strip the deer king’s nine- color fur to make clothes. While they were seizing the deer king,Diaoda 7.1 Culture Heritages’ Value 161 informed the king about the deer’s track. Finally, Diaoda recoiled because of his lack ofgratitude and the deer king was sent back home. It is, in fact, very difficult for visitors to understand the background story and comprehend the connotation of the story, “honesty” and “gratitude”, only by looking at the mural.

Fig. 7.2. Mural of the story of nine-color deer in the 285th cave, the Dunhuang Mogao Grottoes

A traditional way of developing and utilizing the value of cultural her- itages is by exhibition. The major mode can be divided into the following three types. (1) Exhibition of the heritage itself. Most unearthed relics and well- preservedheritage sites such as old buildings, old ramparts and stone carvings are generally displayed and exhibited in a systematic way in museums or are open to visitors at the site. (2) Restoration and imitation. Since heritage resources are non-renewable and easily destroyed, it is impossible to see the original scene of some im- portant heritages that have been adversely affected by time, nature or war. People can only restore or imitate the heritages based on archaeological re- sults or historic documentation. Although we cannot make the heritages look exactly likethe ancient ones in materials or construction technologies, we can still retain the “historic authenticity” so far as position, structure, scale and style are concerned. (3) Creation. Such kinds of resources are mostly developed by imaginative creation based on the documentation that is to be authenticated, myths, literary works, such as “the Palace of the Journey to the West”, “the Eighth Immortal Palace”, “The Shuihu Town”. Clearly, traditional development and utilization of cultural heritages’ in- formation are still mainly oriented at cultural heritages noumenal informa- tion, and the implicit values of cultural heritages have not been fully utilized. 162 7 Digital Development and Utilization ofCultural Heritages’ Information

Digital technology has brought a new approach to the development and uti- lization of cultural heritages. On the one hand, it avoids direct contact with the heritages and thus prevents potential damage. On the other hand, by synthetic utilization of image processing,artificial intelligence, multimedia and animation technologies, we can fully mine the implicit historic, artistic and scientific value of cultural heritages.

7.2 Process and Technical Framework of Digital Development and Utilization

As has been mentioned above, cultural heritages’ information can be divided into two categories. One is the noumenal information and the other is the implicit information. Development of cultural heritage information means the mining of valu- able implicit information using the noumenal information and some relevant researches, background knowledge and historical documentation. Take the Dunhuang murals as an example, the contents expressed in the murals are mostly concerned with Buddhism, including Buddhist stories, Bunsen, a bi- ography of Buddha, donors, as well as principal and subsidiary causes etc. the Dunhuang murals have rich contents, such as god and spirits, characters, animals, landscapes, architectures and decorative patterns, which are firstly extracted as source materials. Theimplicit content, such as Buddhist sto- ries and cultural evolution, will be mined by scripts. For example, the dress style of characters in the Dunhuang murals reflects the evolution of local dress culture-shifting gradually from styles of the Western Regions to the Han style. The utilization of cultural heritages’ information mainly involves re- creation based on the extracted information, for example style learning based pattern creation and animation creation. In key technologies, we will focus on computer-aided imitation of murals and the creation of patterns in the style of the Dunhuang murals. The technical process of the development and utilization of cultural her- itages’ information can be described as follows. Firstly, extract the noumenal information of cultural heritages, mine the implicit information, and express them in a formalized way. On the one hand, we can use target extraction and 3D reconstruction to extract the basic source material and artistic style information. For example, the above technologies can help to extract various kinds of basic pattern elements of murals, includ- ing characters, animals, clothing, architectures, musical instruments, furbe- low, daily necessities, ornaments, landscapes, and so on. In addition, they can also help to form a new complete element from some of the parts in different murals, or achieve 3D information by several 2D mural imageelements.On the other hand, to mine the implicit values we should collect and analyze the research results and write the scripts so that they can beutilizeddigitally. 7.2 Process and Technical Framework of Digital Development and Utilization 163

Secondly, we should create new materials on the basis of developed in- formation of cultural heritages, by extracting the style and pattern elements from the paintings and by creating similar works by pattern replacement or style learning. For ancient buildings,thelayoutof the buildingsextracted can be used to make rapid modeling in a typical architectural style. Finally, we should store and manage thesourcematerials discussed above and provide services such as multi-media browsing, retrieval,query,and so on. And then a database should be used to store the source extracted materials, the mined information of implicit values and the recreated information for future usage. The above process is illustrated in Fig.7.3.

Fig. 7.3. Process of development and utilization of cultural heritagevalues

The technologies mainly include segmentation and extraction of basic source materials, expression and extraction of artistic styles, and expres- sion and mining of valuable implicit information. Extraction technologies for different forms of cultural heritages are different. Generally speaking,sur- face color and textureextraction mainly rely on imagesegmentation, whereas shape extraction mainly depends on shape feature extraction, shape expres- sion anddescription. Digital matting is important for the extraction of basic source materials of cultural technologies. As an important means of image editing and processing, digital matting will be discussed in detail in section 7.3. 3D reconstruction, based on patterns with similar style in many 2D images, is also frequently used to solve the problem that a one-view image lacks information. Expression and extraction of artistic style refer to extracting relics’ artistic value and expressing it in a form that can be reused by computer. Generally, cultural heritage relics are also art works. They often reflect the artists’ cre- ative idea in a specific background, theme and style. Style refers to common 164 7 Digital Development and Utilization ofCultural Heritages’ Information features that a group or an art school represents in some period and area. To extract the art style of cultural heritages, it is necessary to define a formalized expression model. Take calligraphy and painting as an example, style can be described from the structural layout, characters, line drawing, color paint- ing, and so on. Hence, style extraction can be conducted from these aspects. Fig. 7.4 demonstrates a sample of art knowledge extraction from murals. (a) is the caisson pattern of the Northern Zhou dynasty in the 428th cave of the Dunhuang Mogao Grottoes, and (b) is a common structure layout extracted from the pattern. For another example, ancient buildingsinChina’s Ming and Qing dynasties have a standardized architectural design, all consisting of similar foundations, body and roof, regardless of their sizes.

Fig. 7.4. Sample of extraction of layout style. (a) Original pattern; (b) Extracted layout

Extraction and expression of implicit information of cultural heritages are related to relevant research results, aiming to describethe history, art and scientific values contained in the cultural heritages in a formalized way. These formalized descriptions can be recognized and used directly by computer. Takethe image in muralsasanexample, thestoriesdescribedbythemurals are an important aspect reflecting their cultural and historic values, hence we could turn these stories into animation scripts. The digital technologies for utilizing cultural heritages’ information in- clude computer aided imitation of ancient paintings, style learning based re-creation and animation production, and so on. Imitation is a frequently used creation method for learning skills and copying of works. In computer-aided imitation, digital technologies are used to aid artists in line drawing extraction, element displacement, filling color, and so on. Style learning based re-creation aims to create works with similar styles. It is commonly used in calligraphy and paintings. By analyzing the appreci- ation and creating process of art works, we can comprehend the basic repre- sentation methods, content expression methods, as well as appreciation and 7.2 Process and Technical Framework of Digital Development and Utilization 165 creation of ideas in a formalized way. Then we can simulate the process of art appreciation, analyze the original patterns, extract the color, shape and lay- out information, and establish a library of pattern elements, colors, layouts and styletemplates in order to makeintelligent creations. Animation production refers to the process of utilizing the extracted or created basic source materials combined with the mined implicit information to produce vivid animations. Such a visual approach can be used for many purposes, e.g. virtual tours, gaming, education, exhibition, and so on. The technical framework among the key technologies is illustrated in Fig. 7.5.

Fig. 7.5. Technical framework of digital development and utilization of cultural heritages 166 7 Digital Development and Utilization ofCultural Heritages’ Information 7.3 Key Technologies for Development and Utilization

There are a great variety of cultural heritages differing greatly in shape and historical and cultural background. Therefore the key technologies used for the digital development and utilization are also different. Here we will take traditional architectures and muralsasexamples.

7.3.1 Source Material Extraction

The extraction of source materials mainly involves the extraction of heritages’ shape, color, texture and the elements consisting of the above properties. The extraction technology relies on a technology newly emerging in recent years calleddigital matting.Linet al. (2007) has summarized the technology of digital matting.Digital matting was firstly put forward by Porter and Duff (1984). The equation of image synthesis is

C = αF + (1 − α)B, (7.1) where F stands for foreground, B stands for background, C issynthetic image, and α is the mask. Matting is the reverse process of synthesis. It seeks to obtain the foreground, background and mask from the given image. There are two categories of common digital matting, blue screen matting and natural image matting. (1) Blue Screen Matting. Blue screen matting initially refers to matting the image in a blue background, and then it refers to matting in a given background. Ideally, the background color, foreground color and image color are in a linear relation. With the simple background in a given case, the foreground value can be precisely calculated. Detailed descriptions can be found in Smith and Blinn (1996). Another method is conducted by taking photos twice, one for the foreground, and the other for the background. Then compare every pixelbetween two photo images, anddetermine transparency by color differences. There is a blue flashing phenomenon in a blue screen matting, and the blue screen will reflect some light on the foreground objects. A common solution is to use the twice shooting technology, which requires taking photos of one scene twice. It is conducted by the beam separation mirror. (2) Natural Image Matting. Natural image matting is far more compli- cated than blue screen matting. Rotoscoping (Mitsunaga et al., 1995) is a frequently used natural image matting technique, which requires sufficient patience and skillful interaction. Recently, a series of natural image mat- ting algorithms have been put forward, which have achieved good results, including the Knockout method (Berman et al., 2000), the Bayesian method (Chuang etal., 2001), the Poisson method (Sun et al., 2004),andthereal- time interaction matting algorithm (Wang etal., 2007). These algorithms can be divided into three categories. One is based on the sampling method, which 7.3 Key Technologies for Development and Utilization 167 assumes that the foreground and the background pixels in an unknown region can be estimated by the neighbor-region pixels which are already manually marked. The sampling points of marked pixels are directly used to estimate the α value. The Knockout and theBayesian method mentioned above be- long to this category. The second category is based on the diffusion algorithm, which does not directly estimate the foreground and background colors but assumes that the foreground and background of a local area are both smooth; for example, it assumes that the foreground and background of a local area belong to a constant or linear relationship. The Poisson method and the closed solution method belong to this category. The third category is a com- bination that integrates the sampling method and the diffusion method. The Soft Scissor method, a real-time interaction matting algorithm put forward by Wang etal. (2007) is a typical example of this kind of method. (3) Other matting methods. Some newly developed methods can not only extract the object area, but extract its reflective or refractive parameters, which are called environmental matting methods (Zongker etal., 1999). En- vironmental matting and integration can beregarded as a light reconstruction technology. In order to map and integrate the shadow of objects, researchers have also proposed shadow matting (Chuang etal., 2003), video matting (Chuang et al., 2002),andsoon.

7.3.2 Expression and Extraction of Ancient Murals’ Artistic Style

Artistic style is the embodiment of heritages’ artistic value. Different forms of cultural heritages have different artistic styles. The artistic style of paintings mainlyrelies on the structurallayout, characters, line drawing,color, and so on. We will take the Dunhuang murals as examples to introduce the tech- niques of expression and extraction of pattern styles. Dunhuang is a symbol of ancient Chinese civilization, in which murals play a very important part. IntegratingChinese painting and Western painting,theyclearlyshowhow the nation absorbed and merged foreign cultures and cast them into a na- tional art form with its own characteristics as they evolved. Therefore, they are highly representative. The mural’s pattern style can be expressed by pattern templates. The pattern template, manifesting the artist’s style and the theme, can be ab- stracted as the color feature, the element feature and the layout feature of one category of pattern. Through analyzing the style features of the Dun- huang murals, we have proposed a pattern style expression template based on style templates, namely Duhuang pattern templates, which are defined as follows. Dunhuang pattern template:= theme, element, layout, color .The expres- sion of the artistic style of ancient murals is illustrated in Fig.7.6. (1) Mural Design Theme. The Dunhuang murals reflect the Buddhist artistic themes prevalent at that time. Murals are presented in different forms, based on their themes. Murals 168 7 Digital Development and Utilization ofCultural Heritages’ Information

Fig. 7.6. Expression of artistic style of ancient murals can be divided into seven categories based on their themes in different forms; Buddhist or Bodhisattvas portrait painting, tale painting, godsand spirits painting, Buddhist story painting,thepainting of asage’s story, portrait and decorative painting. 7.3 Key Technologies for Development and Utilization 169

(2) Pattern Elements. Elements, also called source materials, refer to the basic units that consti- tute a pattern. It can be a stroke, a flower, or even a complete landscape painting. All those basic units can be regarded as elements. Anelementcan be described from the semantic information, the outline information and the colorinformation.

pattern element := semantic feature, outline feature, color feature

Semantic feature refers to the description of the category of elements. For example, trefoil belongstothecategory of plants. The category structure can be expressed by a tree, called a category tree. Each node of the tree stands for one category and has its sole ID. Using the tree structure to express the semantic feature of elements facilitates operations, such as adding, deleting and traversing. An outline feature is the shape of an element without color information. The outline features should be expressed by some parametric shapes. Through study of a Fourier description of the shapes, outlines of all kinds of shapes can be approximated by a group of ellipses. A color feature is composed of an appropriate color set, an inappropriate color set and a defaultcolor set. color feature : = appropriate color set, inappropriate color set, default color set

Each color in the set is expressed as follows:

color := (H1, S1,VV1), (H2,S2,VV2) where color refers to a color range, corresponding to the color cube composed by color (H1, S1, V1) and (H2, S2, V2). Any color chosen from the color cube is judged as an appropriate color. (3) Layout. Layout refers to the abstracted deformation and the composition of a pattern. In some applications of patterns, such as the pattern designof textiles and carpets, the layout of the pattern is highly standardized. To support the reusability of the layout, we abstract a pattern element into a point and adopt the expression model based on a lattice structure. Dividing the pattern element into different units, we can easily get a 2D matrix. Fig. 7.7(a) shows a pattern that is divided into sub-areas. (b) is a 2D matrix in which the number 1 or 0 shows whether there is an element in the corresponding sub-area. 170 7 Digital Development and Utilization ofCultural Heritages’ Information

Fig. 7.7. Pattern and its layout matrix. (a) A sample divided pattern; (b) A sample 2D matrix represents whether theelement exists in the pattern

(4) Pattern Color. The color feature of the pattern refers to the area, distribution, harmony and contrast of various colors. We adopt the expression method based on the color layer to express the color knowledgeof the pattern. The color layer of a pattern is a branch of color information, expressed as

Color layer := C, Color ID, Color name, Coun

Choose an HSV color space identical with visual perception to represent the color value of the color layer. Then a color can be expressed as C:= hue, saturation, value . Color ID represents the index value of the color. Color name represents the name of the color, defined by users. The count represents the number of pattern pixels on the color layer, indicating the size of the color area. Based on the above definition of the style template, for one mural we can extract the pattern theme, pattern elements, pattern layout and pattern colors, and then use Dunhuang pattern templates to express it. The pattern themes can be obtained from archaeological results, and the pattern elements can be extracted by the above mentioned object extraction technology. As for color extraction, we can carry out statistical calculation on the mural patterns according to the definition of the pattern color feature. For the pattern layout, there are two approaches. The first method utilizes graphic image processing software such as CorelDraw and Photoshop to carry out interactive drawing along the mural’s edge, and then store it as the defined pattern layout format, as shown in Fig.7.8. The second method starts from the original pattern, utilizes analog knowl- edge to search for the closest layout, makes revisions, presents the layout of the original pattern and transfers it to the defined pattern layout format for storage. 7.3 Key Technologies for Development and Utilization 171

Fig. 7.8. Expression of the Dunhuang pattern theme using pattern templates

7.3.3 Artistic Style Learning Based Re-creation After extracting aculturalheritages’ style, we may make use of the style to carry out re-creation. Different learning methods and creation methods are available for heritages in different styles. We take murals and ancient build- ingsasexamples, and introduce two re-creation methods based on artistic style learning. The former refers to learning and making patterns in simi- lar styles on the basis of the expression and extraction of the artistic styles of paintings, while the latter aims to learn the construction style of ancient buildings and realize rapid modeling of buildings with similar styles.

• Pattern Creation Based on Picture Composition Knowledge

The design process of pattern templates is in fact equal to the process dur- ing which the designer expresses his or her design ideas. The designer may proceed from a certain layout in the layout database or create a new layout, then appoint the categories of elements applicable for various positions, ap- point applicable colors for the background and the whole color styles. Pattern design knowledge can be expressed as the following quadruple: K := D, L, E, C where D is the descriptive expression of composition knowledge, L isthe pattern layout expression, E is the pattern element set and C is the pattern’s color set. The descriptive expression of composition knowledge D containsthetext description of pattern content, such as the pattern’s style name and usage, as well as several sample patterns (in the form of a lattice diagram). The pattern layout expression has been described in L, which refers to the abstracted deformation and composition of a pattern, as shown in Fig. 7.7. The pattern element set E contains the feature description of pattern elements for composition knowledge as well as the existing sample pattern element (in the form of a vector graph). 172 7 Digital Development and Utilization ofCultural Heritages’ Information

The color set for composition knowledge C contains the background color set and the overall color constraint for the composition knowledge. The content expressed by a painting can never be expressed fully by any amount of description information. This is actually equal to the conclusion that “Symbolic information is not capable of expressing the imageinforma- tion which is only to be sensed but not explained”. Therefore, we integrate image processing technologywith the expression methodbased on samples. In the above composition knowledge-based expression model, a sample set has been applied for patterns and pattern elements so that we can use exist- ing image processing and model recognition technologies, such as histogram statistics, spectrum analysis and shape feature analysis, to extract informa- tion of interest from the sample patterns for deduction and thus make bene- ficial supplements for symbolic deduction. A pattern template can generate a series of different patterns in the same style. The generation of patterns is in fact a process of choosing one el- ement from appropriate elements, choosing one color from an appropriate background color set, properly adjusting the position of elements and finally choosing one from the color plans which conforms to the whole color style and achange in the pattern’s color. This process is a comprehensive deduction process of the template’s element feature, color feature and layout feature. The selection of element, background color and color plan can be divided into two categories, interactive selection and automatic selection. In interac- tive selection, the user participates in making choices about the background color, the element for every position and the color plan. In automatic se- lection, the computer will generate a reasonablecolor, layout and element combinations.

• Rapid Modeling of Chinese Ancient Buildings

Some researches have been done on the rapid modeling of Chinese ancient buildings. Liu et al. (2004) have presented a rapid modeling approach driven by the semantic rules. Chinese traditional architecture is strictly bound by the construction rules. Taking palace buildings of the Ming and Qing dynasties as an example, the construction rules of these buildings can be divided into the fixed rules, the designated rules and the uncertain rules. The fixed rules refer to the common features shared by a type of building. The designated rule refers to the part which needs to be determined by the control parameters. The uncertain rules stand for uncertain information contained by buildings, which leads to differences between buildings but does not affect the whole structure of the building. The whole construction process is as follows: (1) Initialize construction parameters, construct basic elements by con- struction and input initial texture in accordance with the initial parameter. 7.3 Key Technologies for Development and Utilization 173

(2) Starting from the basic shape with initial control parameters, the in- terpreter matches the attribute parameters of current models with the control parameters of rules in the rule database and then selects an appropriate rule. (3) The interpreter explains current rules and transforms them into con- struction directions. (4) Carry out iterations at the first step and the second step until the ter- mination or until there are no rules left to choose, and finish the construction of models. Liu etal. (2006B) have proposed a semantic modeling system for the modeling ofChina’s Jiangnan ancient buildings. The system changes basic modeling units (point, line and triangle) into basic semantic components (street, house and so on). In order to make a city modeling system based on semantics, the system adopts the XML and DTD-based verification technol- ogy to control the urban modeling process. The ontology is used to describe the Jiangnan ancient buildings, and the ontological buildings are expressed byaquadruple C (D, W , V , R) where D represents the corresponding con- cept of architecture, such as Jiangnan ancient buildings; W is the domain space, including all actual cases constituting this domain concept; V is the glossary of the construction concept and R istheknowledge set in the field, which can be regarded as a set of norms and rules of examples formed by the glossary of V . First of all, we use an ontology-based semantic model to generate many single buildings, then a random group of ancient buildings. And then we get the best ancient Chinese buildings templates from a pre-establishment database of the Chinese ancient building group modeling rules. Finally, we generate the optimal Chinese ancient buildinggroups via the pre-defined ancient Chinese buildings templates. The generation process is shown in Fig. 7.9.

Fig. 7.9. Rapid modeling for Jiangnan ancient buildingsofChina 174 7 Digital Development and Utilization ofCultural Heritages’ Information

7.3.4 Computer-aided Imitation of Murals

Due to weather and disasters, ancient murals have become very vulnerable. Therefore, the preservation of the Dunhuang murals has already become an urgent task. Imitation is an important part of the preservation of the Dun- huang murals. At present, DunhuangAcademy has formed a set of scientific traditional imitation procedures based on their experiences over decades. Based on thetraditional mural imitation procedure, we have proposeda computer-aided mural imitation approach and the basic process is as follows. Edge detection and fitting:useimagesegmentation and edge detection technology to acquire the murals’ edge and then use piecewise B´´ezier curves to fit it and obtain the vectorized outline drawing. Sample-base line style displacement: Ask artists to draw some strokes in advance, then extract the style as samples, select the stroke in the style that needs to be drawn after obtaining a vectorized outline drawing,andfinally use a sample style conversion algorithm to produce the line drawing. Defect repairing: The outline drawing obtained through segmentation and edge detection directly is not always complete, due to defects in the murals. To obtain a complete drawing, one is required to make an interactive repair to the defected part. To solve this problem, a line drawing database is set up to provide functions such as the edition, entry, searching and substitution of elements. Computer-aided color filling: For line drawings, we mainly adopt interac- tive colorization based on virtual painting.

• Stroke Model of Line Drawings

A stroke model expresses the size and shade style of murals. In order to show murals’ shape, we defined the line drawing stroke model. The line drawing strokemodel is designed to aid copying theDunhuang muralswith the help of computers. It mainlyfocuses on the strokes’ shape and thickness, and has no special requirements regarding the strokes’ color and texture features. Therefore, based on the analysis of work done by predecessors, we propose a simple but very effective skeleton stroke model based on a piecewise cu- bic B´ezier curve (Liu etal., 2006A). The stroke model includes a pressure parameter used to represent thestroke’sthickness. To use a piecewise cubic B´ezier curve has several advantages: first, it’s easy to implement; second, the shape of the curve which is based on joining the piecewise curves is easy to change. Toenforce a joint that appears smooth, the second-to-last control point of one segment must be collinear with the last control point (which is also the first control point of the adjoining segment) and the second control point of the adjoining segment, a condition known as geometric continuity, denoted by G1, see Fig. 7.10 for an example. 7.3 Key Technologies for Development and Utilization 175

Fig. 7.10. A sample stroke model, fit by piecewise B´´ezier curves, in which Pprev i, Pknot i and Pnext i are on the same line

AB´´ezier curve of degree 3 is defined in terms of Bernstein polynomials: 3 P (u) = PkBEZk,3(u)0 u  1 (7.2) k=0 where Pk are thecontrol points, and BEZk,3(u) are the Bernstein polynomi- als of degree 3. We define a stroke 3 L = w,P i : i = 1, ..., n (7.3)

3 Pi = {Pprev i,PPknot i,PPnext i} (7.4) where w is the thickness of the stroke on the joint Pknot i which is the joint between two B´´ezier curves, which we call “anchor point”. Pprev i is the second- to-last control point of one segment, and Pnext i is the second control point of theadjoining segment. The expression can indicate the positional relationship between B´´ezier curves and the stroke: Pknot i isthepoint on the stroke, but Pprev i and Pnext i are just the control points of the B´´ezier curve. A stroke can be generated by using a pressure sensing pen to draw a digitized curve on a tablet and to fit the digitized curve. The pressure on the anchor points is converted to the thickness. So the artists can draw the examplestrokes very easily.

• Edge Detection Technology Based on DynamicProgramming

It needs a long time to develop the automatic general edge detection tech- nology, due to the complexity of images. However, it is also arduous and imprecise to mark the images’ edges manually. Therefore, in this case, we should study human-computer interaction edge detection technologies. There are three types of interactive edge detection algorithms, a region growth algo- rithm (Floodfill), a snake-based active contour algorithm and an interactive dynamic programming method. The previous methods are not so ideal in their effects when the image contains a lot of color areas, complex content or much noise. And the Dunhuang murals are in very complex state due to 176 7 Digital Development and Utilization ofCultural Heritages’ Information the fading of color and deterioration caused by the passing of thousands of years. Therefore, we have conducted research into semi-automatic edge detec- tion technology based on dynamic programming. This method is very flexible and suitable for interaction as it combines well with human judgment and computational capability. Theedge detection algorithm based on dynamic programming is described as follows. As the image is composed of pixels, we should define a side between two pixels, with a weight value which is determined by image edge features (the determination of weight value is the key point in the effect edge detection algorithm). In this way, such an image becomes a weighted graph. Dynamic programming requires the user to determine an initial point and an end point. Through the computation of the shortest path between the initial point and the end point on the weighted graph, the pixels on the shortest path form an edgeof the image object. The entire process is completed interactively, from aninitialpoint to a middle end point, then from a middle end point to the next middleend point. This step is carried out guidedby human judgment until the edgeisfinally closed. The key part of this algorithm is about how to determine the edges’ weight value.

• Sample Based Style Displacement

Through the extraction of the style attribute of sample strokes on the basis of the stroke model definedintheabove,wecanmakestyle displacement by style attribute substitution based on curvature and displacement. Firstly, we withdraw all strokes from the sample chart to make a sample stroke set, and then extract the outline of a certain given chart (which is composed of many lines), called the target line. For a target line, we choose the stroke style from the sample strokes and realize the style attribute displacement. We can also find the style sample that we need from the sample stroke set through computerized automatic matching and replace the target stroke’s style with this stroke’s style. Process all strokes in the target images by the above method and we can finally obtain a target image with the sample style and finish the style conversion. For the same target images, we can obtain many target images in different styles but with the same content, through the above style displacement, by using different control parameters. Fig.7.11showstheeffect produced by the above algorithm.

7.4 Introduction of Typical System for the Development and Utilization

We will take computer aided art design and creating, and semantic modeling for Chinese ancient buildings as examples to show the typical system effects of development and utilization of cultural heritages’ information. 7.4 Introduction of Typical System for the Development and Utilization 177

The effect produced by the computer-aided imitation System. (a) Image of mural; (b) Line drawing produced by the algorithm; (c) Final result after color filling

7.4.1 Computer Aided Art Design and Creating System The intelligent art design and creating system of Zhejiang University was developed for art design and creating, based on the Dunhuang murals. By extracting different types of national and historic meta-images from the mu- rals and using many intelligent methods to re-create new patterns of Dun- huang styles, the system can be used in textile designs. Its goal is to achieve computer-aided design in the process of artistic pattern creation so that the users can enrich elements and composition knowledge for different applica- tion fields, and complete artistic patterns with different styles with the help of the comprehensive intelligent deduction of computers. Based on the analysis and induction of pattern composition knowledge, the system utilizes the generalized intelligence of the open human-computer system to present an effective knowledge expression approach and a pattern generation and deduction mechanism based on fuzzy guidance. As a result, it means the composition rules can be fully expressed and provides direct control approaches for the user’s knowledge entry and pattern generation. Through collecting,editing and utilizing the wisdom of art designers, the system transforms the designers’ composition and deformation experiences into formalized composition knowledge. It turns design materials into digital elements, so that the system can automatically and quickly produce hundreds of millions of different artistic patterns which are unique in composition, new in form and rich in color. With the assistance of this system, we can signif- 178 7 Digital Development and Utilization ofCultural Heritages’ Information icantlyshorten the design renovation cycle, greatly increase color varieties and improve economic and social benefits. The process of creating a pattern is shown as follows. Firstly, extract the elements, styles and colors from the patterns in the Dunhuang murals, as in Fig.7.12. t t c xtrac E Extra

Elements Layouts Color Database Database Database

Fig. 7.12. Original pattern and extracted template knowledge (Caisson, of Tang Dynasty, of the 120th cave)

Secondly, simplify the Dunhuang caisson template, apply it to the creation of a carpet pattern, and thus create the carpet pattern in a similar style, as shown in Fig. 7.13. Finally, applythenewlycreated pattern to textileproducts, as shown in Fig. 7.14. 7.4 Introduction of Typical System for the Development and Utilization 179

Fig. 7.13. Patterns in various styles generated by computer

Fig. 7.14. Exhibition ofpattern applied to a virtual scene

7.4.2 Semantic Modeling for Chinese Ancient Buildings

This system was proposed by Liu etal.(2006A) from Zhejiang University. The system converted the basic modeling elements (point, line and triangle) to basic semantic components (streets, houses, etc). In order to implement the semantic-based city modeling system, the XML and DTD based techniques are applied to control the city modeling process. As the system is able to handle the accurate and rapid modeling of Jiangnan ancient buildings, it can be used for digital preservation of the Jiangnan ancient buildinggroup. Some generated sample buildings are shown in Fig. 7.15. 180 7 Digital Development and Utilization ofCultural Heritages’ Information

Fig. 7.15. Exhibition of intelligent modeling results of Jiangnan ancient buildings. (a) Effect of a single ancient Chinese Jiangnan building; (b) Effectofanancient Jiangnan city References 181

7.5 Summary and Prospects

In this chapter, we mainly focus on the digitalized development and uti- lization of cultural heritages’ information, including the extraction of the heritages’ noumenal information, the mining of valuable implicit information and re-creation based on the above information. Currently, the digitalized development and utilization of cultural her- itages’ information is now at its initial stage, with few systematic researches. Technically, object extraction, formalized expression of styles and style based pattern creation can mine implicit information about part of the heritages and can be used for creation, mainly for murals. However, key technologies for mining more implicit information about more types of heritages still remain unresolved. As for applications, we now mainly focus on meeting the requirements of some special applications, such as the creation of an intelligent pattern creation system for the textile industry, computer-aided imitation of murals aimed at improving working efficiency. However, digitaldevelopment and utilization will become more mature and more widely used in applications. Interdisciplinary research into the value of our cultural heritages involves several subjects, including archaeology, his- tory, natural science and sociology. For example, for murals we should con- duct research into their painting styles fromanaestheticviewpoint,andthen make extractions and creations on this basis. From the point of view of his- toric culture, the content of murals reflect the social customs, architectural styles, dress styles and even historic events of that time, which involves his- toric sociology, while the study of color and craftsmanship of murals involves the natural sciences. In addition, the few existing examples of development and utilization are specially designed for some specific applications and there are no concept frameworks in existence. However, the digital development and utilization of cultural heritages will become more systemized for more and more types of heritages.

References

Berman A, Vlahos P, Dadourian A (2000) Comprehensive method for removing from an image the background surrounding a selected object, U.S.PatentNo. 6134346 Chuang YY, Curless B, Salesin DH, Szeliski R (2001) A Bayesian Approach to Digital Matting. In: Proceedingsof IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Washington 2:264–271 Chuang YY, Agarwala A, Curless B, et al. (2002) Video matting of complex scenes. In: Proceedings of ACM SIGGRAPH 2002, San An tonio, Texas 243– 248 Chuang YY, Goldman DB and Curless B (2003) Shadow Matting and Composit- ing. In: Proceedings of ACM SIGGRAPH 2003, San Diego 494–500 182 7 Digital Development and Utilization ofCultural Heritages’ Information

LinSY,PanLF,DouF,ShiJY(2007) A Survey on Digital Matting. In Journal of Computer-Aided Design & Computer Graphics 19(4):473-479 (in Chinese) Liu H, Hua W, Zhou D, Bao FJ (2004) Semantic Rule-driven Modeling for Chinese Traditional Architectures. In Journal of Computer-Aided Design & Computer Graphics 16(10):1335–1340 (in Chinese) Liu JM, Lu DM, Shi XF (2006A) Interactive Sketch Generation for Dunhuang Frescoes. In First International Conference of Technologies for E-Learning and Digital Entertainment. Lecture Notes in Computer Science. 3942:943–946 LiuY,XuCF,PanZG,PanYH(2006B) Semantic Modeling for Traditional Architecture of Digital Heritage. Computers & Graphics 30(5):800–814 Mitsunaga T, Yokoyama T, Totsuka T (1995) Autokey: Human Assisted Key Extraction. In: Proceedings of ACM SIGGRAPH 1995, Los Angeles 265–272 Porter T, Duff T (1984) Compositing Digital Images. In: Proceedings of ACM SIGGRAPH 1984, New York 253–259 Smith AR, Blinn JF(1996) Blue Screen Matting: Computer Graphics Proceedings. In: ProceedingsofACM SIGGRAPH 1996, New Orleans 259–268 Sun J, Jia JY, Tang CK,Shum HY (2004) Poisson Matting. ACM Transactions on Graphics 23(3):315–321 Zongker D, Werner D and Curless B (1999) Environment Matting and Composit- ing. In: Proceedings of ACM SIGGRAPH 1999, Los Angeles 205–214 8 Applications of Digital Preservation Technologies for Cultural Heritages

In the previous chapters, we introduced many aspects of some key tech- nologies including digitalization, research aiding, conservation aiding, digital exhibition and digital utilization. These technologies have been successfully applied in some applications in many key heritage conservation units. Mean- while, these applications also draw on some new requirements from these technologies. For site preservation, total stations, 3D scanners and other devices are utilized to make measurements and 3D acquisitions of some sites, and the 3S technology is also applied to manage the sites information. Some sites even utilize some techniques in dynamic information acquisition of the excavation process, and then simulate and represent thewhole excavation process. For example, the “Eternal Egypt” digital museum, developed by IBM (2005), is based on the rich and historic Egyptian pyramid culture, and is officially open to all visitors online. The digital museum provides worldwide access toEgyptian history of more than 5,000 years using pictures, videos and 3D contents. Visitors at the real site can enjoy their journey under the real-time guidance of wireless devices. For grotto preservation, digital technologies are used for the acquisition and conservation of sculptures and murals. Digital technologies are now used tocollect environmentaldata and assist researchers in analyzing the data, and they are also used to restore and imitate various kinds of painted sculptures and murals. For example, computer image processing technology can help experts to extract line-drawings and fill in colors, improving working efficiency greatly. For portable relics, a great number of exquisite and precious relics have been digitalized and stored in computers for digital exhibition, for research and digital restoration. For example, the Northwest University ofChina has been working on the digital mosaic ofQin terracotta warriors. The researchers input the outline of broken relics into computers by 3D scanning and the 184 8 Applications of Digital Preservation Technologies for Cultural Heritages computer will analyze them and simulate the matching process to find a perfect matching one. This chapter focuses on the applications of technologies mentioned from chapters 3 to 7. In Section 8.1, we will take the DunhuangGrottoes, China, as an example to introduce the digital preservation projects for grottoes. In Section 8.2, the digital acquisition and exhibition of the excavation field of heritage sites will be introduced by taking the Jinsha Site, China, as an example. In Section 8.3, we will focus on the digital reconstruction project of heritage sites taking the Hemudu Site, China, as an example. In Section 8.4, we will introduce the digital exhibition of ancient exhibits, taking the Liangzhu Site, China, as an example. In Section 8.5, we will introduce the Digital Michelangelo Project for portable heritages.

8.1 Digital Preservation Project for the Mogao Grottoes

The Dunhuang Mogao Grottoes, firstly built in 366 A.D., contains one of the largest and best preserved collections of Buddhist art treasures in the world. Located in Dunhuang, Gansu, China, they are now rich in content and large in scale after tens of dynasties. It is a huge comprehensive art palace, integrating architecture, painted sculptures and murals. The Mogao Grottoes consist of 5 floors, 492 caves, 45,000 m2 of murals, 2400 painted sculptures, 4000 Apasaras and 5 Tang &Song-style wooden buildings, thousands of lotus pillars and floral floor tiles, and are 1600 m in length from north to south. The Library Cave, found at the beginning of the 20th century, has 50,000∼60,000 pieces of transcribed Buddhism scriptures, documents and relics from the 4th to the 10th centuries. This has drawn extremely great attention, and a new subject called Dunhuangology emerged. Fig. 8.1 (a) shows the famous nine floors of the Mogao Grottoes and Fig. 8.1 (b) is the outside view of the Mogao Grottoes. Fig. 8.1 (c) shows the Overview of the 61st cave, Fig. 8.1 (d) shows the beautiful murals from the 285th cave, Fig. 8.1 (e) shows Buddha status in the 285th cave, Fig. 8.1 (f) shows the south wall of processional road in the 16th cave, and Fig. 8.1 (g) shows mural in the 85th cave. The ancient artists not only inherited the excellent artistic tradition of the Han and other ethnic groups of the Western Regions, but also absorbed and adopted methods of expression from abroad. To preserve this art treasury, China set up the DunhuangArt Research Institute in the 1940s, an organization specializing in the study and preser- vation of the Mogao Grottoes, and set up the Dunhuang Heritage Research Institute in 1950, which was later expanded to become DunhuangAcademy in 1984. Dunhuang Academy has achieved great accomplishments in the conser- vation and study of the Dunhuang Grottoes, in cooperation with universities and institutes both in China and abroad, such as the Institute ofComputing Technology of the Chinese Academy of Sciences, Zhejiang University, China, 8.1 Digital Preservation Project for the Mogao Grottoes 185

Fig. 8.1. The Dunhuang Mogao Grottoes. (a) Theninefloors;(b) Outside view; (c) Painted sculptures in the 61st cave; (d) Mural in the 285th cave; (e) Buddha status in the 158th cave; (f) The south wall of processional road in the 16th cave; (g) Mural in the 85th cave 186 8 Applications of Digital Preservation Technologies for Cultural Heritages the Tokyo National Institute for Cultural Properties, the Getty Conservation Institute of the U.S.A, and the Mellon Foundation (USA).

8.1.1 Digital Acquisition of the Dunhuang Grottoes

Acquisition is the basis for the conservation, study and utilization of the Dunhuang heritages. To acquire and store high-precision information about architectures, murals and painted sculptures from the Dunhuang Grottoes, 3D scanning, digital photography, 3D modeling and image processing tech- nologies have been applied. The goal is to construct digital information about the heritages in the computer for sharing, conservation, restoration, research, exhibition, utilization, and so on. Dunhuang Academy has now acquired high- resolution digital murals of more than 20 murals, and more than 40 caves for digital exhibition.

• High-resolution Acquisition of the Dunhuang Murals

The Dunhuang murals lie on the interior walls of the grottoes. They are huge in size, rich in content and large in quantity. And the grottoes are greatly different in size. Therefore, it is hard to designashooting device. By thetraditional method,workers shoot using atripod at positions marked in advance, which is really very inconvenient. Zhejiang University proposed a set of standard and integrated devices and an operation process to record ancient murals digitally, convenient to operate, high in precision and small in errors. The device consists of platform architecture, a flash lamps set and a tripod head set on a versatile arm, a camera set on a tripod head, a versatile arm set on a rotatable horizontal slip connection component, which connects to a fore-and-aft sliding stick. And the fore-and-aft sliding stick is connected to three main vertical knightheads, a base set under knightheads. Fig. 8.2 shows the shooting device and how the device works. Theshooting steps include: (1) Making the shooting plan based on the architectural construction of the grotto. (2) Designing forms to record the shooting information, including the mark of the grotto, the location of the mural, shutter, aperture, position of every photo on one mural,and so on. (3) Selecting the camera and the lens to accord with the precise requirements and defining the shooting distance. (4) In accordance with the actual screen shots, adjusting lighting, using the color card to adjust aperture and shutter. (5) Determining the program relating to the mobile line of the device. (6) Shooting and real-time recording of the shooting situations. (7) Checking the integrity and clarity of the images. 8.1 Digital Preservation Project for the Mogao Grottoes 187

(8) Creating a mosaic of the photos with software. Some high resolution digital murals was already shown in Fig. 3.25. Now, all the murals from over 20 Mogao Grotto Caves have been digitized. The minimum resolution is at 75 pixels per inch, and the maximum resolution is at 300 pixels per inch.

Fig. 8.2. The mural acquisition device and theworking scene

• High-precision Acquisition of Painted Sculptures in the Grottoes

Painted sculptures are complicated in shape and are of various sizes. Zhe- jiang University has adopted the 3rd Tech (2009) DeltaSphere-3000 3D laser scanner, having a precision of about 1 mm, to digitally acquire 3D models of painted sculptures and caves. Most of the current 3D scanners can only acquire geometric models and are not able to acquire the color information, while 3D scanners that are able to acquire chromatic models, suffer from lack of fidelity and resolution. Therefore, Zhejiang University has also proposed a multi-texture mapping method to acquire the colorful painted sculptures, by using high precision digital cameras. The acquisition processes are shown in Fig. 8.3. 3D models and photos are digitized by 3D scanner and digital camera first, and then multiple 3D model patches and photo texture s are registered together, to generate a full angle color 3D model. Fig. 3.21 shows the digitalization result from the 158th cave of the Mogao Grottoes. Now, over 10 Mogao Grotto caves with all the sculptures inside have been 3D digitalized. 188 8 Applications of Digital Preservation Technologies for Cultural Heritages

Fig. 8.3. Acquisition process for painted sculptures and caves

8.1.2 Microclimate Monitoring in the Mogao Grottoes

Research indicates that microclimatic conditions such as temperature, hu- midity and carbon dioxide density in caves greatly influence the heritage conservation (Camuffo, 1998)(Shi and Zhang, 1997). Therefore, microcli- mate monitoring is quite an important part of the conservation of cultural heritages. First, real-time microclimate data allow the estimation of the cur- rent microclimate environment in culturalheritage sites, and emergent con- servation policy adjustment in case the monitored microclimate parameters exceed some thresholds. Second, long term historical microclimate records are helpful in analyzing the state of the cultural heritageinthepastand in investigating the role of the environment in the deterioration of cultural heritages. However, traditional wired weather monitoring systems such as Camp- bell’s CR1000 are not suitable for microclimatic monitoring in wildland cul- tural heritage sites. To deploy wire in wildland cultural heritage sites is fre- quently quite difficult, and may be inappropriate. For example, in most caves of the Mogao Grottoes, all walls and ceiling are painted with murals. Even field bricks in some caves are also historical relics. This leads to the inappro- priateness of wire deployment in the caves. DunhuangAcademy and Zhejiang University started a cooperative study in the area of microclimate monitoring of the Mogao Grottoes in 2006. In contrast to traditional weather monitoring projects, WSN (Wireless Sensor Network) technology was employed to enable real-time microclimate data collection, while avoiding any wire deployment in the caves. A tiered communication architecture which integrates a wireless sensor network,along-distance wireless polling network and theInternetwasdevel- oped to fitthecomplexgeography of the Mogao Grottoes. Inside the caves, 8.1 Digital Preservation Project for the Mogao Grottoes 189 battery-powered WSN nodes are used to eliminate the requirement of wire deployment. Two types of data sensors, an RH/Temp data sensor and a CO2 data sensor were built to monitor temperature/humidity and CO2 density respectively. In order to save energy, all nodes in the same cave will acti- vate and sleep synchronously with a configurable working cycle. Outside the caves, a long-distance wireless polling network is used to relay microclimatic data to a remote base station. At the base station, a data service server has been setup to provide data storage and sharing on the Internet. In order to ensure the stability of the system in long-term monitoring, the devices were specially designed to survive in a harsh deployment environment, and fault tolerance strategies have been developed for efficient fault recovery. Fig. 8.4 shows the system architecture applied in the Dunhuang Mogao Grottoes.

Fig. 8.4. Monitoring system architecture

The system has been deployed in 57 typical caves of the Mogao Grottoes, including 241 data sensors, 22 data sinks, 1 data router and 1 data service server, as shown in Fig.8.5. Fig. 8.6 gives deployment examples of data sensors, data sinks and data router. Real-time microclimatic data in caves can be transferred to conser- vators at the academy within 1 minute. Therefore, emergency actions, such as reducing the number of visitors, will be immediately executed once the microclimate in the caves exceedssomethreshold.Thepreliminary analysis of gathered data shows that the number of visitors has the greatest impact onthemicroclimatic change in the caves. Fig. 8.7 shows the humidity and CO2 density variation, compared to the number of visitors in C328, in one day. It can be observed that CO2 density even reached more than 2500 ppm when the cave was full of visitors. CO2 density at such a highlevelisnotonlyharmful to murals, but also to people. 190 8 Applications of Digital Preservation Technologies for Cultural Heritages

Fig. 8.5. System deployment in a bird’s eye map of the Mogao Grottoes

Fig. 8.6. Monitoring system deployment in the Dunhuang Mogao Grottoes. (a) RH/Tempdatasensorincave;(b) RH/Temp and CO2 data sensor at the cave entrance; (c) Data sink deployed on street lamp; (d) Data router deployed upon the roof 8.1 Digital Preservation Project for the Mogao Grottoes 191

Fig. 8.7. Microclimate data gathered from the 328th cave. (a) Humidity in the 328th cave; (b) CO2 density in the 328th cave

8.1.3 Digitally-Aided Imitation of the Dunhuang Murals

Because the Dunhuang Grottoes lie in the Gobi Desert region, where they are corrodedbysandstorms and other poor weather all year long, themuralsare quite fragile and under threat from many diseases. Dunhuang Academy has accumulated several decades of experience and set up a series of traditional procedures on how to imitate murals. The procedure mainly consists of five steps, namelytaking digital photos, enlarging photos by projecting them onto walls (to their original size), making revisions and extracting line drawings with reference to the original murals, copying the drawings onto a piece of rice paper and filling in colors. Among the above steps, line drawing extracting takes much time. Generally,amuralof several square meters takes years and, once a mistake is committed, previous work will probably be lost. Digitally-aided imitation provides a new and effective method to conserve and restore the murals. The digital method has no risk of damaging the heritages but many advantages, such as highefficiency, and is repeatable. Zhejiang University and Dunhuang Academy collaborated at developing key 192 8 Applications of Digital Preservation Technologies for Cultural Heritages technologies and systems for digitally-aided imitation of murals (Liu et al., 2006). With the assistance of imitation software, it only took several days to copy a 2 m2 mural in the 275th cave of the Mogao Grottoes. Also, we have finished disease marking of a2m2 mural in the 206th cave. The developed system can extract line drawingsof a mural to meet artists’ needs. Based on the line drawings, artists can study drawing skills and styles in different periods, which are ofgreat research value. Four steps are taken, which are edge detection, sample-base line style displacement, defect repairing and color filling. Firstly, the system extracts the line drawingsof murals by edge detection techniques. Secondly, artists can select and displace the lines into different styles based on the prepared line samples. Thirdly, for those defective regions caused by undetectable reasons, the system can displace other items for them by sample learning and vectorization. Lastly, the artists can fill colors on the line drawings. Fig. 8.8 shows these four steps respectively, from (a) to (d). Typical imitation results are showninFig.7.11.

8.1.4 Color Simulation of the Dunhuang Murals

Sometimes heritages should be recovered to their original appearance or sta- tus. It requires us to find out the cause of heritagedamageandmakepre- vention plans accordingly. To achieve the above goal, color simulation tech- nologies are needed. Zhejiang University and Dunhuang Academy have col- laboratively developed a system (Shi et al., 2006) which can restore murals from their current status to their original appearance, or simulate the color changing process by modeling the color changing curves. We use murals in different caves and at different times to test system effects. During the test, the artists can judge whether the fading simulation, virtual restoration and disease simulation effects are correct and conform to objective rules and ex- perience. Fig. 5.13 shows the results applied to conduct a fading simulation of the Buddha from its original appearance to its current appearance in the 205th cave.

8.1.5 Dunhuang-style Pattern Creation and Product Development

After a one thousand-year, ten-dynasty evolvement, the Dunhuang murals have achieved unique styles and features in their themes, content, layout and colors. Zhejiang University and DunhuangAcademy have collaboratively de- veloped a Dunhuang-style pattern creation and product development system which uses the digital image information of the Dunhuang murals to mine their artistic connotation, and utilizes theintelligent method to simulate the creation of murals in Dunhuang-style. This can not only be used to produce artistic works for our appreciation but also as patterns for fabric design in thetextileindustry. 8.1 Digital Preservation Project for the Mogao Grottoes 193

Basic steps of digitally-aided mural imitation. (a) Edge detection; (b) Sample-base line style displacement; (c) Defect repairing: by manually line drawing the defective area and matching with a sample base, the system will select the most suitable item; (d) Color filling using handwriting pad 194 8 Applications of Digital Preservation Technologies for Cultural Heritages

We use style extraction techniques to extract items from a series of mu- rals in similar styles, including the overall layout, the element and the color styles of the murals. The system then creates mural patterns based on the ex- tracted style knowledgeand user requirements. The computer aided creation system for Dunhuang-style patterns adopts a design method based on style templates. In other words, it utilizes the element, layout and colors of the Dunhuang murals to design the templates in Dunhuang styles and themes, and then creates new patterns by computer. The system’s goal is to apply computer aided design to the creation of art patterns, so that users can enrich the elements and composition knowledge for different fields. Utilizing the generalized intelligence of open human-computer system, the system proposes an effective knowledge expression method and a fuzzy logic guidance-based pattern generation deduction mechanism through analyzing and inducing the pattern composition knowledge. It ensures the picture composition rules are expressed adequately and provides a direct controlling method for users to input knowledgeandgenerate the final pat- tern. The whole system consists of three independent sub-systems: the manage- ment sub-system for the pattern composition knowledge base, the manage- ment sub-system for the element base, and the intelligent pattern generation sub-system. The management sub-system for the pattern composition knowl- edge base not only provides users with a visual interface, but also collects the pattern composition and transformation experience of designers and ab- stracts them into composition knowledge,soastoprovideadeductive basis for generating patterns. The rules of pattern composition include the atomic rule and the chain rule. They are under the management of a pattern com- position rule base and can be added to, edited, modified and deleted. The element base management sub-system adopts standard metafiles as an in- put format. It unscrambles metafiles into stroke information, and stores the information in the standard database files based on common database man- agement systems like Microsoft Access. The information is taken as basic units for creating patterns. Similarly, operations such as adding,modifying, searching and deleting can be conducted in the element base management sub-system. Fig. 8.9 shows a series of typical patterns created. Inaddition, thesystemhas already been applied in thetextileindustry. It can automaticallyand rapidly design artistic patterns which are unique in composition, novel in form and rich in color. We can shorten the time taken to renew patterns greatly, increase product varieties and designs significantly and bring remarkable economic and social benefits. Fig. 8.10 shows pattern creation examples produced by our system. 8.1 Digital Preservation Project for the Mogao Grottoes 195

Fig. 8.9. Some image patterns created by the Dunhuang-style pattern creation and product development system

Fig. 8.10. The product developments examples in the textile industry 196 8 Applications of Digital Preservation Technologies for Cultural Heritages 8.2 Digital Preservation Project for the Jinsha Site

Found in 2001, the Jinsha Site lies in the western suburb of Chengdu, Sichuan Province, China. Since then, an archaeological excavation on the largest scale was initiated in Sichuan province, and the capital of Shu, an ancient state, in deep slumber for three thousand years was aroused and presented to people. The Jinsha Site has thousands of precious heritages, most of which belong to the later period of the Shang Dynasty and the earlier period of the Western Zhou Dynasty, and a small part which belong to the Spring and Autumn Period. TheexcavationoftheJinshaSiteisofgreat significance for the study of the ancient history and culture of the Shu state. In 2006, the Jinsha Site was listed in the 6th group of the national key relic preservation unit ofChina. In 2007, Jinsha Site Museum opened, exhibiting thousands of rare precious treasures to people, including the Sun-Bird and the Golden Mask. The golden wares that have been unearthed in the Jingsha Site include more than 30 golden masks, golden belts, circular golden accouterments and bell-mouthed golden accouterments. The jade wares unearthed are rich in variety and exquisite in style. Among these jade wares, a 22 cm high jade cong has an emerald color and extremely fine engraving, with mini-graving decorative patterns as thin as a hair and a man-shaped figure on its surface. It is absolutely a national treasure. The unearthed bronzes are mostly small in size, including a bronze human-sculpture, a bronze Yuan, bronze dagger, bronze bell,etc.There are also 170 unearthed stonewares, including astone man, stone tiger, stone snake, stone turtle, and so on. Among them, the kneeler human-sculpture is extremely lifelike as is the stone tiger. The jade dagger and jade yuan unearthed from the sites indicate that Jinsha culture is not isolated but connected with the culture of the Yellow River Valley and the Liangzhu culture on the lower reaches of the Yangtze River, which once again proves that Chinese culture has multiple sources. Fig.8.11showssome relics from the Jinsha Site. In 2005, the State Administration of Cultural Heritage officially an- nounced that the imageof the Sun-Bird would be the symbol of the Chinese culturalheritage, as shown in Fig. 8.12.

8.2.1 Information Management and Sharingfor Archaeological Sites

Chengdu Cultural Relics Archaeological Research Institute has cooperated with Zhejiang University in the digitalization of archaeological sites, and has developed a information management and sharing system, which can be applied to build digital archives for the information of the excavation unit, culturallayers and relics unearthed at sites throughdigital acquisition, digital processing and archaeological research.Itisalso applied to provide research assistance and a digital exhibition for world-wide Internet sharing. 8.2 Digital Preservation Project for the Jinsha Site 197

Fig. 8.11. Cultural relics from the Jinsha Site. (a) Gold mask; (b) Green jade cong; (c) Another gold mask; (d) Sun-bird gold foil

Fig. 8.12. The symbol ofChina cultural heritage

The system is composed of three phases, the excavation phase, research phase and exhibition phase. Inthe excavation phase, archaeological stratigraphyisalways utilized to make excavations. Every deposit is discovered, excavated, and finally disap- pears. The information lost is often ofgreat significance for later archaeologi- cal study. Therefore, in this phase, the most urgent requirement of digitaliza- tion is to maximize the acquisition and recording of all kinds of information. 198 8 Applications of Digital Preservation Technologies for Cultural Heritages

Key technologies include acquisition of the excavation site and acquisition of museum preserved cultural relics, introduced in Chapter 3. In the research phase, the requirements include assisting experts to re- trieve the information during excavation, record their research results and aid them in mapping and measurement. Therefore, in this phase it is nec- essary to truthfully show the archaeologist the information obtained and recorded in the excavation phase, and to support the recording and sharing of their research achievements. The research phase is also an important part of heritage site preservation, as it summarizes, concludes and improves the excavation results obtained in the field. In this phase, technologies such as analysis and computer aided mapping are adopted,which are introduced in Chapter 4. In the exhibition phase, the requirements include maximizing the dis- play information of the achievements acquired and researched, and exhibiting them publicly via theInternet.Themajor work in this phase is to display the information obtained in the excavation phase and the results obtained in the research phase, represent the original appearance of the site and provide vis- itors with the digital means to interact with sites. Key technologies involved in this phase include online exhibition technologies introduced in Chapter 6. Theabove process is shown in Fig. 8.13.

Fig. 8.13. Process of information management and sharingfor archaeological sites

8.2.2 Acquisition and Exhibition of the Excavation Field The system has collected the excavation field information of two groups of excavation units in the excavation field of the Jinsha Site. The first group 8.2 Digital Preservation Project for the Jinsha Site 199 includes four excavation units, IXT9641, T9642, T9741, and T9742, and the second group includes four units IXT7128, T7129, T7228, and T7229. Each excavation unit is a square with 5 m edges. As the crossbar among the units inonegroup has been broken, eachgroup is a square with 10 m edges. The first group has several graves and ash pits and the second group has one ash pit. According to different precision requirements and difficulties, we adopted different acquisition methods for the excavation unit and the relics in the unit. For graves and ash pits, taking theash pit H4452 in the second group as an example, we scanned it with the Nmlab 3D Scanner (structured light 3D scanner) developed by ourselves, and then by de-noising, triangularization and merging, we obtained an integrated 3D model. As for the excavation unit itself, we adopted multi-camera shooting and utilized image-based modeling to get a 3D model which is of lower precision. Finally, we conducted merging and texture mapping on the relic model and the unit model. The process is illustrated in Fig. 8.14.

Acquisition of the Jinsha Site

For units already excavated, we also developed a method that can recon- struct the outline in its original appearance, according to the drawings and photos of unit walls. For example, Unit IT8206 of the Jinsha Site excavated in 2006 has many precious cultural relics that have been unearthed,which are of great significance for the study of rites of sacrifice. Hence it is urgent to conduct digital reconstruction of this unit. The system firstly scans the western and southern walls’ outline drawingstoobtain digital images, and then utilizes a quadric Bezier curve to simulate and combine curves, shown in Fig. 8.15. Because the northern and eastern walls were damaged during the construction of Shufeng Garden, we connect them with straight lines instead. 200 8 Applications of Digital Preservation Technologies for Cultural Heritages

Fig. 8.15. Reconstruction of the excavation unit of the Jinsha Site: vectorization of theeastwallof the unit IT8206

After digitalization of the curves of all walls of the unit, the qhull, a convex hull library, is adopted by the system to generate the point cloud of points inside the unit, which is specially used for generating the convex hulls (Barber, 1996). For point cloud color, the system selects images’ colors in the corresponding cultural layer. In addition, the system restored the 3D scanned excavated relics to their original position, based on the recordingdata during excavation. Fig.8.16 shows the reconstruction result for Unit IT8206.

Fig. 8.16. Reconstruction effect of unit IT8206, the Jinsha Site, China. (a) Overview of unit IT8206; (b) Local relic deposition

As mentioned in the previous paragraph, the system utilizes an Nmlab 3D Scanner to obtain 3D models of relics of the Jinsha Site digitally. Fig. 8.17 shows some model data scanned for the Jinsha Site, as well as the scanning work scene and 3D scan process. A 3-channel high resolution multi-projector system is adopted for the 3D digital exhibition of the excavation field of the Jinsha Site, which can make visitors feel like being in the field and represents each step of the excavation to the visitors. Through controlling the view point of the visitors, the system can help visitors know more about the excavation field and enrich the ways of displaying cultural relics, so that visitors can know where relics are from and how they are excavated. Fig. 6.25 is the picture of the excavation field displayedbythemulti-projector system. 8.3 Digital Reconstruction Project of the Hemudu Site 201

Fig. 8.17. Digital acquisition process and effect of the Jinsha relics. (a) Jade Chisel; (b) Polygon meshes of Jade Chisel; (c) Point clouds of Kneeling Statue; (d) Polygon meshed of a Jade Cong; (e) Structured light scanner acquisition scene; (f) 3D acquisition process 8.3 Digital Reconstruction Project of the Hemudu Site

The Hemudu Site lies in Yuyao, Zhejiang Province, China. It was discovered by chance in the summer of 1973 by some farmers when they were building a storm drainage station. The Hemudu Site is the earliest Neolithic site in southeast China. The discovery of the Hemudu Site indicates that there was an advanced primitive culture on the lower reaches of the Yangtze River six or seven thousand years ago. Just liketheYellow River Valley, theYangtze Valley is also the origin of ancient Chinese culture. The Hemudu Site belongs totheNeolithic period.Through excavation, it was ascertained that thesite consists offour cultural layers. During the excavation, many production tools, 202 8 Applications of Digital Preservation Technologies for Cultural Heritages domestic utensils and decorative wares made of stone, bone, wood and pot- tery have been unearthed. Things discovered also include a great quantity of animal and plant remains, huge pieces of wooden architectural relics and agreat amount of artificially cultivated rice. In 1982, the Hemudu Site was listed in the 2nd group of the national key relic preservation unit of China. Hemudu Site Museum was opened to the public in 1993. In addition, a reconstructed site exhibition area was also arranged near the museum, which reconstructs ancient scenes of the Hemudu people. Fig.8.18showsthe museum and some relics excavated from the Hemudu Site.

Fig. 8.18. Hemudu Site Museum. (a) Outside view of the museum; (b) Exhibition area of reconstructed site; (c) Lacquer wooden bowl; (d) Ceramic bowl with pig motif; (e) One exhibition hall; (f) Rice grains 8.3 Digital Reconstruction Project of the Hemudu Site 203

In 2004, Hemudu Site Museum and the Computer Science Department of Zhejiang University began to cooperate in research on the digital reconstruc- tion of the archaeological site. Being one of the important Neolithic sites, the Hemudu Site is ofgreat significance in both research work and as an exhibition. Therefore, it is both meaningful and challenging to restore the site and show it to visitors. Digital technology provides a new method for restoring the site. For the reconstruction of the Hemudu Site, the exhibition and interactive technology introduced in Chapter 6 is adopted. According to the archaeological results, the Hemudu Site is surrounded by mountains on three sides and a lake on the other side with the Zhilin River flowing through it. Most of the site is covered by everglades. There are many stilt style buildings on the site, and most of the ancient people lived on fishing and hunting. Based on the above research results, we first designed the geographical distribution of the ancient site, as shown in Fig. 8.19.

Fig. 8.19. Distribution map of the Hemudu Site

Some reconstruction results are shown in Fig. 8.20. The ancient village at the Hemudu Site that has been virtually recon- structed is displayed on a 3D curved screen in an interactive way, as shown in Fig. 6.14. Considering the real-time requirement, we simplified the model of the reconstructed scene, especially the triangle meshes and introduced LOD technology. Wehavealsodesigned an interactive scene of the spearing offishes. The scene mainly utilize electromagnetic tracking technologytofollow the user’s 204 8 Applications of Digital Preservation Technologies for Cultural Heritages

Reconstruction effect of the Hemudu Site. (a) Stilt style building under construction; (b) Completed building; (c) Everglade and stilt style buildings hand position and direction, and use a data-glove to determine the hand’s gestures. According to the hand’s position, direction and gesture, the posi- tion, direction and speed of the virtual tool are determined, which can then determine the effect and result of the spearing of fish.

8.4 Digital Exhibition of the Liangzhu Relics

Liangzhu Culture is an important ancient culture in the Tai Lake Valley on the lower reaches of the Yangtze River, in which copper and stoneware were both used, and which are about four to five thousand years old. Archae- ological investigation and excavation over more than half a century reveal that Liangzhu Culture is located around the Tai Lake area and that the Liangzhu Site in Yuhang, Zhejiang Province, China, is the distribution cen- ter of Liangzhu Culture. During the period of Liangzhu culture, agriculture moved into the age of the plow. The handicraft industry tended to be increasingly professional, and the jade carving industry was especially advanced. Besides, the creation of large-size jade sacrificial wares also ushered in China’s ritualistic society. And 8.4 Digital Exhibition of the Liangzhu Relics 205 the difference between the graves of the nobility and those of common people reveals a large social polarization. The “primitive characters” on unearthed relics are regarded as the prelude to mature Chinese characters. Jade wares are the quintessence of the material and spiritual culture cre- ated by our Liangzhu ancestors. The jade wares of Liangzhu culture represent the peak of China’s prehistoric culture, which are great in quantity, abun- dant in variety and consummate in carving skill. The Liangzhu culture is also famous for the black pottery that is fine in texture, regular in shape and various in its variety. Its uses are clear to us, and the use in particular of pottery tripods, pottery Dou vessels, and pottery pot have strong Liangzhu cultural characteristics (Fig. 8.21).

Fig. 8.21. Relics excavated from the Liangzhu Site. (a) Jade Cong; (b) Fork-shaped ware; (c) Pottery container; (d) Jade Bi

How to utilize digital technologies to exhibit precious Liangzhu cultural relics, and arrange and display them in a way preferred by users is a subject of great significance. The digital exhibition and display project of Liangzhu cultural relics mainly includes the following two aspects. Firstly, we acquire and collect data about the exquisite items. We mainly acquire shape and surface texture information of the wares and ensure acquisition precision, so as to lay a basis for future digital exhibition and display. Secondly, we display the digital exhibits on a network. By exhibiting them in a digital way, we can avoid the time and space limitations of museum exhibitions, enhance the 206 8 Applications of Digital Preservation Technologies for Cultural Heritages propagation of Liangzhu culture, and provide an important supplement to the museum exhibition. The developed Nmlab 3D scanner was used to acquire the Liangzhu relics. The prototype system can finish the 3D point cloud data collection in 20 s for an object rangingfrom 10 cm×10 cm×10 cm to 60 cm × 60 cm × 60 cm in length, width and height. The scanner and the scanning scene for the Liangzhurelics are shown in Fig. 8.22.

Fig. 8.22. Working scene of acquisition for the Liangzhu relics

After the 3D scanning, we can obtain point cloud data about the relics’ surface. Through de-noising, refining, triangularization and merging of point cloud data, an integrated 3-D model of the relic can be obtained. With the digital images taken by high-precision camera, the technologyof multi-texture binding and blending, we construct an integrated 3D model, which is intro- duced in Chapter 2 and Chapter 3 in detail. The above process is shown in Fig. 8.23. The results of some digitilized the Liangzhu cultural relics are shown in Fig. 8.24. In view of the exquisiteness of the Liangzhu relics, we should retain as many details as possible for their 3-D models and textures in the digitalization process. Usually, a common model file of one relic is approximately from 50 M bytes to 100 M bytes. To make the 3-D high-precision appearance of relics be seen efficiently on the Internet, we adopted the remote rendering method. Itmeansthat therendering is conducted at the server-end,and then the rendering result is transferred by network. This method can not only reduce the requirements of the user’s computer hardware but can also reduce the time spent on transferring models by network effectively. The system adopts 8.4 Digital Exhibition of the Liangzhu Relics 207

Fig. 8.23. Acquisition of the Liangzhu relics

Fig. 8.24. Digitalized cultural relics of Liangzhu. (a) Pottery Dou of Liangzhu Culture; (b) Jade Cong of Liangzhu Culture the technology relevant to remote rendering introduced in Chapter 6, the system display effect is shown in Fig. 8.25. Thesystemalso displays theLiangzhurelics in the digital exhibition halls so that visitors can visit the relics in the hall according to different themes that they are interested in. For example, if some visitors are interested in jade cong of Liangzhu, he/shecanvisittheshowroomwithjadecongasthetheme. Visitors can also explore freely inthehallasif they were in a real museum. The technology for a dynamic exhibition system introduced in Chapter 6 is adopted for the system. The digital museum and the virtual exhibition are shown in Fig.8.26. 208 8 Applications of Digital Preservation Technologies for Cultural Heritages

Fig. 8.25. Remote display of the Liangzhu relics

Fig. 8.26. Liangzhu digital museum 8.5 Summary

In this chapter we have introduced several typicaldigital preservation projects, which were developed by ourselves, including the digitalization ofgrottoes, such as the Dunhuang Grottoes, the digitalization of cultural heritagesites such as the Jinsha Site and the Hemudu Site, and the digitalization of mu- seum exhibitions such as Liangzhu Museum. From theabove, we can see that digital cultural heritage preservation technologies have been applied to 8.5 Summary 209 many fields and applications. And they will be further used in many aspects such as for archaeological excavation, archaeological research, conservation, exhibition in museums, heritage utilization and so on. For digitalheritage preservation, new requirements will bemadeand new directions will be pointed out, including in the technology itself (such as precision, effect and efficiency), the popularization and the feasibility of the technology (such as the cost, and whether it is easy to be used). These are actually the future directions and the impetus behind digital preservation technologies. With the further application and advancement of digital tech- nologies in the archaeological field, digital technologies will definitely be ap- plied more in preservation work and play an increasinglyimportantrolein improving archaeological working efficiency.

References

3rdTech (2009) DeltaSphere high-precision 3D laser scanner. http://www.3rdtech. com/3rdTech products.htm Barber CB, Dobkin DP, Huhdanpaa HT (1996) The Quickhull Algorithm for Convex Hulls. ACM Transactions on Mathematical Software 22(4):469–483 Camuffo D (1998) Microclimate for cultural heritage (developments in atmo- spheric science), European Commission, Environment and Climate Research Programme 23. Elsevier, Amsterdam IBM (2005) External egypt. http://www.eternalegypt.org Liu JM, Lu DM, Shi XF (2006) Interactive Sketch Generation for Dunhuang Fres- coes. Technologies for E-Learning and Digital Entertainment, Springer Verlag, Heidelberg 943–946 Shi XF, Lu DM, Liu JM (2006) Color Changing and Fading Simulation for Fres- coes Based on Empirical Knowledge from Artists. Proceedings of Pacific Rim Conference on Multimedia 2006, Springer Verlag, Heidelberg 861–869 Shi YC, Zhang J (1997) The Dunhuang Caves’ Main Diseases and Precautions Against Them. Northwestern Seismological Journal 19(2):81–87 9 Summary and Prospect

Cultural heritage relics are the sedimentary accumulation of human civi- lization over thousands of years, containing valuable historic, cultural and scientific information. How to preserve, study, and utilize them effectively is a significant scientific and technological issue. Research and technologyhave provided new means for the preservation and utilization of cultural heritage items. With the recent development of information technologies, especially com- puter graphics, computer vision, artificial intelligence, virtual reality, com- puter networks, and so on, digital preservation technologies for heritage items are also growing, including digitalization, research aiding, conservation aid- ing, digital exhibition, anddigital utilization. Meanwhile, more shortcom- ings of existing technologies and devices have been exposed during digital preservation applications, which will in turn promote the development of the technologies anddevices. Digitalization of cultural heritage refers to information acquisition from heritage items, including mainly archaeological excavation sites, museum pre- served sculptures and artifacts, large site scenes, and paintingsandmurals. The purpose is to prevent such information from being lost, and to record the current status. In respect of application, digital photography and digital acquisition technology for large-size paintings and murals can be used to han- dle the digital acquisition of planar images of heritage relics. 3D scanning, 3D information processing and texture mapping technologies can be used to acquire shape and surface information. X-ray and neutron photography can be used to obtain information about internal structure. Wireless sensor technology can be used to acquire information digitally about the status and environmental conditions of heritage items. Some of these technologies are widely used,whilesomearestill under development. With the development of nanotechnology, sensor technology, lighting technology, relative image fu- sion and color correction technology, it is expected that the problem of how to acquire information from super large-scale and super highresolutionheritage 212 9 Summary and Prospect images can be resolved. The development of laser technology, structured light 3D scanning technology and 3D model processing technology will contribute to the high-precision digital acquisition of information from large-scale sites. The development of wireless technology and sensor technology will facilitate the acquisition of wireless, miniaturized and self-organizing information from heritage sites and their environments. There are still more technologies to be developed for information digitalization in the fields of underwater archae- ology and remote sensing archaeology. Ultrasonics and radar will provide feasible solutions for problems in these fields. Research aiding technology refers to heritage site prediction and detection, computer-aided excavation, quantitative archaeological analysis, and so on, aiming to remove subjectivity and uncertainties in archaeological research, and to improve work efficiency. In terms of applications, heritage site predic- tion and harmless detection assistance technologies are capable of assisting ar- chaeologists to discover important archaeological sites more effectively. Some techniques are used to generate line drawings of the excavation field and of unearthed relics in archaeological excavation reports. Computer-aided quan- titative analysis greatly improves the accuracy of archaeological research, e.g. shape analysis, identification, and measurement. Along with the development of digital technology, other new technologies will be applied to archaeologi- cal research. 3S technology will become better integrated. In fact, there are GIS systems already available for specialized archaeological research. Com- puter Supported Cooperative Work (CSCW) can help experts gather to solve problems, such as the identification and dating of relics. CSCW will have a bright future with the improvement of bandwidth and the emergence of high resolution and multi-dimensional information acquisition devices. Some newly developed query, searching,andfeature recognition technologies will also greatly help archaeologists with the classification of cultural relics. Conservation aiding includes mainly digitally-aided cultural heritage eval- uation, situation investigation, environment monitoring, conservation plan- ning, and virtual restoration. Its major purpose is to assist the regular conser- vation process or to provide new conservation methods. Currently, digitally- aided technologies are available for all the above applications. As for the evaluation of relics, a database is the main information management tool used to assist the management of document information of cultural heritage items. Digitally-aided surveys, photographic investigations, disease investi- gations, anddynamic environment monitoring are common methodswhich can greatly improve the efficiency of situation investigations. As for con- servation planning and virtual restoration, conservation processes, e.g. color restoration, crack and exfoliation restoration, and mosaic of broken relics, can be simulated in the computer virtually, and prevent damagetoheritage items caused by testing experiments. However, there are still a lot of prob- lems remaining to be solved; for example, evaluation is still based on human judgment, and digital technologies are recognized only as a tool for informa- 9 Summary and Prospect 213 tion management and search. In fact, the automatic computer evaluation of relics is still not accepted or applied widely. Surveys of cultural heritagesites have developed from the initial artificial survey to the current frequently- usedgeodetic survey, photogrammetry, remote sensing,anddigital mapping; however, as the form of relics and surveying conditions vary greatly, genuine “non-contact” measurements are still difficult to obtain. Compared to the traditional methods, dynamic environment monitoring can output environ- mental parameters in real-time. However, the immaturity of sensor networks had led to difficulties in monitoring the environment in poor conditions, e.g. bad weather. Current technologies work well for small broken areas of relics. But for large-scale color changes and exfoliation, current algorithms are still not competent because of the limitations of computer intelligence. Current technologies work well for the matching of 2D broken relics, but for 3D, are veryhard to put into practical use. Digital exhibition of cultural heritage relics includes mainly online ex- hibitions, reconstructed site exhibitions, and digital interactive exhibitions in exhibition halls. It aims to improve the flexibility of exhibitions, break the constraints of time and space of exhibition, provide visitors with cus- tomization and communication options, and finally change the exhibition from being object-centered to human-centered. In digital exhibitions, besides the contents, participators are also an important part of the exhibitions, playing certain roles. Digital exhibition technology can not only break the constraints of time and space, making it possible to browse and appreciate the heritageinformation at anytime and anywhere but can also enable vis- itors to browse damaged relics after reconstruction. Such technologies can exhibit multi-media information from cultural heritage items in vivid forms, such as 3D models, video, and audio. Digital interaction technology can of- fer visitors the opportunity to participate in exhibitions or to customize or organize their own exhibitions. Digital exhibition of cultural heritage relics, as a new form of exhibition, has already been widely accepted by people, and is expected to be developed further with the growthindemand.New developments in the following areas would be desirable. First of all, how to exhibit the invisible historical, scientific, and artistic values of cultural her- itage items, display them more vividly, and achieve the aim of education and entertainment is a problem deserving study. Some fully immersive remote displaying applications would be worthwhile. With the growing networking bandwidth, some digital theaters in museums, such as full-domes and curved screens, can display digital programs from other museums to enhance inter- museum communication of contents. There is still a long way to go before these projects and systems are accepted and used widely. Some portable de- vices for exhibitions should also be developed to increase convenience and comfort. Digital utilization aims to synthetically utilize the historical, cultural, and scientific value of cultural heritage items by applying the theories and results 214 9 Summary and Prospect of modern science and technology, creating new value in education, scientific technology, and cultural fields. Virtual tourism, digital publishing,souvenir manufacturing, and teaching through entertainment are some examples of heritage utilization. However, for more detailed information, such as the pro- duction of animation and shape and color information with higher complex- ity (for example, from painted sculptures), the key technological problems of development and utilization still remain unsolved. But digital utilization of heritage information is just taking its first steps. Multidisciplinary research including archaeology, history, and computer science needs to be further in- tegrated to greatly improve utilization. To summarize, digital preservation technologies of cultural heritage items have been applied widely, but for only a limited variety of objects and in- formation. A great deal of work is still to be done to develop technologies further, in order to promote the wide use of digital information in the preser- vation and utilization of heritage relics. Along with the perfection of digital technologyandthegrowth of social demand, the digital preservation of cul- tural heritage items, a new subject integrating multiple disciplines, will draw increasing attention from researchers in the field of computer science and heritage preservation. Index

Symbols archaeological, 4–6, 14, 22, 32, 35–37, 2D animation, 26 39, 41, 44, 71–76, 78, 82, 83, 86, 3D, 12–14, 18, 28, 37, 44, 47, 49, 51–53, 87, 135–137, 156, 196, 209, 211, 56, 61, 63, 73, 76, 77, 79, 99, 117, 212 124, 129, 130, 132, 134, 136, 143, archaeological excavation, 5, 36, 37, 39, 162, 163, 187, 199, 200, 206, 212 41, 44, 73, 87, 196, 209, 211 3D Scan, 117, 200 archaeological research, 4, 6, 22, 32, 44, 3D acquisition, 79 71–73, 78, 82, 86, 87, 137, 196, 3D animation, 26 209, 212 3D model, 14, 18, 28, 37, 44, 47, 52, 53, archaeological research aiding, 71–73 56, 63, 73, 76, 77, 99, 124, 129, archaeological research report, 72, 73 130, 132, 134, 136, 143, 187, 199, archaeological sites, 14, 35, 36, 71–76, 206, 212 83, 135, 136, 156 3D reconstruction, 12, 37, 49, 51, 52, archive management, 4, 6, 32 61, 162, 163 artistic style, 159, 162, 163, 167, 171 3D scanning data processing, 13 artistic style learningbased re-creation, 3S (GIS, RS, and GPS),9 171 automatic mosaic, 64 A B acquisition, 39–41, 64 B´´ezier curve, 174, 175 acquisition of archaeological exploration browser/server, 106, 125 information, 39 acquisition of cultural layer shape C information, 40, 41 calibration, 46, 51, 55, 65, 151 acquisition of relic detail information, camera calibration, 46, 55, 65 40 Canny edge detection, 79 acquisition of relic location information, character animation, 139 39 client/server architecture, 16 acquisition planning, 64 clockwise shot, 63 animation, 5, 25–27, 37, 122, 135–137, close-range photogrammetry, 73, 76, 139, 141, 142, 156, 162, 164, 165, 90, 95, 96, 98 214 color, 63, 64, 96, 108–110, 118, 160, animation creation, 162 167, 169, 170, 172, 192, 211, 212 216 Index color correction, 63, 64, 211 93, 95, 96, 122, 124–129, 132, color fading, 109 135–139, 153, 156, 163, 166, 179, color feature, 96, 160, 167, 169, 170, 183, 184, 186, 187, 196, 200, 204, 172 205, 207–209, 211, 213, 214 color layer, 170 digital camera, 10, 63, 187 color restoration, 108, 110, 118, 212 digital exhibition, 25, 27, 31, 122, 124, color Simulation, 192 127, 132, 135, 139, 156, 183, 184, computer aided, 71, 162, 174, 181 186, 196, 200, 204, 205, 207, 211, computer aided imitation of murals, 213 162, 174, 181 digital guide in exhibition hall, 153 computer aidedline drawinggeneration, digital mapping, 92, 95, 96, 213 71 digital matting, 163, 166 computer-aided color filling, 174 digital measurement, 71, 72, 76, 83, 92 connectivity-based segmentation, 20 digital photography, 9–11, 35, 37, 40, conservation, 4–6, 9, 17, 32, 35, 46, 70, 45, 70, 93, 186, 211 79, 89–93, 96, 99, 100, 103, 106, digital preservation, 5, 6, 32, 179, 184, 111, 115–119, 183, 184, 186, 188, 208, 209, 211, 214 209, 211, 212 content-based retrieval, 125 digital reconstruction exhibition, 122, cultural, 1, 3–6, 9–11, 17, 18, 21, 24, 135–138 25, 29, 31, 32, 35, 36, 39, 43, 86, digital theme modeling, 125–129 89, 90, 92, 93, 103–105, 111, 115, digitalization, 9, 35, 64, 68, 70, 93–95, 139, 156, 162, 163, 188, 196, 208, 115, 117, 119, 123, 183, 187, 196, 211–214 197, 200, 206, 208, 211, 212 culturalheritage, 1, 3–6, 9–11, 17, 18, digitally aided, 89–91, 93–95, 107–109, 21, 24, 25, 29, 31, 32, 35, 36, 39, 115, 118, 119 43, 86, 89, 90, 92, 93, 103–105, digitallyaided conservation and 111, 115, 139, 156, 162, 163, 188, restoration, 89, 118 196, 208, 211–214 digitally aided environmental investiga- culturalheritage preservation, 3, 4, 9, tion, 94 21, 32, 208 digitallyaided investigation, 89–91, 93, cultural heritage relics, 6, 11, 163, 211, 95 213 digitally aided mapping, 119 cultural layer, 36–43, 72, 141, 200 digitallyaided restoration, 107–109, cultural relics, 1, 2, 4, 9, 11, 12, 35, 115, 118 91–96, 99, 100, 103, 106, 114, digitally aided virtual restoration of 118, 119, 124, 127, 128, 148, 156, surface color, 109 198–200, 205, 206, 212 disease, 89, 92, 93, 96, 99, 119 customized arrangement, 123–125, 127 disease detection, 19 D disease detection and marking, 93, 96 date aiding, 71 disease investigation, 89, 92, 93, 99, 119 datingbased on typology, 79 double-buffer mechanism, 29 defect repairing, 174 drawing generation, 77 defect repairingy, 192 dynamic environmental, 92, 99–102, deformation template, 114 106, 118, 119 degree of freedom (DoF),31 dynamic environmental monitoring, 92, digital, 5, 6, 9–11, 25, 27, 31, 32, 35, 99–102, 106, 118, 119 37, 40, 45, 63, 70–72, 76, 83, 92, dynamic environmental sensing,102 Index 217

E heritage, 123, 159 edge detection, 20, 77, 79, 174–176, 192 heritage digitalization, 123 edge detection and fitting,174 heritage values, 159 edge-based segmentation, 20 high fidelity, 45 eigen-decomposition, 79, 80 eigenvector, 22, 59, 79, 80 I element extraction, 159 image, 18–21, 32, 49, 66, 79, 83, 93, 94, environmental, 89, 91, 93, 100, 103, 105 108, 111, 113, 162, 163, 166, 170, environmental data analysis and 172, 174, 183, 186, 211 display, 105 image enhancement, 18, 19 environmental data monitoring, 89 image feature extraction, 49 environmentalinformation transmis- image fusion, 66, 211 sion, 103 imagematting,166 environmental investigation, 91, 93, 100 image processing, 18, 20, 32, 79, 83, 93, excavation unit, 38, 44, 45, 72, 76, 116, 94, 111, 162, 170, 172, 183, 186 196, 198–200 image segmentation, 19–21, 108, 113, exhibitions, 4–6, 29, 31, 32, 44, 121–123, 163, 174 125, 129, 135, 139, 141, 144, 156, implicit value, 161–163 205, 208, 213 inpainting, 108, 109 exhibits data processing, 123 intelligent information processing, 22 expert system, 23, 81, 82, 84 interaction, 25, 27, 31, 32, 44, 122, 131, expression of related information, 40 133, 147, 166, 167, 175, 176, 213 extraction of source materials, 166 interactive, 111, 144, 146, 156 interactive experience in theexhibition F hall, 144, 146, 156 feature points, 41, 42, 47, 117 interactive narration, 146 forward kinematics, 27 interactive repair of cracks, 111 frequency-based image enhancement, inverse kinematics, 27 19 investigation reports generation, 93 fusion of texture image, 54 iterative closest point (ICP) transform, fuzzy reasoning, 24 110

G L Gaussian mixture model, 97 large scene information acquisition, 55, geometry rectification, 63, 64 56 gesture interaction, 32 laser scan, 41, 42, 61, 79, 134 GIS (geographic information systems), least square method, 58 15 least squares method, 58 global, 54, 64 line laser, 13, 41 global color balance, 54 linear transform, 110 global optimization, 64 local color fusion, 54 GPS (global positioning systems),14 LoD (level of detail), 28 gradient analysis, 74 long range communication, 18 gray code, 49, 50 gray scale, 18–20, 49–51, 66 M ground penetrating radar, 37, 41, 87 MAC (media access control), 17, 103–105 H mapping Investigation, 89, 91, 92 head mounted display, 30, 143 mesh, 13, 52, 59, 61 218 Index mesh model, 13, 59, 61 photogrammetry, 37, 39–42, 65, 76, 90, mesh simplification, 52 92, 95, 213 metadata expression, 101, 123, 126, 127 photographic investigation, 89, 91–93 microclimate, 107, 188 point cloud, 13, 52, 57, 58, 200, 206 microclimate environmental monitor- polygon, 13, 14, 28, 41, 53, 59, 61, 129, ing, 107 130, 201 microclimate monitoring, 188 prediction and detection of archaeologi- modeling ofChinese ancient buildings, cal sites, 71, 86 172 principal component analysis, 22 mosaic of relic fragments, 114 procedural animation, 27 multi-computer collaborative rendering, progressive data transmission, 125 152 multi-level precision heritage data R generation, 129 real-time rendering,27 multi-projector calibration, 147–149 region-based segmentation, 20 multi-projector display system, 30 registration, 41, 46, 47, 55, 57, 61, 137 multi-texture fusion, 46, 55 registration and merging, 41, 46, 47, museum, 2, 6, 44–46, 48, 55, 56, 69, 92, 55, 57, 61 93, 109, 115–117, 122, 127, 128, registration of virtual and real scene, 134, 143–146, 148, 153, 154, 156, 137 183, 196, 198, 202, 205–208, 211, relic identification, 81, 82 213 remote rendering, 125, 126, 130–132, museum preserved sculptures and 134, 206, 207 artifacts, 44–46, 48, 69, 211 rendering, 25–29, 31, 61, 123–126, 130–132, 134, 137, 147, 152, 153, N 155, 206, 207 rendering and display, 31, 123–125 natural interaction, 25, 31, 122 restoration effect exhibition and neural networks, 114 evaluation, 107 noise, 11, 13, 20, 67, 113, 175 route planning, 146 non-contact measurements, 12 RS (remote sensing),14 non-photorealistic rendering,26 normal vector orientation, 13 S sample mean match, 110 O sample-base line style displacement, online heritageexhibition, 122–125, 174,192 128–130, 132, 156 scene animation, 136, 137 outline feature, 169 scope prediction modeling, 74 self-occlusion, 53 P sensor, 12, 13, 17, 48, 51, 94, 101–104, pattern, 5, 24, 162, 167, 169–172, 177, 107, 119, 139, 188–190, 211–213 181, 192 shape restoration, 108, 109 pattern creation, 5, 24, 162, 171, 177, single Gaussian model, 96 181, 192 site virtual reconstruction exhibition, pattern element, 169, 171 137 pattern layout, 170, 171 skeleton animation, 139, 141 pattern style, 167 soil perturbation theory, 75 pattern template, 167, 172 spot laser, 13 phase shifting, 49, 50 stereo display, 29–31 Index 219 strokemodel, 174, 176 the Liangzhu Site, 184 structured light, 13, 48, 51, 151, 199, theLiangzhu site, 55, 132, 204 212 threshold-based segmentation, 20 structuredlight encoding, 48, 51 TOF (time-of-flight), 12, 41 structured light scan, 13 total station, 4, 39, 42, 43, 95, 96 style extraction, 164, 194 tracking, 31, 32, 111, 116, 138, 148, 203 sub-pixeldecoding,48 triangulation, 12, 13, 41, 42, 51, 52, 154 surface, 42, 45, 47, 52, 72, 75, 83, 108, triangulation measurement, 13, 41, 42, 205 51 surface cost analysis, 72, 75, 83 surface dirt removal, 108 U surface texture, 42, 45, 47, 52, 205 utilization, 4–6, 9, 24, 32, 35, 39, synthesis reasoning, 24, 25 159–162, 166, 181, 183, 186, 209, 211, 213, 214 T texton, 21 V texture, 9, 14, 19–22, 26–29, 35, 37, 40, viewpoint tracking,137 45–47, 52–55, 61, 63, 69, 90, 93, virtual restoration, 44, 45, 55, 90, 96, 98, 107–109, 111, 116, 130, 107–110, 114–118, 192, 212 160, 163, 166, 172, 174, 187, 199, visual area analysis, 72, 75 205, 211 texture acquisition, 46, 55 W texture coordinates, 28, 52, 53 weighted fuzzy logic, 82 texture fusion, 14 white-point transform, 111 texture mapping, 14, 29, 52, 55, 61, Winter Palace, 135, 143 116, 199, 211 wireless, 6, 9, 16, 17, 101, 103, 104, 107, texture recognition, 21 119, 143, 147, 153, 154, 183, 188, texture restoration, 108, 109 189, 211, 212 The Dunhuang Mogao Grottoes, 92, wireless communication, 17, 103, 104, 160, 164, 184, 189 153 the Hemudu Site, 55, 132, 141, 184, WSN (wireless sensor network), 17, 18, 201–203, 208 103, 188, 189 the Jinsha Site, 44, 132, 145, 154, 184, WYSWYG (What You See is What 196, 198–200, 208 You Get),42