https://doi.org/10.2352/ISSN.2470-1173.2021.14.CVAA-042 © 2021, Society for Imaging Science and Technology Recovery of underdrawings and ghost-paintings via style transfer by deep convolutional neural networks: A digital tool for art scholars Anthony Bourached,a George H. Cann,b Ryan-Rhys Griffiths,c and David G. Storkd aComputer Science Department, University College London, London, UK bDepartment of Space and Climate Physics, University College London, London, UK cDepartment of Physics, University of Cambridge, Cambridge, UK dPortola Valley, CA 94028 USA ABSTRACT interpretation of such underdrawings is central in ad- dressing numerous problems in the history and interpre- We describe the application of convolutional neural net- tation of art, including inferring artists' praxis as well work style transfer to the problem of improved visual- as interpreting, attributing, and authenticating such ization of underdrawings and ghost-paintings in fine art works.1, 2 For example, copies of some paintings have oil paintings. Such underdrawings and hidden paintings been exposed as forgeries through an analysis of un- are typically revealed by x-ray or infrared techniques derdrawings revealed through such x-ray imaging; after which yield images that are grayscale, and thus devoid all, a forger generally does not have access to the un- of color and full style information. Past methods for in- derdrawings and so can duplicate only what is visible ferring color in underdrawings have been based on phys- to the naked eye.3 ical x-ray fluorescence spectral imaging of pigments in ghost-paintings and are thus expensive, time consum- X-ray and infrared imaging do not directly reveal col- 4{6 ing, and require equipment not available in most conser- ors of underdrawings. In most studies, this lack of vation studios. Our algorithmic methods do not need color information does not pose a major impediment such expensive physical imaging devices. Our proof-of- because (it is widely acknowledged) the artist generally concept system, applied to works by Pablo Picasso and uses the same colors throughout the development of Leonardo, reveal colors and designs that respect the the work; the colors in the primary, visible artwork are natural segmentation in the ghost-painting. We believe likely quite similar to the ones the artist used through the computed images provide insight into the artist and its development. In such cases it is simply the geometric associated oeuvre not available by other means. Our design that is relevant. For instance, in a computational results strongly suggest that future applications based study of the compositional style of Piet Mondrian, x-ray on larger corpora of paintings for training will display images of his Neoplastic geometric paintings revealed color schemes and designs that even more closely resem- the designs of prior designs, which the artist ultimately ble works of the artist. For these reasons refinements to rejected. Such \near miss" designs, along with the fi- our methods should find wide use in art conservation, nal accepted designs, could be used to train statisti- 7 connoisseurship, and art analysis. cal models of Mondrian's compositional principles. In that study, x-ray images sufficed because color was of Keywords: ghost-paintings, style transfer, deep neu- no concern. ral network, computational art analysis, artificial intel- ligence, computer-assisted connoisseurship There are, however, some paintings in which the de- sign of the underdrawings are not directly related to that of the visible painting, and these pose a rather 1. INTRODUCTION AND different challenge in art analysis. These are under- BACKGROUND drawings in which the entire compositions and designs were painted over by a second unrelated design, which is Many paintings in the Western canon, particularly real- the visible artwork. Such hidden artworks or so-called ist easel paintings from the Renaissance to the present, \ghost-paintings" arise when artists re-used canvases, bear underdrawings and pentimenti (from Italian, \to either because they were unsatisfied with their first repent")|preliminary versions of the work created as paintings, or more commonly when they could not af- the artist altered and developed the final design. The ford new canvases. In some cases the orientation of the hidden design is rotated 90◦ when the artist preferred Send correspondence to [email protected]. landscape format to portrait format|or vice versa| 1 IS&T International Symposium on Electronic Imaging 2021 Computer Vision and Image Analysis of Art 2021 042-1 for the second painting, as in Picasso's Crouching beg- most conservation studios. Moreover, ultraviolet radia- gar. In some cases the rotation is 180◦ with respect tion has shallow penetration power, and may not reveal to the visible painting, as for instance Rembrandt's An underdrawings for purely physical reasons. old man in military costume. Prominent paintings that This, then, is the overriding goal of our work: to com- have of such ghost-paintings include: pute a digital image of the ghost-painting as close to the original|in color, form, and style|so the general pub- • Kazimir Malevich's Black square (1915) lic and art scholars can better study a hidden artwork, all without the need for such expensive physical sensing Rembrandt's An old man in military costume • equipment. (1630{31) In brief, our approach is to transfer the style| • Vincent van Gogh's Patch of grass (1887) specifically the color|from comparable artworks to the grayscale underdrawing. Related color style transfer Pablo Picasso's The blue room (1901), Mother • in fine art scholarship has been applied to the reju- and child by the sea (1902), The crouching beg- venation of fine art tapestries, whose vegetable pig- gar (1902), Barcelona rooftops (1903), Old guitarist ments are fugitive, and thus faded over centuries. In (1903{04), and Woman ironing (1904) some cases the cartoons or source paintings survive, • Ren´eMagritte's The portrait (1935), and The hu- where the oil pigments retain the reference colors. How- man condition (1935) ever, tapestry ateliers frequently alter the designs of the works|adding or deleting figures, changing their poses, • Edgar Degas' Portrait of a woman (c. 1876{80) altering backgrounds, and so forth|and thus a simple overlay of the cartoon or painting design atop the image Francisco Goya's Portrait of Do~naIsabel de Porcel • of a faded tapestry will not lead to a coherent image. (1805) Color transfer with modest tolerance for spatial dispar- • Leonardo's The Virgin of the rocks (1495{1508)∗ ities can produce acceptable \rejuvenated" tapestries, but a full solution will likely require methods based on 16 Indeed, as many as 20 out of 130 paintings paint- deep neural networks. ings by van Gogh examined at the van Gogh Museum, We begin in Sect. 2 with a brief overview of the prob- Amsterdam, have at least partially completed ghost- lem of separating the design of the underdrawing from paintings.8 the x-ray or infrared reflectogram containing the mix- There is of course scholarly and general interest in tures of the underdrawing and the primary visible im- \lost" works, as they give a richer understanding of age. We then turn in Sect. 3 to the problem of map- the artist and his or her oeuvre.9 Art scholars inter- ping of style and colors (learned from representative artworks) to the ghost-painting, an extension and im- pret such ghost-paintings for a number of reasons, such 17 as to better understand an artist's career development provement upon earlier efforts, and enhancing the de- and choice of subjects (both general and specific), and sign of a underdrawing by style transfer from an ensem- to learn what an artist did or did not wish to preserve ble of drawings. We focus on this second stage, where 10, 11 we use deep neural network methods for style trans- in his oeuvre. It is clear that such tasks are best 18, 19 served by a high-quality digital image of the underdraw- fer. We present in Sect 4 our image results for two ing, including features (such as color) not captured by reconstructed ghost-paintings from a single style image x-radiography and infrared reflectography. To date, the and one drawing and from an ensemble of artworks. We principal non-destructive method for recovering colors also discuss how such methods can empower art schol- in ghost-paintings is through x-ray fluorescence spec- ars to better interpret these hidden works. We conclude troscopy, which reveals the elemental composition of in Sect. 5 with a summary of our results, current limita- pigments. Such measured elemental compositions are tions and caveats, and future directions for developing then matched to pigment databases so as to infer the this digital tool for the community of art scholars. likely pigments and their proportions, which in turn indicates the colors in the ghost-painting.12{15 This 2. IMAGE SEPARATION method requires expensive equipment not available in An x-ray of a painting containing a ghost-painting re- ∗There are two versions of this work: one in the Lou- veals the visible and the ghost compositions overlapped. vre and the other in the National Gallery London, which is Thus the first processing stage is to isolate the hidden shown in Fig. 3, below, and is the focus of our efforts. image from the primary, visible image. This can be 2 IS&T International Symposium on Electronic Imaging 2021 042-2 Computer Vision and Image Analysis of Art 2021 a very challenging computational task, even in simple to qualitatively excellent results.23{25 The metrics for cases.20 The leading algorithmic approach is signal sep- such success of these algorithms are generally qualita- aration or blind source separation, which was first devel- tive, however: the modified image should \appear as in oped for separating the sounds of separate sources from the style of" the style images.