1G3-ES-5-04

The 34th Annual Conference of the Japanese Society for Artificial Intelligence, 2020

The Morandi Room Entering the World of Morandi’s through Machine Learning

Shigeru Kobayashi∗1 Ryota Kuwakubo∗1 Shigeru Matsui∗1 Yoshiyuki Otani∗1 Xinqi Zhang∗1 Daisuke Niizumi∗2 ∗1Institute of Advanced Media Arts and Sciences ∗2daisukelab

In this study, we propose a new way to appreciate artwork based on the growing interest in the active appreciation of artwork and the development of machine learning technology. We chose Italian painter Giorgio Morandi, who was active in the first half of the 20 century and known for his unique composition and coloring, as the theme, and developed a hands-on exhibit in which spectators freely arrange and compose objects that reproduce the painter’s motifs, and generates images that reproduce the painter’s coloring by machine learning. From the results of a questionnaire survey of 29 people, we confirmed that experiencing this exhibit deepened their interest in the painter.

1. Introduction and Related Work age groups. For example, Visual Thinking Strategies, pro- posed in the 1980s by cognitive psychologist Abigail Housen In 1950, Alan M. Turing proposed the question, “Can and Philip Yenawine, the former Director of Education at machines think?” in his paper “Computing Machinery and the Museum of in New York, emphasize the Intelligence.”[Turing 50] As can be seen from the fact that ability of spectators to share their impressions of a work of the first of a series of questions later known as the “Turing art, and are believed to foster the ability to observe, think Test” was “Please write me a sonnet on the subject of the critically, and communicate [Yenawine 13]. Therefore, the Forth Bridge,” art has long been a hot topic when thinking method is now used in many museums. Just as the pur- about machine intelligence, and many people are working to pose of the experience of appreciation in art education has apply machine learning to the visual arts. For example, in ∗1 been expanded by unilaterally conveying the author’s back- the Next project , conducted in 2016 by ING, ground and historical considerations of art, the methods for Microsoft, TU Delft, and the Mauritshuis museum, the sur- appreciation have been expanded one after another. For ex- viving works of the 17th-century painter Rembrandt were ample, in addition to conventional methods such as display- analyzed in detail by experts and machine learning, and a ing an original work at a museum or projecting an image “new work” with motifs, compositions, and touches typical ∗2 of the work with a projector, new methods using various of Rembrandt was created. Furthermore, DeepArt ,run technologies have been proposed, such as projecting images by researchers in Germany, Switzerland, and Belgium, has onto a large wall∗3, displaying in a VR space [Lu 08], and offered a web service since 2016 that leverages the algorithm displaying with information using AR technology [Serio 13]. created by Leon Gatys [Gatys 15] to enable users to easily As we have seen, while research utilizing machine learning create a new “artwork” by simply uploading a photo and has focused mainly on the production of artworks, the im- choosing a style. Moreover, artist and programmer Gene portance of experiencing artworks has been noticed, and Kogan has been working on a project “Abraham” since 2019 various methods have been proposed. Based on this back- to build an agent that autonomously generates unique and ground, this study attempts to create a new way to experi- original art [Kogan 19]. These are but a few of the many ence the appreciation of artwork utilizing machine learning efforts made by artists and researchers that focus on using We chose Giorgio Morandi, an Italian painter who was ac- machine learning to produce art. All of these are efforts tive in the early 20th century, as the subject of this study. using machine learning to focus on production in artistic Morandi, known as one of the foremost painters of the 20th activities. However, an essential activity in art is not only century, produced more than 1,300 oil paintings over about its production, but also its appreciation. 50 years. Most of these are still lifes [Morandi 19]. His work In addition to being positioned as an essential element is characterized by its unique composition where objects are along with production in school education, the appreciation gathered together rather than separated, its coloring, and of artwork is also considered to help improve a student’s its distortion of contours (see fig. 1). It is not only appre- ability to think and communicate among a wide range of ciated by many artists but is often the subject of advanced research efforts. Contact: Shigeru Kobayashi, Institute of Advanced Me- dia Arts and Sciences, 4-1-7 Kagano, Ogaki-shi, ∗3 For example, an event called “Immersive Museum” is sched- Gifu, 503-0006 Japan, 0584-75-6641, 0584-75-6637, uled to be held in Tokyo from April 2020, in which images [email protected] of works by impressionist artists such as Degas and Renoir are ∗1 https://www.nextrembrandt.com/ projected over the entire field of vision so that people can enter ∗2 https://deepart.io/ the world of . https://immersive-museum.jp/

1 1G3-ES-5-04

The 34th Annual Conference of the Japanese Society for Artificial Intelligence, 2020

of the exhibition space, we placed books related to Morandi that visitors could freely view.

Display Camera

PC (behind the wall)

Fig. 1: Giorgio Morandi, (1946) For example, House of ZKA, a collection of architectural

investigations, held a workshop “Morandi-esque” in 2018 to Objects

create 3D models by tracing distortions in the contours of Books objects by Morandi as a collection of arcs and printed with a 3D printer to exhibit∗4. In addition, Guido Salimbeni Fig. 2: System et al. presented a system that supports the placement of objects in 3D space by suggesting arrangements modeled on Morandi’s work [Salimbeni 19]. Moreover, in order to gain a deeper understanding of Morandi, many workshops have been held in conjunction with exhibitions, in which spectators can deepen their understanding of Morandi by simply reproducing his works. For example, at a workshop “dissect Morandi’s technique” held in April 2016 by painter Yoichi Miyajima and others as an event related to an exhi- bition held in Tokyo in 2016, they reproduced objects and tables appearing in Morandi’s works in full size and verified viewpoints and lighting using a digital camera and light- ∗5 ing equipment . Inspired by these workshops, our study Fig. 3: A visitor experiencing the exhibit presents a hands-on exhibit utilizing machine learning to motivate spectators to understand the artist and the world 2.2 Algorithm of his work more deeply. Among the characteristics of Morandi’s works, we left the composition to the spectator and handled the reproduc- 2. Implementation tion of coloring by machine learning models learned from Morandi’s works. We mainly evaluated two algorithms, 2.1 Concept and overview of the exhibit pix2pix by Phillip Isola et al. [Isola 17], and CycleGAN “The Morandi Room” is a hands-on exhibit that pro- by Jun-Yan Zhu and Taesung Park et al. [Zhu 17]. We also poses a new way of appreciating artwork in which spectators examined the colorization proposed by Aditya Deshpande try to reconstruct a part of the world of the artist’s work et al. [Deshpande 15] and the style conversion proposed by themselves by arranging objects imitating the main mo- Leon A. Gatys et al. [Gatys 16] However, in the latter two tifs utilized in Morandi’s work, and appreciating the world cases, the context of the captured images remained strong, through “eyes” using machine learning models derived from and the parts learned from Morandi’s images became weak, Morandi’s surviving paintings. Fig. 2 shows the overall so we focused on the first two for building our exhibit. system, and fig. 3 shows a visitor experiencing the exhibit. pix2pix is an algorithm that allows conversion in both The entire exhibition space is W 3,960 mm ʷ D 2,655 mm directions from A to B or B to A during inference by pro- ʷ H 2,250 mm, and a table W 910 mm ʷ D 845 mm ʷ H viding pairs of corresponding images from domain A and B 810 mm is placed in it. The image of objects (up to nine during learning. In this exhibit, we searched for intermedi- objects of four types) on the table is captured by a digital ate representation that can handle different inputs in com- camera at a distance of about 2,000 mm and a height of mon, such as images of artworks actually drawn by Morandi about 1,700 mm and is output by HDMI with 1,280 ʷ 720 during learning, and images of objects that reproduces the pixels at 60 fps. The output is captured in an HMDI cap- motif used by Morandi during inference. We compared the ture box connected to a PC with an NVIDIA GeForce GTX Canny edge detection only, a color-based segmentation us- 1080, the center 512 pixel square of the frame is cropped, ing k-means clustering only, and edge detection and segmen- and given to a machine learning model for inference to gen- tation, and chose the combination of both because obtained erate an image. The resulting image is then upscaled using the best results (see fig. 4). bilinear interpolation to 1,024 pixel squares and output to CycleGAN is an algorithm that provides domain A and a 300 mm square monitor embedded in the frame. In front domain B images without pairing during learning and al- lows conversion in both directions from A to B or B to A ∗ 4 http://www.houseofzka.com/morandi-esque during inference. During learning, the images taken in the ∗5 http://fuminaosuenaga.com/workshop/morandi/

2 1G3-ES-5-04

The 34th Annual Conference of the Japanese Society for Artificial Intelligence, 2020

exhibition environment are given as domain A, and the im- of Morandi’s original work. Others tried unusual arrange- ages of the paintings drawn by Morandi are given as domain ments, such as by turning objects upside down. Addition- B. During inference, the camera images are given as input, ally, several tried bringing in objects unrelated to Morandi’s and the images are generated end-to-end (see fig. 5). work, such as a smartphone, to see how the result would

G(x) G(x) change. Original painting or photo G

D D

Fake Real

Canny edge detection & Color-based segmentation

(b) Train a pix2pix model to map intermediate images → paintings

G

Intermediate representation edge & seg. (a) Preprocess to create an intermediate image from an original picture (c) Runtime pipeline: a photo → an intermediate image → a painting Fig. 4: pix2pix Fig. 6: Example of arrangements by visitors

3.2 Evaluation by questionnaire We created an online form where visitors could post their impressions anonymously after their experience, providing the link with a QR code. Approximately one-fifth of the (a) Training: photos paintings (b) Runtime: photo → painting about 600 visitors experienced the exhibit, and 29 people Fig. 5: CycleGAN responded. First, we asked if visitor’s had prior knowledge Subjective evaluation of the outputs of the two algorithms of Morandi. 3 respondents (10.3%) said “I am very familiar for the test data showed that pix2pix had a less overall fail- with him,” 5 (17.2%) responded “I have seen his work,” ure rate and provided more stable results, whereas Cycle- 3 (10.3%) selected “I have heard of him,” and 18 (62.1%) GAN had a more significant difference in quality depending responded “I did not know anything about him.” on placement. Therefore, we chose pix2pix as the default Second, we asked the 8 respondents who were most fa- algorithm and switched to CycleGAN in response to ques- miliar with Morandi “Did you feel the images displayed in tions from spectators. the frame as Morandi-like?” In response to this question, four respondents (50%) answered affirmatively, citing the 2.3 Dataset similarity of objects, coloring, and shades to the original. For the images of Morandi’s paintings, we collected what Third, we asked about changing interest in Morandi. 19 appeared to be still-life paintings by Morandi on the Inter- people (65.5%) responded positively. Those who said they net, and prepared a data set consisting of 179 images that did not know Morandi said, “When I saw his paintings after were confirmed using Morandi’s catalog. As for the images the experience, I felt a sense of closeness to him.” “Until to be used as training data for CycleGAN, we placed ob- now, I was not very interested in still-life paintings, but I jects on the table where environmental conditions such as could find the pleasure of imagining the painter’s intentions, lighting were determined at the exhibition site and took 51 such as composition and light settings.” and “I wanted to variations. know the intention of his work by positioning the object 2.4 3D Models myself.”. Those who said they were very familiar with him We modeled the objects to be placed on the table in full said, “It led me to wonder what the elements that I thought scale based on the photo album “Morandi’s Objects” by Morandi were.” The three persons who answered negatively photographer Joel Meyerowitz, featuring the object used (10.3%) replied: “I’m not interested in painting.” “There as a motif by Morandi [Meyerowitz 15], and dimensional was no explanation on Morandi’s art history and status as a data obtained from the Morandi Museum. We created 3D painter, so I couldn’t understand it.” “The exhibit itself was models of objects with complex shapes like pots or bottles, interesting, but my interest in the painter did not increase printed them out using a 3D printer, and painted them with because it was not my favorite style.”. acrylic paint. For objects with simple shapes, such as cubes, Free responses indicated that one person voluntarily re- we shaped them out of cardboard and colored them with searched Morandi immediately after viewing the exhibit. acrylic paint. Besides this, respondents expressed interest and concern about copyrights, etc., such as “It will lead to fake pic- 3. Results tures.” “Morandi would not like it. In the future, visual artists should think about rights like AI learning rights.” 3.1 Example of arrangements by visitors and “Can the images produced by this work be subject to We presented this exhibit at a four-day exhibition. The copyright?”. visitors during the four days generated 789 unique pat- terns as shown in fig. 6. Some seemed to be conscious

3 1G3-ES-5-04

The 34th Annual Conference of the Japanese Society for Artificial Intelligence, 2020

4. Discussion pose images with the aid of machine learning that reproduce the coloring of his paintings. From the results of a question- Based on these results, we discuss three major points: naire survey of 29 people, we confirmed that experiencing how effective the exhibit was as a new way to appreci- this exhibit deepened their interest in the painter. ate artwork, the reproduction of features characteristic of Morandi’s paintings, and concerns about copyright. References First, regarding the effectiveness of our exhibit as a new way of appreciating Morandi’s paintings, over 65% of those [Deshpande 15] Deshpande, A., Rock, J., and Forsyth, D.: who responded to our questionnaire reported increased in- Learning large-scale automatic image colorization, in terest in Morandi’s work, with one of them voluntarily re- Proceedings of the IEEE International Conference on searching Morandi after the experience. Moreover, many Computer Vision, pp. 567–575 (2015) spectators tried out their own arrangements, 789 variations [Gatys 15] Gatys, L. A., Ecker, A. S., and Bethge, M.: were made during the exhibition. From these facts, we con- A Neural Algorithm of Artistic Style, CoRR, Vol. sider that this exhibit was effective as an opportunity for abs/1508.06576, (2015) spectators to become interested in Morandi. Next, as to whether or not our exhibit was able to re- [Gatys 16] Gatys, L. A., Ecker, A. S., and Bethge, M.: Im- produce the characteristics of Morandi’s painting such as age Style Transfer Using Convolutional Neural Networks, composition, coloring, and contour distortion, half of the Proceedings of the IEEE Computer Society Conference respondents who had previously seen Morandi’s works said on Computer Vision and Pattern Recognition, Vol. 2016- that they felt Morandi’s uniqueness, suggesting that a cer- Decem, pp. 2414–2423 (2016) tain level of reproducibility was achieved. Regarding com- [Isola 17] Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A.: position, it may be useful to provide support to explore Image-to-Image Translation with Conditional Adversar- Morandi’s unique composition by presenting images of orig- ial Networks, in Proceedings of the IEEE Conference on inal paintings with similar arrangements. Additionally, al- Computer Vision and Pattern Recognition, pp. 1125– lowing the height of the camera and the angle of lights to 1134 (2017) be easily changed would help the spectators create a more Morandi-like composition. Regarding coloring, our imple- [Kogan 19] Kogan, G.: Artist in the Cloud: Towards an mentation uses simple segmentation. Utilizing segmenta- Autonomous Artist (2019) tion masks created with U-Net, Mask R-CNN, etc. for con- [Lu 08] Lu, L.-F. L.: A Art Caf´e: A 3D Virtual Learning version to an intermediate representation would improve Environment for Art Education, Art Education, Vol. 61, the color quality. Regarding the distortion of the contour, No. 6, pp. 48–54 (2008) it may be possible to reproduce by correcting intermediate images by referring to the original objects in photographs [Meyerowitz 15] Meyerowitz, J.: Morandi’s Objects, Dami- by Meyerowitz. We want to try out these ideas in the fu- ani Editore (2015) ture. Finally, in terms of copyright, the revised Copyright Act, [Morandi 19] Morandi, G.: Giorgio Morandi, Silvana Edi- which came into effect in January 2019 in Japan, added an toriale (2019) item which states that it is legal to create machine learning [Salimbeni 19] Salimbeni, G., Leymarie, F. F., and models using images available on the Internet. To make Latham, W.: Generative System to Assist the Artist in sure, we asked a lawyer who is familiar with AI and data- the Choice of 3D Composition for a Still Life Painting, utilization to review our exhibit, and he told us that it was in Machine Learning for Creativity and Design (NeurIPS unlikely to be problematic from a copyright perspective un- 2019 Workshop) (2019) less it produced images that would be considered identical ´ to copyrighted work. In the first place, our intention was [Serio 13] Serio, A. D., Ib´a˜nez, M. B., and Kloos, C. D.: not to create a new work in the style of a deceased author Impact of an augmented reality system on students’ mo- like the Next Rembrandt project, nor is our intention to tivation for a visual art course, Computers & Education, claim copyright on the images produced. However, as the Vol. 68, pp. 586–596 (2013) results of the questionnaire showed concerns about copy- [Turing 50] Turing, A. M.: Computing Machinery and In- right, it is necessary to convey the intent of this exhibit telligence, Mind, Vol. 59, No. 236, pp. 433–460 (1950) carefully by adding appropriate explanations so as not to be misunderstood. [Yenawine 13] Yenawine, P.: Visual thinking strategies: Using art to deepen learning across school disciplines, 5. Conclusion Harvard Education Press (2013)

In this study, we proposed a new method for the active [Zhu 17] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A.: appreciation of artworks utilizing machine learning. We Unpaired Image-to-Image Translation using Cycle- chose Italian painter Giorgio Morandi as the theme, and Consistent Adversarial Networkss, in Proceedings of the developed a hands-on exhibit in which spectators freely ar- IEEE International Conference on Computer Vision, pp. range objects that reproduce the painter’s motifs and com- 2223–2232 (2017)

4