Ukiyo-e Analysis and Creativity with Attribute and Geometry Annotation Yingtao Tian Tarin Clanuwat Chikahiko Suzuki Asanobu Kitamoto Google Brain ROIS-DS Center for ROIS-DS Center for ROIS-DS Center for Tokyo, Japan Open Data in the Humanities Open Data in the Humanities Open Data in the Humanities NII NII NII Abstract Attribute Value Title Kizukansuke The study of Ukiyo-e, an important genre of pre-modern (画題) (6木|勘助7) Japanese art, focuses on the object and style like other art- Painter Hirosada work researches. Such study has benefited from the renewed (!>) (Z貞) interest by the machine learning community in culturally important topics, leading to interdisciplinary works includ- Format Middle-size / Nishiki-e ing collections of images, quantitative approaches, and ma- (判¶) (?判/f!) Year in AD chine learning-based creativities. They, however, have sev- 1849 eral drawbacks, and it remains challenging to integrate these (西T) works into a comprehensive view. To bridge this gap, we ······ propose a holistic approach1: We first present a large-scale Ukiyo-e dataset with coherent semantic labels and geomet- Figure 1: An example of Ukiyo-e work in ARC Ukiyo-e ric annotations, then show its value in a quantitative study of Collection (Object Number arcUP2451 ) titled Kizukansuke Ukiyo-e paintings’ object using these labels and annotations. by painter Hirosada. The painting on the left is accompanied We further demonstrate the machine learning methods could by metadata for this work on the right. For example, meta- help style study through soft color decomposition of Ukiyo- data further indicates this work is a middle-sized Nishiki-e e, and finally provides joint insights into object and style by (multi-colored woodblock printing) produced in 1849. composing sketches and colors using colorization. Introduction objects depicted over time, and the latter allows the identi- th th fication of artists (Suzuki, Takagishi, and Kitamoto 2018). The Edo period of Japan (16 to 19 century) has seen The renewed interest by the machine learning community the prosper of Ukiyo-e (oã!), a genre of pre-modern in the culturally essential topics has led to works address- Japanese artwork that consists of paintings and woodblock ing the traditional Japanese artworks from an interdisci- printings. Unlike early dominating Emakimono (!巻物, plinary perspective. Along this line of research, building picture scroll) and Ehon (!%, picture book) that focus on open collections of digitized images has been proposed for famous figures and stories in Sinosphere culture and clas- Ehon (Suzuki, Takagishi, and Kitamoto 2018) and Ukiyo- sic Japanese stories, the topic of Ukiyo-e extends broadly to e (Art Research Center, Ritsumeikan University 2020; daily subjects, such as characters like beauties and Kabuki Pinkney 2020). Further works use quantitative approaches (é~Ã), landscape arts, animals and plants in everyday into the object for artworks, such as studying the geometry life, and even contemporary news. As an example, Fig- features of Buddha statues (Renoust et al. 2019) and Ukiyo- ure 1 shows a Ukiyo-e depicting a Kabuki performance. arXiv:2106.02267v1 [cs.CV] 4 Jun 2021 e faces (Renoust et al. 2019), Alternatively, inspired by the The popularity of woodblock printing makes it possible to art nature of painting, machine learning-based creativity has produce paintings on a larger scale at a lower cost, which been leveraged for studying style, such as painting process contributes to the flourish of Ukiyo-e and leaves us with a generation (Tian et al. 2020) and image synthesis across art- vast collection of artworks in this genre (Kobayashi 1994; work and photorealistic domains (Pinkney and Adler 2020). IUS 2008). Such an extensive and varied collection provides These works provide valuable connections between machine a valuable corpus for Japanese artwork research. learning and the humanities research of Japanese artwork. The subject of such artwork study could be multi-faceted involving several aspects, of which two crucial are the ob- We, however, also notice that these works present sev- ject in the painting, such as the outline and the shape of eral drawbacks. For example, collection on digitized im- depicted figures, and the style of painting, such as textures ages may either comes with no semantic (Pinkney 2020) or and colors. For example, the former reveals the trend of is in a format not designed with machine learning-based ap- plications in mind.Furthermore, quantitative approaches are 1https://github:com/rois-codh/arc-ukiyoe- only conducted on a small set of artworks (Murakami and faces Urabe 2007) or require extensive human labor to adapt for Ukiyo-e (Renoust et al. 2019), and machine learning-based creativity works may deal more with cross-domain art ex- pression (Pinkney and Adler 2020) than the very domain of artwork on which humanities research focuses. Finally, the art study into a particular genre requires insights into both the object and style to acquire a comprehensive understand- ing. Current works, however, only address one of the object or style, falling short of the expectation. Facial Region Landmarks To overcome the aforementioned drawbacks and to pro- Left Eye Center, Left, Right, Up, Down Right Eye Center, Left, Right, Up, Down vide deeper insight into the artistic style of Ukiyo-e, we pro- Left Eyebrow Left, Right, Up pose a new approach that is (1) holistic in both studying the Right Eyebrow Left, Right, Up object and style through the joint use of images, labels, and Left Pupil Center annotations, and (2) powered by large scale data and state- Right Pupil Center Mouth Left, Right, Up, Down of-the-art machine learning model than the prior works. To Nose Center, Left, Right summarize, our main contributions are as follow: Jawline Upper Left & Right, Mid Left & Right, Chin Bottom • We present a large-scale (11; 000 paintings and 23; 000 Figure 2: An exmaple of detected landmarks and the ex- faces) Ukiyo-e dataset with coherent semantic labels and tracted face in Figure 1’s Ukiyo-e painting. On the left, the geometric annotations, through augmenting and organiz- red dots show detected facial landmarks and the rectangle ing existing datasets with automatical detection. shows the bounding box inferred from these landmarks. The • We are the first to conduct a large-scale quantitative study right image shows the extracted face from the bounding box. of Ukiyo-e paintings (on more than 11; 000 paintings), The table lists a summary of all landmark locations. providing understanding into object in artworks by jointly quantifying semantic labels and geometric annotations. • We show that machine learning-based models could pro- vide insights into style by decomposing finished Ukiyo- e images into color-split woodblocks that reflect how Ukiyo-e images were possibly produced. • We study and show machine learning-based creativity model could engage problems that arise jointly studying Figure 3: Faces with their landmarks. In each row, we show object and style by separating geometry shapes and artis- six examples of extracted faces annoated with their corre- tic styles in an orthogonal and re-assemblable way. sponding landmarks in the same format as Figure 2. Dataset experts. It has 11; 103 entries of painting and the associated Art research in traditional paintings often asks questions re- metadata, one example of which is shown in Figure 1. This garding the work, like the author and production year. One service allows researchers to dive into curated metadata for focus in such research is on faces since they could help an- comparative study for art research. swer these questions through quantitative analysis. In this direction, Collection of Facial Expressions (Suzuki, Takag- Another dataset is Ukiyo-e Faces Dataset (Pinkney 2020), ishi, and Kitamoto 2018; Tian et al. 2020) provides a large- a public available dataset of Ukiyo-e faces extracted from scale (8848 images) set of coarse-grained cropped faces. Ukiyo-e images available online. With 5; 000 high-quality Another study (Murakami and Urabe 2007) deals with facial faces, this dataset plays an essential role in controllable im- landmarks which are more fine-grained than cropped faces age generation across Ukiyo-e faces and photo-realistic hu- to support quantitative analysis. However, its manual label- man faces (Pinkney and Adler 2020). However, as this ing process only allows analysis on a small set (around 50 dataset focuses on image synthesis, it does not include meta- images) of Ukiyo-e paintings. data for Ukiyo-e paintings from which faces are extracted. To combine both works’ advantage, we extend existing datasets through augmentation and automated annotation, Geometric Annotation with Facial Landmark resulting in a large-scale Ukiyo-e dataset with a more fine- Detection grained facial feature. The rest of this section details the Inspired by Pinkney (2020) , we use an face recognition process and analysis of our new proposed dataset. API, Amazon Rekognition (link), to detect facial landmarks in in Ukiyo-e Faces Dataset paintings. Despite targeting Fundamental Datasets photo-realistic human face images, this API demonstrates We build our work based on two foundation datasets. One compelling accuracy on Ukiyo-e paintings. Since the de- of them is ARC Ukiyo-e Collection (Art Research Center, tected faces may not be well-aligned, we infer the possibly Ritsumeikan University 2020), a publicly available service rotated bounding box of faces for cropping faces from the that provides access to digitized Ukiyo-e paintings primar- painting, inspired by the preprocessing in FFHQ (Karras, ily in the Edo period, plus metadata compiled by domain Laine, and Aila 2019). In Figure 2 we show an example of (a) Distribution of years with respect to authors. (b) Distribution of years of all works in the dataset. Painter Examples Hirosada (Z貞) Kogyo (y.) Kunichika (国Å) Kunisada (1st gen) (国貞 初代) Kunisada (2nd gen) (国貞 f代目) Kunisada (3rd gen) (国貞 三代目) Kuniyoshi (国芳) Toyokuni (1st gen) (豊国 初代) Toyokuni (3rd gen) (豊国 三代目) Yoshitaki (芳滝) (c) Example of paintings, represented by the extracted faces, by authors.