Computational Modeling of Artistic Intention: Quantify Lighting Surprise for Painting Analysis
Total Page:16
File Type:pdf, Size:1020Kb
Computational Modeling of Artistic Intention: Quantify Lighting Surprise for Painting Analysis Saboya Yang∗, Gene Cheungy, Patrick Le Calletz, Jiaying Liu∗ and Zongming Guo∗ ∗Institute of Computer Science and Technology, Peking University, China yNational Institute of Informatics, Japan zUniversity of Nantes, France Abstract—The use of strong lighting contrast to accentuate fascinate us. One notable artistic trend started in Renaissance objects and figures in a painting—called Chiaroscuro—is popular paintings is Chiaroscuro2: the use of strong contrasts between among Renaissance painters such as Caravaggio, La Tour and light and shadows to accentuate the three-dimensionality of Rembrandt. In this paper, we propose a new metric called LuCo objects and figures in compositions. Though its usage can to quantify the extent to which Chiaroscuro is employed by be traced back to early Byzantine art, it was most famously an artist in a painting. This measurement could be used to popularized by Caravaggio (1573-1610), who influenced later assess the capability of any system to fulfill the original artistic intention and consequently ensure minimal disruptions of Quality painters including Peter Paul Rubens (1577-1640), Georges de of Experience. We first argue that Chiaroscuro is a device for La Tour (1593-1652) and Rembrandt van Rijn (1606-1669). artists to draw attention to specific spatial regions; thus it can be As an example, we observe that in Crucifixion of St. Peter understood as a restricted notion of visual saliency computed by Caravaggio in Fig. 1(a), only the figures in the bottom of using only luminance features. Operationally, using a set of the painting are well lit. Extreme usage of Chiaroscuro is also local luminance patches we first compute a Bayesian surprise called Tenebrism3, which became the signature style for artists value, where the prior and posterior probabilities are computed like La Tour (see Smoker in Fig. 1(b)). assuming a Gaussian Markov Random Field (GMRF) model. Inverse covariance matrices of the GMRF model are estimated via sparse graph learning for robustness. We construct a histogram using the computed surprise values from different local patches in a painting. Finally, we compute a skewness parameter for the constructed histogram as our LuCo score: large skewness means luminance surprises are either very small or very large, meaning that the artist accentuated lighting contrast in the painting. Experimental results show that paintings by Chiaroscuro artists have higher LuCo scores than 19th century French Impres- sionists, and Rembrandt’s self-portraits have increasingly higher LuCo scores as he aged except for his late period—both trends are in agreement with art historians’ interpretations. (a) LuCo=1.70 (b) LuCo=3.76 Keywords—luminance contrast, painting analysis, graph-based Fig. 1. (a) Crucifixion of St. Peter by Caravaggio (1601), LuCo=1.70; (b) statistical learning Smoker by La Tour (1646), LuCo=3.76. Given the prevalence of Chiaroscuro in classical paintings, I. INTRODUCTION in this paper we propose a computational image model to In the working definition of Quality of Experience (QoE) estimate the extent of (Lu)minance (Co)ntrast employed by an proposed by Qualinet [1], it is stated that QoE “results from artist in a given painting, summarized succinctly in a metric the fulfillment of user expectations with respect to the utility called LuCo. We claim that this metric correctly quantifies and/or enjoyment of the application or service in the light an artist’s intention to manipulate lighting contrast to draw of the user’s personality and current state”. In the context visual attention to local regions; thus preserving LuCo score of creative content such as art paintings, artistic intention is in a media delivery chain would mean preserving the origi- another key factor to consider when evaluating the ability nal Chiaroscuro artistic intent. Compared to previous visual of a system to convey the right QoE. Thus the mandate saliency models [2], [6], [7], [8] that consider many low-level of any technical media delivery system should include the features and their correlations when constructing a saliency ability to faithfully convey the original artistic intent. From this map, our computational model has comparable complexity but perspective, a computational model of artistic intention would focuses exclusively on lighting contrast, resulting in a more be practically useful. In this paper, we propose to address this sophisticated and in-depth model based solely on luminance challenging topic focusing on one particular painting style1. features. Centuries have passed, and classical paintings by grand Operationally, given an observed network of patches in a masters of yester years like Rembrandt and Vermeer still local spatial region, we first compute a Bayesian surprise value [3], where the prior and posterior probabilities are computed 1We focus exclusively on Chiaroscuro in this paper, and leave the compu- tation modeling of other painting styles as future work. 2https://en.wikipedia.org/wiki/Chiaroscuro 978-1-5090-0354-9/16/$31.00 c 2016 IEEE 3https://en.wikipedia.org/wiki/Tenebrism assuming a Gaussian Markov Random Field (GMRF) model painting via a computation model. A texture feature vector was for the patch network. To ensure robustness, the inverse first constructed using Gabor wavelet coefficients of different covariance matrix (precision matrix) for the GMRF model is scales and orientations. A Gabor wavelet energy value is then estimated via learning of a sparse graph with few parameters, obtained, where larger energy values mean more contours and leveraging on recent advances in graph signal processing more visible brushstrokes. However, color and brushstroke are (GSP) [4]. We construct a histogram using the computed only two characteristics of an artist’s style. In contrast, we surprise values per local region. Finally, a skewness parameter focus on quantifying the amount of Chiaroscuro employed by is computed from the constructed histogram as the LuCo score an artist for a given painting. for the painting; a skewed histogram means that the luminance surprises are either very small or very large—in other words, III. PRELIMINARIES the artist accentuated lighting contrast to draw attention to a few spatial areas. To understand our proposed metric LuCo in Section IV, we first overview two key concepts in this section in order: We conduct experiments on a wide variety of classical Bayesian surprise, and graph spectral signal decomposition. paintings and observe the following two trends. First, among artists known for Chiaroscuro, we observe similar high com- puted LuCo scores. This is in contrast to paintings by 19th cen- A. Bayesian Surprise tury French Impressionists with lower computed LuCo scores. Informally, surprise is the arrival of an unexpected event Second, focusing on portraits by Rembrandt, we observe a or observation, one that is incongruent to the previous set of gradual increase in LuCo scores as he aged except for his expectations. Itti and Baldi formalized one notion of surprise, late period. This trend is consistent with commentaries by au- called Bayesian surprise [3], as a general information-theoretic thoritative art historians [5]. For comparison, we consider two concept. It measures the difference between the observer’s competitor types to detect luminance trends: i) computation of prior belief constructed based on previous observations, and skewness of luminance gradients across multiple scales in a his posterior belief based on previous and new observations. painting, and ii) computation of skewness of saliency values in an obtained saliency map computed using [2], [6], [7], [8]. First, denote by P N (y) the prior probability distribution X1 We observe that none of the competing schemes reveal the of a signal y, y 2 RM , constructed from a set of N previous N same trends we observe in computed LuCo scores, validating observations X1 = fx1;:::; xN g. Denote by P N+1 (y) the X1 the usefulness of our proposal. posterior probability distribution of y, constructed from pre- N The outline of the paper is as follows. We first overview vious observations X1 and new observation xN+1. Bayesian related works in Section II. We then review basic concepts of surprise is computed as the Kullback-Leibler (KL) divergence Bayesian surprise and GSP in Section III. We describe our [15] between the prior and posterior probabilities: graph-based metric LuCo in Section IV. Finally, results and N S(X1 ; xN+1) = KL(PXN+1 (y);PXN (y)); conclusions are presented in Section V and VI, respectively. 1 1 Z P N+1 (y) X1 = PXN+1 (y) log dy: (1) II. RELATED WORKS 1 P N (y) X1 Li and Chen [9] extracted color, shape and relative location features to automatically assess the aesthetic quality of a The definition in (1) is general; we discuss our definitions of prior and posterior probabilities PXN (y) and P N+1 (y) in painting. Other stylistic elements like lighting contrast and 1 X1 brush strokes are not considered. In contrast, we propose Section IV-A. a computational model to measure how much Chiaroscuro was employed by an artist, which captures one type of artist B. Graph Spectral Signal Decomposition intention and can be used for QoE evaluation. A undirected graph G is composed of nodes N and To discover correlations among artists, Bressan et al. [10] undirected edges E that connect nodes in G. A graph-signal built a graph to connect artists based on their similarities, M x, x 2 R , is a signal on a M-node graph G. x can computed using low-level features by the Fisher kernel [11]: be decomposed into its graph frequencies via graph Fourier combines discriminative features into a general kernel function. transform (GFT) [16]. Denote by W the adjacency matrix, Wang and Takatsuka [12] proposed a Self Organizing Map where wi;j ≥ 0 is the weight of the edge that connects nodes i (SOM) based hierarchical model considering color, composi- P and j.