Detecting Rule of Simplicity from Photos

Detecting Rule of Simplicity from Photos Long Mai, Hoang Le, Yu-Chi Lai Feng Liu Yuzhen Niu National Taiwan University of Portland State University Portland State University Science and Technology Portland, OR 97207-0751 Portland, OR 97207-0751 #43, Sec. 4, Keelung Rd., fl[email protected] {mtlong, hoanl, Taipei 106, Taiwan yuzhen}@cs.pdx.edu [email protected] ABSTRACT Simplicity refers to one of the most important photography composition rules. Simplicity states that simplifying the image background can draw viewers’ attention to the subject of interest in a photograph and help them better comprehend and appreciate it. Understanding whether a photo respects photography rules or not facilitates photo quality assessment. In this paper, we present a method to automat- ically detect whether a photo is composed according to the (a) Homogenous or empty background rule of simplicity. We design features according to the defi- nition, implementation and effect of the rule. First, we make use of saliency analysis to infer the subject of interest in a photo and measure its compactness. Second, we segment an image into background and foreground and measure the homogeneity within the background as another feature. Third, when looking at an image created with the rule of simplicity, different viewers tend to agree on what the subject of interest is in this photo. We accordingly measure the consis- (b) Out-of-focus background (c) Cluttered background tency among various saliency detection results as a feature. Figure 1: Rule of simplicity. Simplifying the back- We experiment with these features in a range of machine ground in an image by making it homogenous, learning methods. Our experiments show that our methods, empty, or out of focus, can emphasize the subject together with these features, provide an encouraging result of interest and help viewers to better comprehend in detecting the rule of simplicity in a photo. and appreciate it (a) and (b). An image with cluttered background is difficult for viewers to focus on Categories and Subject Descriptors the main subject (c). I.4.9 [Image Processing and Computer Vision]: Appli- interest in a photo standing out from its surroundings [10]. cations Simplicity can be achieved by simplifying the image background, such as making it homogeneous, empty, or com- Keywords pletely out of focus, as shown in Figure 1 (a) and (b). A photo that respects the rule of simplicity can easily draw Photo Quality Assessment, Photography Rules Detection viewers’ attention to the subject of interest and help them better comprehend and appreciate it. In contrast, an image 1. INTRODUCTION with cluttered background is difficult for viewers to focus on Photo composition is an important technique used by pho- the main subject, as shown in Figure 1 (c). tographers to create a high-quality photo. Composition refers Computational understanding of how a photo is created to the placement of visual elements in a photo. Among a is important for applications like photo quality assessment variety of composition rules, simplicity is one of the most and photo authoring. For example, while the rule of simplic- important ones. Simplicity is used to make the subject of ity has been widely practised by professional photographers, amateur users often tend to ignore it or cannot effectively use it due to the lack of expertise or equipment. The exis- tence of the rule of simplicity in a photo is a good evidence Permission to make digital or hard copies of all or part of this work for of high quality. personal or classroom use is granted without fee provided that copies are This paper focuses on the rule of simplicity and presents not made or distributed for profit or commercial advantage and that copies a method to detect it in a photo. Detecting the rule of bear this notice and the full citation on the first page. To copy otherwise, to simplicity requires identifying the subject of interest in a republish, to post on servers or to redistribute to lists, requires prior specific photo. While image content understanding has been pro- permission and/or a fee. MM’12, October 29–November 2, 2012, Nara, Japan. gressing these years, generic content understanding is still Copyright 2012 ACM 978-1-4503-1089-5/12/10 ...$15.00. an ongoing research problem. We use saliency analysis as 85 an alternative. Saliency measures low-level stimulus to the human vision system [8] and has been used to infer impor- 80 tant content in an image in multimedia applications such as ) % multimedia retargeting [14] and video summarization [13]. ( 75 We consider that an image with a compact saliency distri- 70 bution is likely to respect the rule of simplicity. Second, on rate i when looking at a photo created with the rule of simplicity, 65 cat i f the subject of interest captures a viewer’s eye immediately. i GBVS 60 FT Therefore, different viewers tend to agree on what the sub- ass l c GC ject of interest is in the photo. We then measure the consis- 55 tency among various saliency detection results as a feature. 50 Third, we segment an image into background and foreground 0 10 20 30 40 50 60 70 80 90 100 and measure the homogeneity within the background as an- α : percent of saliency value in the selected region (%) other feature. These features are used in a range of machine Figure 2: classification accuracy vs. α value learning algorithms to detect the rule of simplicity. Our work is relevant to the recent research on photo quality assessment (e.g. [4, 9, 11, 12]). These methods measure the whole subject of interest; however, it will include some α how an image respects the rule of simplicity as a feature of the background area. On the other hand, a small value and use it together with others for photo quality assessment. can make sure that only the image region belonging to the This paper aims to explicitly determine whether the rule of subject of interest is included; however, the selected region α simplicity is applied to an image by designing novel features may miss some part of the subject of interest. We set the and using them in machine learning algorithms for detec- value experimentally by cross validation on a training data tion. To the best of our knowledge, our method is the first set. The performance of this feature in detecting the rule of α to detect the simplicity photo composition rule in a photo. simplicity with respect to the value is illustrated in Fig- ure 2. This figure shows that the optimal α value for the GBVS saliency map, FT saliency map, and GC saliency map 2. FEATURE DESIGN are 85%, 70%, and 55% respectively. We design three types of features according to the spa- tial distribution of visual elements, the background simplic- 2.2 Background Simplicity ity, and the consistency among the saliency detection results The rule of simplicity states that the background should from different methods. We below describe how these fea- be simplified. We accordingly measure the simplicity in tures are designed to detect the rule of simplicity in a photo. the background as a feature. We segment an image into two regions using the normalized cut image segmentation 2.1 Subject of Interest Compactness method [15] and select the big region as the background. The rule of simplicity recommends that the subject of We measure the simplicity value in the background as the interest should be surrounded by a simplified background. average distance between two image blocks. This suggests that the image region with the subject of in- f 1 d b ,b terest tends to be compact. We accordingly measure the sim = |B| ( i j )(2) compactness of the subject of interest in an image as a fea- bi,bj ∈B ture to detect the rule of simplicity. where bi and bj are two blocks in the background B,and As inspired by the success of image saliency in multimedia d b ,b applications, we use saliency analysis to infer the subject of ( i j ) is the distance between these two blocks. We use interest. Specifically, saliency analysis produces a saliency the distance function from the segmentation method [15]. 2 2 map for an input image. Since saliency analysis is often −||X(bi)−X(bj )|| −||F (bi)−F (bj )|| noisy, we divide each saliency map into n × n blocks with d(bi,bj )=e σs e σc (3) n = 20 and compute the average pixel saliency value in a where F (b)=[v, v.s. sin(h),v.s.cos(h)] block as the saliency value for the block. We then identify the subject of interest as a minimal number of blocks that where X(b) denotes the position of the block center of b. altogether contain at least α% of the total saliency in the F (b) is a color descriptor vector, where h, s,andv are the image. We finally compute the ratio between the number hue, saturation, and value for the average color of b in the of blocks in the subject of interest and the total number HSV color space. σs and σc are two parameters with value of blocks in an image to measure the compactness of the 1.5 and 0.01 respectively. subject of interest. 2.3 Saliency Detection Consistency N f α The first two features are constructed according to how cpt = N (1) the rule of simplicity is defined and implemented. We now where Nα is the number of blocks in the subject of interest describe how we define a feature based on the goal and ef- and N = n2 is the total number of blocks in the image.

Load more