Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10) Visual Contextual Advertising: Bringing Textual Advertisements to Images Yuqiang Chen† Ou Jin† Gui-Rong Xue† Jia Chen† Qiang Yang‡ †Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China ‡Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong †{yuqiangchen, kingohm, grxue, chenjia}@apex.sjtu.edu.cn ‡[email protected] Abstract context based advertisement. Therefore, adverting for mul- timedia data is increasingly in need. Advertising in the case of textual Web pages has been studied extensively by many researchers. However, Considering the problem of recommending advertise- with the increasing amount of multimedia data such as ments for Web images, traditional methods largely rely on image, audio and video on the Web, the need for recom- the textual contexts of images, such as surrounding text and mending advertisement for the multimedia data is be- social tags, to extract keywords and then obtain relevant coming a reality. In this paper, we address the novel advertisements through textual information retrieval. How- problem of visual contextual advertising, which is to ever, there are a large amount of Web images with little or directly advertise when users are viewing images which no text contexts. Furthermore,text can be noisy and ambigu- do not have any surrounding text. A key challenging is- ous, which could reduce the accuracy for the recommended sue of visual contextual advertising is that images and advertisements. advertisements are usually represented in image space and word space respectively, which are quite different In this paper, we present a novel approach to bring textual with each other inherently. As a result, existing meth- advertisements (ADs) to images, so that images and ADs are ods for Web page advertising are inapplicable since they closely matched to the true meaning of images rather than represent both Web pages and advertisement in the same the textual context of images. Specifically, we focus on the word space. In order to solve the problem, we propose visual contextual adverting problem, which aims to recom- to exploit the social Web to link these two feature spaces mending textual advertisements for Web images without the together. In particular, we present a unified generative help of any textual context, such as surrounding text for the model to integrate advertisements, words and images. images. Unlike previous methods for contextual advertising, Specifically, our solution combines two parts in a prin- in our approach, advertisements are recommended entirely cipled approach: First, we transform images from a im- based on the visual contextual of an image rather than its age feature space to a word space utilizing the knowl- edge from images with annotations from social Web. textual context in our setting. As a result, our approach can Then, a language model based approach is applied to recommend advertisements even for images with little or no estimate the relevance between transformed images and textual context. advertisements. Moreover, in this model, the probabil- Visual contextual advertising can be applied to various ity of recommending an advertisement can be inferred Web applications in reality. For example, many images in efficiently given an image, which enables potential ap- online albums lack annotations, and it will be helpful to ad- plications to online advertising. vertise the images based on the content of them. Another example is that when visiting entertainment websites, it will 1 Introduction be beneficial if we can advertise the cloths or watch worn by a fashion star when browsing the star’s image, in which Online advertising is one of the booming sectors of the Web case no corresponding text is available. Yet another example based business world. Huge revenue has been made each is when one browse his own images on a mobile phone, in year in this area. Traditional online advertising researches, which case appropriate Ads are required to be chosen for the such as contextual advertising (Ribeiro-Neto et al. 2005), user with high accuracy. focus on delivering advertisements for textual Web pages so Ideally, in order to perform visual contextual advertis- that they are matched as closely as possible. Researchers ing, the algorithm must first understand the images and on contextual advertising area have proposed various ways then make appropriate recommendations based on the un- to deal with the problem, e.g. (Ribeiro-Neto et al. 2005; derstanding. However, images are often in a image fea- Lacerda et al. 2006; Broder et al. 2007). However in recent ture space, such as color histogram, or scale invariant fea- years, the rapid increase in multimedia data, such as image, ture transform (Lowe 2004) descriptors, while the advertise- audio and video on the Web, presents new opportunities for ments are usually represented in a word space, which usually Copyright c 2010, Association for the Advancement of Artificial consists of textual features from bid phrases, landing pages, Intelligence (www.aaai.org). All rights reserved. etc. As a result, traditional contextual text advertising meth- 1314 syntactic features to calculate the relevance score between a web page and advertisements (Broder et al. 2007). All the mentioned studies intend to advertise based on textual web page context, rather than visual textual. Mei, Hua, and Li presented a novel contextual advertising algo- rithm using both surrounding textual information and local visual relevance (Mei, Hua, and Li 2008). An image adver- tisement would be seamlessly inserted into the target image relying on a saliency map. However in (Mei, Hua, and Li 2008), image information was mainly used as a complement for textual annotation when annotations were insufficient or the quality was low. In this paper, we consider a more chal- lenging scenario where no surrounding text is given and ad- vertisements are recommended entirely based on the visual Figure 1: An illustrative example of our solution for visual context. contextual adverting problem. Originally images are repre- sented in different feature space from advertisements, which 2.2 Image Annotation poses difficulties for advertising. With the help of annotation Another closely related area is image annotation. Duygulu data from social Web, we can link the images and advertise- et al. regarded the image annotation as a machine translat- ments, and thus advertise directly based on the context of ing process (Duygulu et al. 2002). Some other researchers images. model the joint probability of images regions and annota- tions. Barnard et al. (Barnard et al. 2003)investigatedimage annotation under probabilistic framework and put forward a ods are unable to handle this problem. number of models for the joint distribution of image blobs In order the overcome the above-mentioned problems, and words . Blei and Jordan (Blei and Jordan 2003) devel- we exploit the annotated image data from social Web sites oped correspondence latent Dirichlet allocation to model 1 such as Flickr to link the visual feature space and the word the joint distribution. In (Jeon et al. 2003), continuous- space. An illustrative example of our proposed method for space relevance model was proposed to better handle contin- visual contextual advertising is given in Figure 1. To be spe- ues features and be free from the influence of image blobs cific, we present a unified generative model, ViCAD, to deal clustering. In (Carneiro et al. 2007), image annotation is with the visual contextual advertising. ViCAD runs in sev- posed as classification problems where each class is defined eral steps. First, we model the visual contextual advertising by images sharing a common semantic label. While visual problem with a Markov chain which utilizes annotated im- contextual advertising presented in this paper has some sim- ages to transform images from the image feature space to ilarity with image annotation, some key differences exist. the word space. With the representations of images in word A major difference is that the advertisements correspond to space, a language model for information retrieval is then ap- groups of fixed keywords rather than collections of indepen- plied to find the most relevant advertisements. Moreover, dent keywords as in the case of image annotation. As such, we show that the inference of the model can be performed there is a need to tradeoff advertisement selection with the efficiently by constraining the word space for representing accuracy of individual words. Advertisement selection also image to be a smaller subspace, which allows potential ap- relates to diversity of selected advertisements as a whole, plications of using our model in online image advertising. and other important factors such profit. 2 Related Work 3 ViCAD Algorithm for Visual Contextual 2.1 Contextual Advertising Advertising The online advertising problem is getting increasingly im- 3.1 Problem Formulation portant with the development of the Web business. A pop- First we define the problem of visual contextual advertis- ular approach in online advertising is contextual advertis- ing formally. Let W = {w1, w2, . , wm} be the vocab- ing, which is to place advertisements in a target web page ulary space, where wi is a word and m is the size of vo- based on the similarity between the content of target page cabulary. Let T be the advertisement space. In this space, and advertisement description. Various researchers have ad- each advertisement ti ∈ T is represented by a feature vector 1 2 m dressed the problem of contextual adverting (Ribeiro-Neto (ti , ti , . , ti ) on the word space W . We denote V as the et al. 2005; Lacerda et al. 2006; Broder et al. 2007). In image space, in which each image vi ∈ V is represented by 1 2 n (Ribeiro-Neto et al. 2005), ten strategies of contextual ad- feature vector (vi , vi , . , vi ) on the image feature space vertising were proposed and compared.