A Practical Exploration System for Search Advertising Parikshit Shah Ming Yang Sachidanand Alle [email protected] [email protected] [email protected] Adwait Ratnaparkhi Ben Shahshahani Rohit Chandra [email protected] [email protected] [email protected] ABSTRACT advertisers a lever to increase the exploration rate for their In this paper, we describe an exploration system that was imple- cold ads, mented by the search-advertising team of a prominent web-portal • New practical methods of evaluation - such as a novel way to address the cold ads problem. e cold ads problem refers to of tracking the performance of good versus bad ads found the situation where, when new ads are injected into the system via exploration. Since our exploration system is tightly by advertisers, the system is unable to assign an accurate quality coupled with the production click model, and the design to the ad (in our case, the click probability). As a consequence, and evaluation are crucially inuenced by the same; we the advertiser may suer from low impression volumes for these describe this interaction in later sections. cold ads, and the overall system may perform sub-optimally if the Exploration has a rich history in the machine learning literature click probabilities for new ads are not learnt rapidly. We designed [2–5, 9]. Exploration approaches have been studied in various a new exploration system that was adapted to search advertising contexts such as news article recommendation, native advertising, and the serving constraints of the system. In this paper, we dene search advertising [9, 10], reinforcement learning and its application the problem, discuss the design details of the exploration system, to computer games [8]. Furthermore, exploration has also been new evaluation criteria, and present the performance metrics that used to radically improve the traditional A/B-testing paradigm such were observed by us. as the work on Multi-World Testing [1]. In contrast to previous existing works in the applied literature, which focus on designing 1 INTRODUCTION exploration to achieve fast online learning with low regret, in this work we describe a practical approach that is specically tailored A basic problem faced by any content delivery system is the cold- to search ads and is thus tailored to the ad auction – both in terms start problem: when new (“cold”) content is injected into the system of the exploration implementation as well as the evaluation. that competes with other “warmer” content, how does the system is paper is organized as follows: In Section 2 we discuss the learn about the quality of the new content and deliver it appropri- relevant background concerning the search advertising system. In ately and reliably? In addition to improved modeling techniques, Section 3 we describe and discuss the details of our exploration a critical component is a mechanism for exploration. A typical system. In Section 4 we discuss the evaluation criteria and the exploration system randomly boosts cold content over existing metrics. competing content in order to learn about it. While conceptually simple, the mechanism must be designed carefully to manage the exploration-exploitation tradeo in key metrics such as user engage- 2 BACKGROUND ment and revenue, while balancing various other implementation Search advertising, the method of placing relevant online advertise- challenges. ments on web pages that show results from search engine queries, e work reported in this paper was conducted with a view to has become an important part of the online user experience. Search improving the performance of the system on cold ads. We detail advertising is an extremely aractive proposition for advertisers our eorts to implement and evaluate a new exploration system for because the search query provides a powerful relevance signal that the search advertisting system. While the basic exploration scheme can be used for targeting only the most appropriate ads. Conceptu- that we have implemented is based on well-known ideas (ϵ-greedy, ally, a typical search advertising system consists of the following upper-condence bound (UCB) based approach), we introduced components: novel aspects to the design that may of broader interest. Among • Campaign management system: this is an inventory of ad- these are: vertisements (or creatives), along with their title, descrip- • Methods for factoring bid information in the sampling tion, and search keywords based on which they become mechanism that guards against revenue loss and provides eligible for display when the search query is similar to the Permission to make digital or hard copies of all or part of this work for personal or keyword. Each advertiser creative also has associated with classroom use is granted without fee provided that copies are not made or distributed it a bid. Importantly, new advertisments are introduced for prot or commercial advantage and that copies bear this notice and the full citation by the adverisers in the campagin management system on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permied. To copy otherwise, or republish, periodically. to post on servers or to redistribute to lists, requires prior specic permission and/or a • Matching: e matching system is reponsible for under- fee. Request permissions from [email protected]. standing the query and retrieving all the relevant ads (from KDD’17, August 13–17, 2017, Halifax, NS, Canada. © 2017 ACM. 978-1-4503-4887-4/17/08...$15.00 the campaign management system) that match the query DOI: hp://dx.doi.org/10.1145/3097983.3098041 context. • Click-model: e click-model reports an estimate of the i.e. the product of its bid and predicted CTR must be above probability of a click for ad a, in the context of a queryq by a a certain threshold (we denote it by treserve. If the cre- useru, denoted bypq;a;u . Each element of the triple (q;a;u) ative falls below this monetization threshold, the creative can be further expanded into more detailed features; for is deemed to be not worth (in monetary terms) the adverse example the query can be broken down into the tokens user experience created by the introduction of the ad on and the characters, the advertisement is associated to a the search page. particular doman, campaign, ad-group and has title and • Auction: A creative can be blocked from gaining impres- description features, and lastly the user information may sion volume because it competes in a deep-market query contain the user-cookie, location, time, gender, IP address with many other ads with a higher rank-score. For in- type, etc. which are used as training inputs to the model. stance, if no more than ve ads are eligible for display on • Auction: e auction determines the nal selection of cre- the search page, and there are at least ve other ads with atives to be displayed, their relative page positions, and a higher score (i.e. the product of bid and predicted CTR), the price-per-click for each. Since search revenue is click- the new creative will not gain impression volume. driven (i.e. an ad is only monetized if upon display it is A crucial observation is that the above blockers to impression clicked by the user), both the bid as well as the click proba- volume are all sensitive to the predicted CTR, pq;a;u . AQF directly bility is used in a rank-score for selection/ranking of nal depends on the predicted CTR, and the CTR is the only system- ads. e most commonly used rank-score is the product of related factor in the reserve price as well as the auction (since the bid and the estimated click-probability. the bids are controlled by the advertisers). us, if the CTR is A basic challenge that is faced in the context of search adver- predicted inaccurately on cold ads, impression volume is directly tising is the cold-ad problem. A cold ad refers to a creative that impacted. Furthermore, a substantial volume of creatives returned is introduced by an advertiser which has relatively few or no his- by matching are cold, and thus the overall performance of the torical impressions (we call the laer “frozen” ads). e cold ad system depends crucially on predicting on cold ads accurately. Fig. problem is actually a two-fold problem: 1 shows a histogram of impression volume by its coldness level, and (1) When the ad is cold and has insubstantial historical impres- indicates that a signicant fraction of ads returned are cold/frozen sions, many of the ad-specic features are unknown and (a frozen ad is one that has no impression history whereas a cold thus inaccurately learnt. us, the predicted CTR (i.e the ad is one with small impression history). click-through-rate) of the ad, i.e. pq;a;u is not accurate and can aect the performance of this creative in adverse ways. 2.2 Click Model and Cold Ads is in turn can cause the overall system to behave sub- e current production click-model is a supervised feature-based optially, either showing poor quality new ads or showing model: stale ones repeatedly instead of new, beer quality ads. p(clickjq;a;u) = pq;a;u := F (fj (q;a;u)) (2) When a new ad is introduced by an advertiser, the intent where f (q;a;u) is the jth feature extracted for query q, is for it to gain impression volume. However, due to the j j=1;:::;N predicted CTR being inaccurate, some new ads may be ad a and( user u, and) F is chosen from a parameteric function class underpredicted and fail to gain impression volume.