A Data-Driven Metric of Incentive Compatibility Yuan Deng Sébastien Lahaie Duke University Google Research Durham, NC New York, NY [email protected] [email protected] Vahab Mirrokni Song Zuo Google Research Google Research New York, NY Beijing, China [email protected] [email protected] ABSTRACT practice are not incentive-compatible, notably in the domain of An incentive-compatible auction incentivizes buyers to truthfully ad auctions. The display advertising industry, including Google, reveal their private valuations. However, many ad auction mech- has recently switched to first-price auctions [8, 21, 34]. Moreover, anisms deployed in practice are not incentive-compatible, such some ad exchanges have allegedly used soft floors in addition to as first-price auctions (for display advertising) and the general- standard reserve prices [37], which mixes the incentives of first- ized second-price auction (for search advertising). We introduce and second-price auctions. Generalized second-price auctions are a new metric to quantify incentive compatibility in both static widely deployed for search advertising; although inspired by Vick- and dynamic environments. Our metric is data-driven and can be rey auctions, it was recognized early on that these auctions are not computed directly through black-box auction simulations without truthful [15, 26, 36]. relying on reference mechanisms or complex optimizations. We As originally defined, incentive compatibility is a binary notion— provide interpretable characterizations of our metric and prove a mechanism is either incentive compatible or it isn’t. To achieve that it is monotone in auction parameters for several mechanisms a more nuanced comparison between mechanisms, or between used in practice, such as soft floors and dynamic reserve prices. We different parametrizations of a single mechanism, there has been empirically evaluate our metric on ad auction data from a major a growing amount of interest in developing metrics that quantify ad exchange and a major search engine to demonstrate its broad incentive compatibility. The most common approach is to rely on applicability in practice. the concept of regret, which is a buyer’s utility difference between best responding and truthful reporting [4, 6, 10, 13, 32]. Regret is CCS CONCEPTS an appealing measure because its units are linked to utility and directly interpretable. However, computing a regret-based metric • Theory of computation → Computational advertising the- requires solving for the best response of a buyer for every possible ory. valuation, which can be a complex optimization task. Even worse, KEYWORDS auction systems in practice are becoming more complex and opaque incentive compatibility metric, ad auction, truthfulness so that their mechanics can be difficult to model let alone optimize ACM Reference Format: against. Yuan Deng, Sébastien Lahaie, Vahab Mirrokni, and Song Zuo. 2020. A Data- In this paper, we introduce a new data-driven metric to quantify Driven Metric of Incentive Compatibility. In Proceedings of The Web Confer- incentive compatibility for both static and dynamic environments ence 2020 (WWW ’20), April 20–24, 2020, Taipei, Taiwan. ACM, New York, based on Myerson’s classic characterization of the relationship NY, USA, 11 pages. https://doi.org/10.1145/3366423.3380249 between allocation and payment rules of incentive-compatible auc- tions [30]. To compute our metric, one simply applies small per- 1 INTRODUCTION turbations to bids and records the resulting bidder utilities. For Mechanism design addresses the problem of achieving desirable out- ad auctions this can be achieved by black-box simulations over comes by eliciting private valuation information held by multiple auction logs or, for a more faithful evaluation that captures bidder agents. A mechanism is incentive-compatible—also called truth- behavior, by applying the perturbations to small slices of experimen- ful or strategyproof—if it guarantees that an agent’s (weakly) best tal traffic. Once this data is collected our metric can be computed strategy is to truthfully reveal its private information. This leads using straightforward database queries, along with the usual stan- to straightforward participation with predictable actions on the dard errors and confidence intervals. This simplicity and scalability part of the agent. However, many auction mechanisms fielded in is a major advantage over previous methods that rely on know- ing the reference mechanisms that are incentive-compatible [27] This paper is published under the Creative Commons Attribution 4.0 International or complex optimizations to compute profile-by-profile best re- (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution. sponses [4, 13, 32]. WWW ’20, April 20–24, 2020, Taipei, Taiwan Our metric takes the form of an index that lies between 0 and 1 © 2020 IW3C2 (International World Wide Web Conference Committee), published for reasonable mechanisms: we show that it is non-negative if utility under Creative Commons CC-BY 4.0 License. ACM ISBN 978-1-4503-7023-3/20/04. https://doi.org/10.1145/3366423.3380249 WWW ’20, April 20–24, 2020, Taipei, Taiwan Yuan Deng, Sébastien Lahaie, Vahab Mirrokni, and Song Zuo under truthful bidding is non-decreasing in an agent’s true valua- ex-post regret and the error in multi-class classifiers, enabling the tion, and that it is at most 1 if overbidding is a weakly dominated application of structural support vector machines to the design of strategy. The bounds of our metric are meaningful: it is always low-regret mechanisms. Recently, Duetting et al. [12] introduce the 1 for incentive-compatible auctions, and for first-price auctions idea of using deep learning for auction design and Balcan et al. [4] where the bid most directly influences the payment it has value 0. apply statistical learning techniques to estimate interim incentive Moreover, for a mixture of a truthful auction and the corresponding compatibility using regret as a measure. All of these regret-based first-price auction with the same allocation rule, our metric exactly approaches require complex optimizations in order to compute evaluates to the fraction of the truthful auction in the mixture. Our profile-by-profile best-responses, while our metric can be computed metric can be viewed as a measure of the marginal benefit of a via black-box auction simulations and simple database queries. bidding strategy which bids a scaled version of the true value, with Recent work attempts to address these computational difficulties the same scaling factor throughout (i.e., uniform bid shading). To by approximating regret. Feng et al. [20] design online algorithms to add another interpretation, we show that the metric is associated minimize (and therefore compute) regret, with an eye towards fast with the difference between the payment function that truthfully convergence, and evaluate their algorithms over GSP with synthetic implements the allocation rule and the one used in the auction. data. Colini-Baldeschi et al. [9] introduce the concept of IC-Envy, Our metric can be applied to both static and dynamic mecha- which is easier to compute but bounds or even equates to regret nisms. A dynamic mechanism maintains state so that an agent’s in important domains like position auctions. IC-Envy can also be bids can influence future payoffs. For instance, an ad auction might used to bound social welfare loss due to misreports, but we are not use dynamic reserve prices, which are set based on past bid dis- aware of a way to extend the concept to dynamic environments. tributions [22]; or the auction might throttle bidders who have Other than the regret-based approaches, Pathak and Sönmez lost too many auctions in the past. To the best of our knowledge, [33] provide a ranking for mechanisms without payment, such regret-based approaches have not been extended to dynamic envi- as matching mechanisms, based on the number of instances that ronments. We provide closed-form characterizations of our metric are manipulable. Troyan and Morrill [35] introduce the concept for several static and dynamic mechanisms deployed in practice, of an “obvious manipulation” and propose to compare mechanism such as soft floors and dynamic reserve pricing. This leads usto incentives based on this criterion. Lubin and Parkes [27] quantify a notion of incentive monotonicity, which is the property that our strategyproofness according to the divergence between a mecha- metric is monotone in an auction parameter. We have found this nism’s payoffs and those of a strategyproof reference mechanism. In useful for reasoning about how certain parameters influence in- contrast, our metric does not rely on the access to a reference mech- centives. For dynamic reserve pricing, our metric demonstrates an anism, which may be hard to characterize or compute in certain interesting and intuitive trade-off between incentive compatibility domains. and revenue: the metric achieves its lowest value under the most Another line of related research is on testing incentive com- aggressive form of dynamic reserves. patibility, initiated by Lahaie et al. [24]. They propose a general We demonstrate the broad applicability of our metric by drawing framework to test incentive compatibility by segmenting query on data from the auction logs of the Google Ad Exchange and the traffic into buckets and systematically perturbing buyers valuations Google search engine.