Article

Journal of 1-19 ª American Marketing Association 2020 Inefficiencies in Digital Markets Article reuse guidelines: sagepub.com/journals-permissions DOI: 10.1177/0022242920913236 journals.sagepub.com/home/jmx Brett R. Gordon, Kinshuk Jerath, Zsolt Katona, Sridhar Narayanan, Jiwoong Shin, and Kenneth C. Wilbur

Editor’s Note: This article is part of the JM-MSI Special Issue on “From Marketing Priorities to Research Agendas,” edited by John A. Deighton, Carl F. Mela, and Christine Moorman. Written by teams led by members of the inaugural class of MSI Scholars, these articles review the literature on an important marketing topic reflected in the MSI Priorities and offer an expansive research agenda for the marketing discipline. A list of articles appearing in the Special Issue can be found at http://www.ama.org/JM-MSI-2020.

Abstract Digital advertising markets are growing and attracting increased scrutiny. This article explores four market inefficiencies that remain poorly understood: ad effect measurement, frictions between and within advertising channel members, , and ad fraud. Although these topics are not unique to digital advertising, each manifests in unique ways in markets for digital ads. The authors identify relevant findings in the academic literature, recent developments in practice, and promising topics for future research.

Keywords digital advertising, ad effect measurement, advertising channel friction, ad blocking, ad fraud

Digital advertising markets offer unprecedented innovations to consumer attention and media usage: after initial growth in marketers. Businesses can now advertise to finely targeted sets desktop display and search, recent growth has been more con- of individuals with customized commercial at spe- centrated in social networking, video, audio, and mobile ads. cific locations and times in a variety of formats. Compared with However, there are indicators that unregulated markets for traditional advertising, digital ads promise better targeting and digital advertising have experienced problems. For example, relevance, personalized ad content, programmatic sales based the European Union has fined more than $9 billion in on real-time auctions, and measurement of the co-occurrence three antitrust cases, and the U.S. Federal Trade Commission of individual consumer ad exposures with a variety of online fined Facebook $5 billion after it broke a 2012 consent order and offline response behaviors. These features have fundamen- (Case 19-cv-2184). Prominent politicians have criticized the tally altered marketers’ spending: digital advertising revenues industry and proposed structural reforms. New privacy laws reached $108 billion in 2018, up 117% over 2014 ( Bureau [IAB] 2019), and they are expected to grow 19% and surpass cumulative traditional advertising rev- enues in 2019 (eMarketer 2019a). Brett R. Gordon is Associate Professor of Marketing, Kellogg School of Digital advertising markets sell a wide variety of search and Management, Northwestern University ([email protected]. display advertising opportunities to marketers. Although digital edu). Kinshuk Jerath is Associate Professor of Marketing, Columbia Business advertising has existed since 1995, market structures are still School, Columbia University ([email protected]). Zsolt Katona is Cheryl and Christian Valentine Associate Professor of Marketing and Director of the changing rapidly. For example, a census of marketing technol- Fisher Center for Business Analytics, Haas School of Business, University of ogy firms showed an increase from 150 in 2011 to 7,040 in California, Berkeley ([email protected]). Sridhar Narayanan is 2019 (Brinker 2019). The IAB Tech Lab recently introduced a Associate Professor of Marketing, Graduate School of Business, Stanford series of broad-based initiatives, including a new real-time University ([email protected]). Jiwoong Shin is Professor of Marketing, Yale School of Management, Yale University (jiwoong.shin@yale. bidding standard, and Google moved from second-price to edu), Kenneth C. Wilbur is Professor of Quantitative Marketing and Business first-price auctions for display ads. Publishers have introduced Analytics, Rady School of Management, University of California, San Diego a variety of ad formats, with spending typically following ([email protected]). 2 Journal of Marketing XX(X) mandate transparency and consent requirements for data- some of the inefficiencies we describe. We close with a broader driven advertising and user identification practices. discussion of policy-relevant research opportunities. Several comprehensive reviews of digital platform markets It is important to note that these four inefficiencies predate have advised new regulation. For example, the Australian digital advertising markets. More than 100 years ago, an exec- Competition and Consumer Commission (2019) made 23 rec- utive famously claimed that half of his ads were working but ommendations, including that “a specialist digital platforms lamented his inability to determine which half. The first news- branch be established” to proactively monitor digital markets, paper advertising agent represented advertisers but was paid enforce laws, conduct inquiries and recommend actions to commissions by publishers (Crouse 2010). Before online address consumer harm and market failure. The U.K. House advertising, ad blocking affected television ad markets via the of Lords Select Committee on Communications (2019) reached remote control, video cassette recorder, and digital video similar conclusions, noting “a lack of understanding among recorder. Numerous media outlets have fraudulently overre- policy-makers.” Similar conclusions have been reached by ported circulation numbers to increase ad prices. Traditional the European Commission Directorate-General for Competition; advertising markets developed partial solutions; for example, the U.K. Department of Digital Culture, Media & Sport; and the the Audit Bureau of Circulation verifies print publishers’ audi- U.K. Digital Competition Expert Panel; moreover, the U.S. Fed- ence data. However, all four inefficiencies manifest differently eral Trade Commission and the U.K. Competition and Markets in digital advertising markets than in traditional advertising Authority, among others, are conducting ongoing investigations. markets and therefore are likely to require different solutions. The purpose of this article is to review four prominent fea- We write for several audiences. Many aspects of digital tures that may limit digital advertising markets from reaching advertising markets remain poorly understood, so we describe their maximum allocative efficiency (i.e., the extent to which opportunities for future research. We hope these suggestions markets distribute digital advertising opportunities to the agents appeal to academics as well as the rapidly growing number of that value the opportunities most, as in Harberger 1954)1: scientists employed by large technology companies (Athey and Luca 2019). We also write for business practitioners who seek Ad effect measurement is the estimation of incremental to understand digital advertising markets, to make individual effects of advertisements on consumer behaviors. Ignor- campaigns more efficient, or to develop new ventures that can ance or uncertainty about ad effects may inefficiently enhance market efficiency. Finally, we aim to help policy mak- distort advertisers’ reservation prices, demand, budgets, ers understand root causes of inefficiencies in digital advertis- and bids for ads. ing markets. Extant investigations have given less attention to Organizational inefficiencies occur within advertising the four inefficiencies discussed subsequently than to other organizations and between advertisers and their self- issues (e.g., antitrust, safety, disinformation, consumer interested agencies; these inefficiencies may lead to sub- privacy, market transparency). We do not take a position on optimal advertising decisions. whether or how governments should regulate digital advertis- Ad blocking is a consumer technology that prevents ads ing markets, but we hope this article will help inform policy from being displayed. Ad blockers may inefficiently makers about certain key issues. appropriate advertising revenues and harm publishers’ incentives to provide content. Ad fraud is a collection of practices that misrepresent Digital Advertising Effect Measurement advertising inventory or disguise machines as humans in order to steal advertising expenditures. Most industry What Is Ad Effect Measurement? estimates indicate that fraud takes 10%–30% of total Advertising effect measurement is the process of quantifying digital . the incremental effect of an ad on consumer behavior. The incremental (or causal or marginal) effect of an ad is the incre- These features are important: 75% of brand marketers indi- mental number of outcomes obtained as a result of the cam- cated that ad effect measurement is the leading “threat to digital paign, such that these outcomes would not have occurred in the ad budgets in 2019,” and 69% of agency professionals reported absence of the campaign. that ad fraud as the leading threat (Benes 2019). We dedicate Firms measure ad effects to quantify the incremental return one section to each topic, selectively describing relevant sci- of their marketing investment—to determine whether their entific literature and credible industry knowledge and then money was well spent—and use this information to help guide identifying opportunities for future research. Although we dis- future marketing decisions. Other motivations may include cuss each issue in isolation, they may interact to exacerbate satisfying internal reporting requirements and benchmarking one campaign’s performance against others. Broadly speaking, campaign outcomes can be divided into 1 We focus on allocative efficiency because it is a classic and fundamental two groups: branding and direct response (DR; or performance). measure of market performance and is a frequent criterion economists use to recommend and evaluate government policies. We do not intend to discount Brand advertising seeks to enhance the firm’s long-term pros- other policy objectives such as consumer welfare, distributional concerns, pects by improving consumers’ perceptions, attitudes, and competitiveness, or innovation. awareness of the brand. However, quantifying these metrics, and Gordon et al. 3 linking them to profits, can be difficult. In contrast, a DR cam- according to ingrained habits or their valuation of the ad paign is about driving immediate results on a conversion metric, content (Becker and Murphy 1993; Tuchman, Nair, and such as purchases, store visits, registrations, downloads, or web- Gardete 2018). As a result, it may be difficult for adver- page visits. Our discussion of ad effects primarily involves DR tisers to distinguish incremental effects of ads from campaigns, although it is important to recognize that brand consumers’ baseline propensities to attend to the advertising represents a significant portion of online ads.2 brand’s ads. An implication is that many ad experi- In many ways, advertising measurement boils down to ments recover something closer to an intent-to-treat answering a simple question: Did the campaign work? And yet effect rather than average treatment effects or treat- advertising measurement is anything but simple. The purpose ment-on-the-treated effects. of this section is to highlight common measurement challenges, 5. The complexity of ad effects. Advertising effects may be to review measurement techniques, and to propose questions nonlinear in the number and types of ads a consumer for future research. This review is intentionally selective and sees. Marginal effects of ads vary with wear-in, wear- should not be viewed as exhaustive, as our strategy is to convey out, or weariness (Chae, Bruno, and Feinberg 2019; the challenges and methods insofar as they explain market Schmidt and Eisend 2015); competitor advertising inefficiencies and motivate suggestions for future work. (Danaher, Bonfrer, and Dhar 2008; Shapiro 2018); and ads in other media (e.g., Joo et al. 2014; Lewis and Nguyen 2015; Naik and Raman 2003). The inability Measurement Challenges to measure all of a consumer’s advertising exposures The practice of ad measurement is hardly new. However, there makes it difficult to obtain a fully accurate view of ad is a sense that digital advertising measurement presents a mix effects in many settings. of both old and new challenges. These challenges include, but The severity of these challenges varies across campaigns, are not limited to, the following: but all represent significant obstacles to quantifying ad effects 1. Measurement and data availability. Despite the volume reliably and accurately. and granularity of data available, many advertisers are Next, we discuss the two major approaches to measuring ad still unable to connect ad exposures to outcomes at the effects: designing strategies to create experimental data ex ante individual level. Among other reasons, this is due to and analyzing observational data ex post. Randomized experi- long purchase cycles, unobserved stages of consumer ments have become increasingly common as more online ad decision making, and a lack of access to distribution platforms provide such capabilities to advertisers, although channel members’ customer data. Proving the connec- usage still appears to be limited. In contrast, methods that rely tion between measurable outcomes, such as clicks or on observational data are often more accessible, and thus, mar- likes, and bottom-line metrics, such as sales, is another keters have more broadly adopted them. The next two subsec- common struggle. tions discuss the strengths and weaknesses of each approach 2. Strategic advertiser behavior. Marketers often target with brief discussions of some representative work. The final their advertising (with regard to, e.g., timing, character- subsection discusses some promising directions for future istics, location) based on expectations of future research. demand; for example, a car dealer could advertise a temporary price reduction before a holiday weekend. Ad treatment often correlates with ad response—and Experimental Studies of Advertising Effects possibly other marketing actions, such as temporary Advertising experiments are widely considered the gold stan- price reductions—creating a confounding correlation dard to estimate the effects of a marketing action on consumer between ad levels and outcomes (Barajas et al. 2016). behavior. Experiments randomly allocate units (e.g., consu- 3. Strategic platform behavior. Digital platforms optimize mers, markets) across treatment and control conditions. With their advertising delivery. For example, if a digital sufficient sample sizes, on average the only difference between advertising platform is paid per click, the platform conditions is advertising in the treatment condition. Incremen- attempts to show ads to consumers with high predicted tal conversions can be calculated by comparing outcomes click probabilities, creating another source of confound- between the advertising treatment condition to conversions in ing variation. the no-advertising control condition. 4. Strategic consumer behavior. Consumers may pay By creating exogenous variation by design, randomized attention to, or withhold attention from, advertisements experiments address a subset of the measurement problems discussed previously. A randomly allocated control group directly addresses the problem of strategic advertiser behavior 2 Advertisers sometimes rely on other metrics to evaluate ad effects, such as by inducing exogenous variation in treatment. Furthermore, “copy testing” metrics or social media engagement metrics. In display and search campaigns, some advertisers focus on the number of ad clicks, even most online ad platforms implement experiments in a balanced though that measure is directly proportional to total campaign expenditure and manner across conditions so as to neutralize the issue of stra- may come at the expense of free clicks on . tegic platform behavior. 4 Journal of Marketing XX(X)

The academic literature increasingly uses experiments to implicit assumption being that this will lead to increased adver- understand advertising effects, including influential collabora- tising spend on the platform. A nonexhaustive list of platforms tions between academics and firms. The pioneering work that offer experimentation tools includes AdRoll, Facebook, includes a collection of papers by Randall Lewis, David Reiley, Google, JD.com, MediaMath, Microsoft, and YouTube.4 and Justin Rao, who explored the effectiveness of digital dis- Although these tools are increasingly popular, widespread play advertising on Yahoo.com by randomizing the ad shown adoption by advertisers may take some time.5 to different visitors (Lewis, Rao, and Reiley 2011; Lewis and Simonov and Rao (2019) provide some systematic public Reiley 2014).3 This work was among the first to make clear evidence about advertisers’ usage of experiments. The authors both the benefits and difficulties of relying on large-scale examine online retailers’ search advertising expenditures at online experiments for the purpose of measuring ad effects. Bing.com during the period when the results from Blake, Lewis, Rao, and Reiley (2015) provide an excellent review Nosko, and Tadelis (2015) were publicized in the trade press of the early academic literature on measuring digital advertis- and popular media. Simonov and Rao (2019) found that 11% of ing effects. firms whose branded keyword ads were not regularly pur- Sometimes the unit of randomization is at the geographic chased by competitors (similar to eBay) discontinued brand market level, rather than the individual consumer. In these search advertising. However, the incidence of experimental cases, researchers have often relied on quasi-experiments or advertising variation was essentially unchanged and was uncor- matched market tests to generate suitable treatment and control related with the size of individual firms’ advertising effects. In groups (Kalyanam et al. 2017; Vaver and Koehler, 2011). summary, some firms reduced their ad spend on their own Blake, Nosko, and Tadelis (2015) document a particularly branded keywords after the eBay study results were publicized; influential geo-experiment, in which eBay halted Google but it appears that they did so without running their own tests search advertising in a sample of cities and continued search first. ads in other cities. Prior to the study, executives at eBay A common problem with ad experiments is that many ad believed sponsored search was effective at generating incre- effects are small and require surprisingly large sample sizes to mental sales. However, the results indicated a return on adver- achieve reasonable statistical power. Lewis and Rao (2015) tising spending (ROAS) of 63%, mainly because consumers reported the results of 25 digital display advertising field could easily substitute organic search results for eBay when the experiments with samples of 500,000 people or more. Despite sponsored links were removed. This was possible because large sample sizes, most experiments produced confidence eBay’s organic links ranked highly and competitors’ keyword intervals for ROAS wider than 100%, with the smallest confi- ads appeared infrequently. dence interval exceeding 50%. Advertising response data are eBay’s market leadership position makes it natural to won- highly variable, making it difficult to separate advertising’s der whether these results would generalize to firms with weaker influence on conversions from unobserved factors. The diffi- market positions. This observation motivated at least two sub- culty of measuring small advertising effects is a fundamental sequent studies. First, Coviello, Gneezy, and Goette (2017) problem for advertising measurement. Whereas a product’s used the same market-level research design to evaluate paid price acts like a hammer on consumer behavior, advertising search at Edmunds.com, for which organic search results do not is perhaps closer to a feather in a strong wind. enjoy the same high rank as eBay. The authors found that half Cost is another barrier. Every consumer held out of the of paid traffic was lost when branded paid search was turned treatment group is a consumer who does not see the ad. If the off. Second, Simonov, Nosko, and Rao (2018) also found that ad is profitable, the firm may be unwilling to bear the oppor- the result did not hold similarly for most other online retailers, tunity cost of not treating consumers in the control group. as competitors could poach branded keyword searchers when Compounding this problem, some digital ad experiments serve the focal brand does not purchase ads on its own keywords. These two studies suggest that the findings in Blake, Nosko, and Tadelis (2015) could be limited to companies with a similar 4 Google: See Johnson, Lewis, and Nubbemeyer (2017a) and https://www. market position as eBay, and they highlight the challenges in thinkwithgoogle.com/intl/en-gb/marketing-resources/data-measurement/a- generalizing results from any single experiment to other revolution-in-measuring-ad-effectiveness/. Facebook: See Gordon et al. (2019) and https://www.facebook.com/business/help/552097218528551?helpref¼uf_ settings. permalink. Microsoft: https://about.ads.microsoft.com/en-us/blog/post/july- Major digital platforms increasingly offer tools for adverti- 2019/experiments-test-your-campaign-changes-with-confidence. YouTube: sers to run experiments to measure incrementality on their plat- https://www.blog.google/products/ads/new-tools-creative-storytelling- forms. These tools are usually offered freely, though /. AdRoll: AdRoll (2019). MediaMath: Chalasani et al. (2017). sometimes only to sufficiently large advertisers, with the goal JD.com: Du et al. (2019). 5 Platform- or business-specific factors may complicate an advertiser’s ability of helping them improve their advertising outcomes—the to align the conditions in an experiment with regular, “business as usual,” advertising conditions. For example, an advertiser may need to set up multiple campaigns in the experimental platform to test different conditions. 3 Field experiments in marketing go back more than 30 years (e.g., in direct Each campaign may make differential use of the advertiser’s account history, mail [Bawa and Shoemaker 1987; Chapman 1986]; television advertising which could alter the campaign’s performance and affect the external validity [Eastlack and Rao 1989; Lodish et al. 1995]). of the experiment’s results. Gordon et al. 5 consumers in the control group with public service announce- sufficient number of experiments to trace out ad effects across ments at the advertiser’s expense. When ad effects are spending levels and media; in contrast, an MMM relies only on unknown, one might reasonably regard experiment costs as historical data. Second, the firm does not bear any opportunity either positive or negative, depending on one’s prior about the costs of holding back advertising from a control group. Third, ad effects. firms are more often capable of satisfying the technical and Lower costs of experimentation could increase the number data requirements of MMMs. Fourth, MMMs can help socia- of ad experiments that are run (e.g., Schwartz, Bradlow, and lize the overall concept of advertising measurement within the Fader, 2017). Johnson, Lewis, and Nubbemeyer (2017a) pro- organization, which can help build internal support for more posed “ghost ads” as a method to “identify ads in the control rigorous measurement practices in the future. group that would have been the focal advertiser’s ads had the In the absence of exogenous variation, these methods rely on consumer been in the treatment group.” Google implemented the assumption that they produce valid incremental effects. ghost ads and has reduced experimentation costs by an order of However, given the challenges of advertising measurement magnitude. Johnson, Lewis, and Nubbemeyer (2017b) present raised earlier, we should expect that some of the variation in a meta-study of 432 field experiments at Google that used the advertising levels is not independent of the error term, as firms ghost ads design. Ghost ads appears to be gaining traction often divide advertising budgets across time, markets, and within the advertising industry (Nanigans 2019). products in expectation of future demand. Furthermore, ad Sahni, Narayanan, and Kalyanam (2019) propose another spending in different media tends to be highly correlated, approach to lowering ad experiment costs in the context of which makes it difficult to isolate channel-specific ad effects. retargeting (i.e., targeting ads to previous visitors). To be clear, these concerns represent significant hurdles to the They use a single-impression public service announcement potential causal interpretation of effects obtained from obser- campaign to tag all consumers who were eligible for retarget- vational data. That said, in many situations this is the most ing ads. This created a single control group, offering a valid feasible path advertisers can take, so many make do with these baseline to estimate treatment effects of multiple varying imperfect ad effect estimates to inform their decision making. advertising intensities. In contrast to settings with aggregate data, another common As low-cost experimentation technologies continue to be paradigm is to use individual-level data (or cookie-level data) developed and gain wider acceptance, we expect advertisers to measure advertising effects. In these cases, advertisers often to increasingly adopt—and perhaps even demand—experimen- apply multi-touch attribution models, which are bottom-up tation services on other advertising platforms. approaches that seek to assign credit (to attribute) to each ad that preceded a user’s conversion.6 For example, suppose that a Observational Studies of Advertising Effects user clicks on a sponsored search ad, then clicks on a display ad, and then makes buys the advertised product online. To what Most firms either do not or cannot measure ad effects using extent did either ad contribute to generating the conversion? experimental or quasi-experimental methods. Instead, they rely The most common approach is a last-click attribution model on observational data collected in the normal course of busi- that assumes that the display ad is solely responsible for the ness. In such cases, academics and firms have developed many conversion, because it occurred last. Many variations on this techniques to estimate ad effects from observational field data. style of model exist, and a survey indicates their usage is wide- Probably the most common approach among firms is to spread (Forrester Consulting 2014). analyze the market-level relationship between sales and adver- A related measurement approach using individual-level data tising. This top-down approach is known as a marketing mix compares the conversion rates of users who were exposed to an model (MMM), or similarly a Media Mix Model, which dates ad campaign with users who were not exposed (e.g., Abraham back to at least the 1960s (Borden 1964). The model typically 2008; often called “lift”). Lift methods rely on consumer char- takes the form of a time-series or panel regression of aggregate acteristics data and model the joint likelihood of exposure and sales on aggregate marketing spending, or impressions, in each conversion to estimate the ad effect. The hope is that, after advertising medium. Additional controls include factors such controlling for enough characteristics, the two groups are sim- as macroeconomic conditions, weather, seasonality, other mar- ilar to the point where the ad effect can be given a causal keting mix variables (e.g., price), and competitor activity. Once interpretation. However, because exposure is influenced by estimated, the model can be used to measure ROAS and to advertiser and platform targeting strategies, conversion differ- forecast sales at various levels of marketing spending in differ- ences between groups may be caused by strategic behavior or ent channels. unobserved characteristics, even in the absence of advertising. Chan and Perry (2017) present an excellent discussion of the Gordon et al. (2019) demonstrate the difficulty of using uses of MMMs, highlighting pitfalls and identifying opportu- observational data to estimate valid incremental ad effects. nities for improvement. There are several arguments in favor of They reanalyzed 15 Facebook ad experiments comprising 1.6 MMMs. First, the scope at which they operate—aggregate spending—allows MMMs to play a key role in a firm’s budget- ing process, helping the firm divide its market budget across 6 See Abhishek, Despotakis, and Ravi (2017) and Berman (2018) for media. It would be difficult for many firms to implement a theoretical analyses of multitouch attribution. 6 Journal of Marketing XX(X) billion ads served to 500 million users. Despite having rich Future Research Opportunities data, they were unable to recover true ad effects accurately Measuring the returns to all of a firm’s digital advertising or consistently using observational methods. Even the sign of expenditures is a daunting task, especially if the goal is to do the bias was unpredictable: most ad effects were overestimated, so continuously and to immediately incorporate insights into but some were underestimated. As the authors put it, “even the next advertising decision. Competent, expensive, large- good data prove inadequate to yield reliable estimates of adver- scale attempts to estimate ad effects have failed to measure tising effects.” effects with reasonable precision. Following are some ques- Fortunately, it is possible to use observational data to tions that researchers could consider for the future and some address some of the advertising measurement problems. In brief thoughts on how to make progress on them: general, quasi-experimental approaches make use of specific information about the timing of events or the data-generating process to lend a causal interpretation to the estimates and may 1. Advertisers can improve the usefulness of observational require additional assumptions. Causality is not free. models. One strategy is to practice continuous experi- One such approach is found in Nair et al. (2017). The data mentation across geographies, media, and groups of they analyzed came from a casino that had previously targeted consumers, building more random variation into the promotions to consumers based on their observed gaming data (Zantedeschi, Feit, and Bradlow 2017). For exam- behaviors. The authors exploited knowledge of the firm’s pre- ple, some digital advertising platforms offer targeting at vious targeting policy to obtain unbiased estimates of consumer the zip code level, which could enable greater statistical response to promotions. Using the estimates to segment con- power than traditional market-level randomizations. It sumers, they show, via a field experiment, that the segmenta- would be useful to understand how firms could effi- ciently design this experimentation and how to incor- tion scheme contributed to a targeting policy that generated porate it into existing models. higher incremental profits than alternate policies. 2. Firms can produce and leverage larger panel data sets to Another strategy is to take advantage of local randomization improve statistical power. Most advertising experi- with respect to time or some other variable. Liaukonyte, Teix- ments are implemented in traditional static split/test eira, and Wilbur (2015) measured changes in brand website designs. However, large panel data sets may offer traffic and conversions in narrow two-minute windows around increased statistical power (Berman and Feit 2018). For the airing times of television advertisements. They based the example, if the timing of advertising treatment can be assumption of quasi-random advertisement treatment times on randomized, treatment/control comparisons could a detailed understanding of television networks’ sales prac- potentially be made within individual consumers, tices, which resulted in quasi-random ordering of advertise- thereby removing confounding individual fixed effects. ments within commercial breaks. This helped ensure that A few firms can track both ad exposures and conver- systematic differences between traffic and conversions in sions in single-source customer data (e.g., Apple, pre-ad and post-ad windows of time could be attributable solely AT&T, Google, Verizon). Market research firms and to the presence of the TV ads. Narayanan and Kalyanam (2015) internet service providers are probably best positioned studied position effects in search advertisements by using local to create single-source panel data that measure both ad randomness in position when competing advertisers were very exposures and ad responses for a large number of close to the threshold of winning or losing the bid for a partic- , although doing so would require compliance ular position. This local randomness allows for any differences with data privacy rules. Concerns may also remain in subsequent behaviors by consumers to be attributed to the about endogenous person-/time-based targeting if ads position the ad was in. are delivered nonrandomly. Finally, advertising exposure may be influenced by factors 3. Advertising agencies could develop approaches that that are arguably independent of consumers’ preferences for integrate ad effect measurement with advertising pro- the brands being advertised. Hartmann and Klapper (2018) curement. One such approach, described in Lewis and implement this strategy, using quasi-random geographic varia- Wong (2018) and implemented by an agency called tion in exposure to ads during the Super Bowl to study the Nanigans, involves manipulating advertisement bids effects of ads in the beer and soda categories. Probability of to induce exogenous variation in advertisement selec- exposure to Super Bowl ads depends on the performance of a tion. Waisman et al. (2019) addressed a similar problem consumer’s local team. Super Bowl ads are planned at the at JD.com by using a multi-armed bandit approach to national level many months before the event, so advertising combine continuous experimentation with optimal levels should be independent of shocks to local audience sizes. exploitation of experimental findings. These examples demonstrate that advertisers can measure 4. It may be possible to develop new quasi-experimental some ad campaign effects without relying on randomized designs for digital advertising. A variety of such strate- experiments. However, extra work is required to identify these gies have been used in other media. For example, tele- situations, and they may not cover all the scenarios relevant to vision ad effects have recently been estimated using advertising decision making. such instrumental variables as advertising treatment Gordon et al. 7

discontinuities at geographic television market borders 8. As more platforms offer experimentation as a service (Shapiro 2018), exogenous shocks to TV ad demand dur- and more advertisers take advantage of this capability, ing elections (Sinkinson and Starc 2019), exogenous TV it is unclear how marketers should interpret the treat- ad insertion timing (e.g., Du, Xu, and Wilbur 2019, among ment effects obtained when these effects depend on others), and assignments of TV networks to local cable competitors’ advertising policies. An experiment deli- channel numbers (Biswas, Dube, and Simonov 2019). It vers the ad effect that is incremental conditional on all may be possible to find and exploit similar exogenous other marketing activities in the market. Lin et al. discontinuities in digital advertising markets to measure (2019) provide a framework for interpreting ad effects causal ad effects in observational data. For example, Hill obtained under parallel experimentation. Future work et al. (2015) show that, under particular assumptions, var- could provide guidance on how to best generalize iation in ad viewability can help measure ad effects, con- experimental ad effects, which may require the adver- sidering that approximately 45% of ads rendered never tiser to form beliefs about competitors’ future policies. appear in the viewable portion of a user’s screen. 5. Firms may be able to integrate the results from observa- In summary, advertising effect measurement can increase tional models and randomized experiments. Some firms advertising profits, but good solutions are often unavailable. use experiments for tactical decisions (e.g., does this new Ad effects are typically small, and conversions are highly vari- ad design outperform the previous design?), whereas able, so conclusive experiments are costly and rare. Observa- they rely on MMMs for strategic decisions (e.g., how tional data are plentiful, but statistical analyses often do not much budget should we allocate to search versus dis- uncover causal effects because advertisements are allocated play?). The information contained in the experiment strategically in ways that correlate with treatment. Although could help inform the aggregate model, but it is not as many people may share the intuition that it should be possible simple as feeding the results of the experiment into the to design large-scale studies to systematically estimate ad MMM. One possibility is that effects from experiments effects, it is not yet clear when that goal will be achieved. could be imposed as informative priors when estimating the MMM. Whatever the solution, many marketing mea- surement firms are actively searching for a unified mea- Organizational Frictions and Inefficiencies surement model to help deliver on this broad goal. Classical economic theories of market efficiency assume that 6. Managers may be able to embed experimentation purchasers allocate budgets to maximize their own welfare. within their profit-maximizing objective. The estima- However, markets for digital ads are intermediated by specialists tion of causal effects produces an input—the ad whose self-interested actions may distort advertising decisions. effect—intended to aid a manager’s decision-making Principal/agent problems and moral hazard are well-known process. Ideally, the decision to experiment and to inte- sources of economic inefficiency. Generally speaking, asym- grate experimental findings into profit maximization metric information enables agents to extract inefficient informa- can help the manager to make use of the results for tion rents from principals (Laffont and Martimort 2001). Perhaps subsequent marketing decisions. Feit and Berman the best-known example is the dead-weight loss due to double- (2019) provide a recent example of such a framing. marginalization in , which results from the diver- 7. Digital ad sellers may require techniques to validate gence of manufacturer and retailer interests in a distribution their experimentation platforms. An increasing number channel. However, contracting problems have received limited of platforms offer experimentation services to help attention in the specific context of digital advertising markets. measure ad effects. Experimentation on ad platforms For example, many marketers ask digital advertising agencies to typically allows the advertiser to achieve greater scale, evaluate their own performance; under what conditions and con- and the platform’s control of the experiment should tracts are agencies incentivized to report truthfully? help it develop higher-quality measurement technology. We discuss two types of organizational inefficiencies arising However, ad platforms also face a possible credibility in the digital advertising ecosystem. Intrafirm inefficiencies may problem, as advertisers may worry that the platforms result from misalignments between corporate officers, functional 7 have an incentive to inflate ad effectiveness estimates. departments, or business units within a firm due to different To date, we are unaware of any systematic external incentives or strategic objectives. Interfirm inefficiencies may audits or validations of the experimental tools that plat- occur between marketing organizations and their external agen- forms provide to advertisers. cies, between competing agencies serving the same marketing client, between complementors within a value chain, or between colluding purchasers and sellers of advertising.

7 There are also challenges and conflicting incentives if platforms try to estimate interactions between ads on competing platforms. For example, it Intrafirm Inefficiencies would be difficult for Google to credibly offer advertisers a tool to estimate interactions between and Facebook ads (see, e.g., https:// Chief financial officers often set marketing budgets and review marketingland.com/where-is-google-attribution-256098). marketing financial performance, a common source of tension 8 Journal of Marketing XX(X) within a firm (McKinsey 2013). Many finance executives display ad impression: publisher, ad server, supply-side plat- believe that marketing executives should be able to measure form, ad network, ad exchange, demand-side platform, multi- return on advertising spend (Fitz 2018). As explained previ- ple data management platforms, third-party verifiers, ad ously, such beliefs may be misplaced. Misalignment of internal agency, and, finally, the advertiser (Choi et al. 2019). Each beliefs and incentives may distort marketing tactics. Marketing agent takes a cut of the advertising transaction. The Association may be perceived as a short-run lever that can be used to meet of National Advertisers (ANA; 2017) concludes that “58 cents particular financial targets. Marketing managers may hire out- of each dollar ultimately purchased media inventory and audi- side consultants to shift blame away from themselves in the ence exposure from a publisher, with 42 cents of programmatic event of poor or unmeasurable outcomes. For example, they investment consumed by supply chain data and transaction may use retargeting campaigns, which send advertisements to fees.” Therefore, the share of digital ad spend going to inter- shoppers who previously viewed the firm’s website and there- mediaries is nearly triple the traditional 15% agency commis- fore have a higher expected baseline propensity to purchase. sion, although it is split between many more entities in the This creates a positive correlation between ad exposure and programmatic advertising supply chain. conversion. Lacking better causal metrics, managers may In addition to the programmatic marketplace, advertisers incorrectly present inflated lift metrics derived from correla- can purchase guaranteed inventory. In such transactions, the tions (Lambrecht and Tucker 2013; Sahni, Narayanan, and advertiser specifies the parameters of the purchase (e.g., tar- Kalyanam 2019). geted demographics, number of ads to be delivered, time Another potentially unintended consequence of an internal frame) and the agency quotes a price. However, K2 Intelli- desire to measure ad effects is a distortion between short-run gence (2016) documented widespread obfuscation of agency and long-run objectives. It is seldom possible to reliably esti- markups in such transactions. Arbitrage may reduce market mate advertising effects on long-run outcomes, such as brand efficiency due to moral hazard and asymmetric information attitudes (Du, Joo, and Wilbur 2019). In contrast, direct between advertiser and agency, although agencies have coun- response campaigns focus on short-run actions that are easier tered that they can add value to low-cost advertising inventory to reliably link to ad exposure. Tension between long- and by applying analytics. short-run objectives is difficult to quantify, but it is possible Another type of inefficiency may arise when advertisers fail that underinvesting in brand-building advertising could reduce to coordinate agencies whose work complements each other or long-run profits. Also pertinent in this context is the potential generates spillovers across marketing channels. As one exam- misalignment between the firm’s time horizon and that of the ple, “last-click” attribution models incentivize digital agencies manager, with the manager often taking a more short-run view to compete over credit for attributed conversions. As another than the firm as a whole. example, a large body of research has found significant spil- A third type of intrafirm inefficiency may result from mis- alignment between functional groups within the organization. lovers between advertising placements in traditional media and digital activities such as brand search, website traffic, online Previous research has shown that poor integration of marketing 8 and sales teams may lead to suboptimal advertising policies sales, and social media conversations. Few marketers are and customer focus (Homburg, Jensen, and Krohmer 2008; known to coordinate their advertising activity to account for Smith, Gopalakrishna, and Chatterjee 2006). Marketing goals such spillovers. An integrated channel would consider the may conflict with other functional groups as well, such as when effect of traditional advertising on volume of online search, the marketing objectives are misaligned with those of the pro- types of keyword searched, search advertising expenditure, and curement department. For example, procurement may use a sales, and would quantify spillovers to allocate advertising reverse auction to award a contract to an advertising agency budgets across media (Kim and Balachander 2017). However, to meet a goal of minimizing expenditures. The low-bidding the typical approach is for marketers to hire specialist agencies agency may then provide lower-quality service or fail to fully for each individual medium (e.g., traditional advertising, realize marketing objectives (Neff 2015). marketing, social networking, search engine Finally, distinct business units, brands or campaigns within optimization, website analytics), each of whom operates inde- the same firm may enter rivalrous bids on the same advertising pendently and competes for a larger share of the marketer’s inventory. For example, Narayanan and Kalyanam (2015) advertising budget. show that after a large retailer acquired three of its rivals, the Another type of inefficiency related to advertising agencies four brands continued to compete with each other by entering is how to properly align incentives to produce efficient adver- rivalrous bids in advertising keyword auctions. tising creative content. On one hand, advertising agencies’ creative ideas can be stolen during the pitch process, especially Interfirm Inefficiencies 8 Many digital advertisers have replaced the traditional agency- See, for example, Dinner et al. (2014), Fossen and Schweidel (2017), Joo et al. (2014), Lewis and Reiley (2013), Liaukonyte, Teixeira, and Wilbur of-record model with a complex array of external agencies. (2015), Du, Xu, and Wilbur (2019), among others, for effects of TV ads on When a consumer requests a web page, any subset of the fol- digital behaviors. Lewis and Nguyen (2015) document spillovers from digital lowing players may be involved in delivering each digital display advertising to search behaviors. Gordon et al. 9 when clients work with multiple agencies (Dan 2014). There- collusive bidding strategies and punishments for defection. fore, some agencies may not reveal the full idea during the Decarolis and Rovigatti (2019) found that advertising agency contracting process, potentially leading to adverse selection consolidation is associated with decreased costs in keyword or underinvestment in idea production (Horsky, Horsky, and auctions when merging agencies represent competing bidders. Zeithammer 2016). On the other hand, advertisers have diffi- There has been a recent trend toward global consolidation of culty evaluating the quality of agencies’ creative strategies due advertising agencies (Pathak 2018), suggesting that such to the complexity and high dimensionality of the messaging effects may be considerable. problem. Although messaging content is still largely driven by human intuition, there is increasing movement toward algorith- mic production and validation of advertising messages in digi- tal environments. Future Research Opportunities Three recent trends affect the market for advertising agency We suggest several important research opportunities within services. First, a long-term trend has emerged toward this area. in-housing, in which marketers create internal advertising agencies. In-housing is used primarily when firms have internal 1. Recent theoretical work in information economics has creative abilities or straightforward advertising requirements introduced Bayesian persuasion models and the analy- (Horsky 2006). A recent ANA (2018) survey of large adver- sis of strategic information design (Kamenica and Gen- tisers shows that 78% of advertisers had in-house agencies in tzkow 2011, Bergemann and Morris 2016). Bayesian 2018, up from 42% in 2008, and that 90% of those advertisers persuasion models characterize optimal informational continue to work with external agencies in addition to their strategies (i.e., what message to send, to whom, and internal teams. Second, digital platforms have simplified their when or how much information to reveal) to influence customer interfaces, a move that may encourage some adver- receivers’ decisions. This seems to be a promising the- tisers to forgo agency services. For example, Google recently oretical framework on which to build an understanding made “smart display campaigns” the default interface to Goo- of optimal advertising strategies for advertising content, gle Ads. The streamlined purchase process relieves the adver- targeting, and platform design (Fuchs, O¨ ry, and Skrzy- tiser of control over bidding, ad placement, user targeting and pacz 2015; Mayzlin and Shin 2011; Zhong 2018). final control over the ad creative. However, advertisers using 2. Principal/agent– and theory of the firm–style analyses this interface may pay advertising costs that exceed advertising have rarely been applied to advertising agencies. It may revenues, because they no longer can enter a maximum bid per be that such settings can be understood with straightfor- click. Third, there is an emerging trend toward large advertisers ward applications of canonical models. However, given requesting log-level (i.e., impression-level data) from sup- the particular institutional details, we suspect there is ply chain partners such as advertising exchanges and publishers substantial scope to extend existing theories in the pro- (Sluis 2019). Data sharing enables advertisers to monitor part- cess of applying them to advertising phenomena. Future ner agencies and suppliers, trim wasteful spending, and find research could include normative guidelines about how undervalued inventory. optimal contract terms depend on features of a market Another type of interfirm inefficiency may occur when and the contracting parties. Contract terms likely affect advertisers compete in advertising auctions with complemen- incentive alignment, effort elicitation, payment models, tors within the same value chain. For example, Hilton and in-housing/outsourcing decisions, and agency selection Expedia both sell Hilton rooms in , and Expedia and coordination. takes a commission when it sells a Hilton room; yet both place 3. There are opportunities to further develop and improve search ads on keywords like “hotel New York.” Cooperative techniques to optimize digital advertising content and advertising policies may even exacerbate this competition: Hil- context. Machine learning has demonstrated progress in ton may subsidize the advertising expenditures of Expedia, identifying sentiments (Cambria et al. 2017) across a even though this directly increases Hilton’s keyword advertis- wide variety of unstructured data including text (e.g., ing costs by making the keyword auctions more competitive. Netzer et al. 2012), images (e.g., Liu, Dzyabura, and Cao and Ke (2019) and Jerath, Ke, and Long (2018) examine Mizik 2018), and video (e.g., P´erez-Rosas et al. 2013). such practices. The meaning of advertising content depends on the Finally, there are inefficiencies that may increase adverti- context (Poria et al. 2017, Rafieian and Yoganarasim- sers’ profits at the expense of overall market efficiency. One han 2019), such that algorithms are needed to detect and such effect may occur when competing advertisers use a com- appropriately understand the context and work with mon advertising agency, as doing so may coordinate marketing multimodal data (Soleymani et al. 2017). Similar policies or facilitate information sharing (Villas-Boas 1994). opportunities may be available when a marketer can Another may occur when the structure of programmatic adver- access log files from multiple partners, as the data may tising marketplaces facilitates collusion between purchasers. be used to optimize advertising frequency, quantify the Digital advertisements are sold in high-frequency auctions with impact of context on conversion, optimize procurement repeated contact among bidders, creating ripe conditions for techniques, or evaluate ad agency services. 10 Journal of Marketing XX(X)

4. Perhaps the most promising area for research involves detect ad blocker usage and withhold content from consumers designing auction and advertising allocation systems who do not allow ads; Zhu et al. (2018) found that 30% of the that more effectively align the incentives of the parties top 10,000 detect ad blockers. Some publishers, involved (e.g., Hu, Shin and Tang 2016). Johnson and including Facebook, invest heavily in website engineering to Lewis (2015) propose a cost-per-incremental-action circumvent ad blockers and unblock ads. Other publishers pricing model for advertising, with the goal of aligning explicitly request that users whitelist the site (i.e., selectively all participants’ objectives on profit maximization for disable their ad blocker), so the site’s ads may be displayed. the advertiser, thereby reducing the misaligned incen- Software developers have responded with “ad blocker-blocker- tives common in advertising pricing models. Additional blockers,” which enables the consumer’s browser to obfuscate research is needed to test such models in the field and to ad blocker usage, thereby preventing publishers from withhold- adapt them to various digital advertising markets. ing content. The overall effect of ad blocking on advertisers, consumers, and publishers is unclear. Using the circulation spiral theory of Ad Blocking newspapers (e.g., Gabzewicz, Garella, and Sonnac 2007), one Description and Background might predict that ad blockers will put advertising-supported media out of business. By reducing ad revenues, ad blockers Early digital advertising efforts developed intrusive formats increase direct-access prices and thereby reduce publishers’ such as pop-up ads or autoplaying audio/video ads. This led incentives to produce high-quality content. Taken to its to consumer demand for ad blockers, applications that allow extreme, ad blockers would destroy the ad-supported internet users to passively block advertising from showing up in their and thereby become irrelevant, as there would be no more ads browsers. Most ad blockers came in the form of free-to-use to block. A contrary argument holds that ad blocking serves browser extensions enforcing a set of community-defined rules consumer interests: some ad blockers even position themselves for ads. Recent estimates of ad blocking prevalence vary from as a consumer movement (Katona and Sarvary 2018). Ad 8% to 47% (Searls 2019). In mid-2019, Google incorporated ad blockers say they only block the most intrusive ads and that blocking features into its market-leading Chrome browser, users can whitelist any site they want. In fact, ad blockers while continuing to enable third-party ad blockers. whitelist some sites by default. The phenomenon of ad blocking highlights an important This brings us to an important question: How do the ad tension in the current digital advertising ecosystem. Consumers blockers make money? Most ad blockers do not directly charge are flooded by ads, most of which are not interesting or rele- users for downloads or ad blocking services. The typical model vant. Prior to ad blocking technology, consumers had no choice ad blockers use is to demand payment from publishers in but to tolerate the ads; most users were not willing to pay for exchange for whitelisting-by-default. If the publisher pays and most content, so content providers relied on advertising reven- conforms to some guidelines about ad formats and intrusive- ues. Ad blocking threatens this model, as advertisers’ most ness, the ads will be displayed to ad block users by default. sought-after consumer segments are often the most likely to Large publishers typically pay 30% of ad revenue that would install ad blockers. otherwise be blocked; small publishers can be whitelisted for The most popular ad blockers work by intercepting browser free if they agree to follow programs such as the “acceptable requests to lists of known ad servers, so advertisers are not ads” standard. charged for blocked ads. However, some ad blockers are more Publishers have argued that such practices amount to extor- aggressive: they only block ads after they are requested, tion, and these allegations have resulted in numerous lawsuits. thereby wasting the advertiser’s money. One ad blocker, Some of the most important cases have taken place in Germany AdNauseam, even clicks on all blocked ads, in order to inject where the company that makes Adblock Plus, Eyeo, is based. noise into the user data that underpins digital advertising. The German Supreme Court has ruled that ad blocking and the Platforms such as web browsers and mobile operating sys- practice of soliciting payment for whitelisting is legal (Ha tems enable or restrict ad blocking services. However, plat- 2018). forms that sell ads have conflicting incentives. Their revenues depend on consumer experience but also advertising exposures. It is thus not surprising that critics argue that Google Future Research Opportunities blocks relatively few ads and interferes with competing ad The phenomenon of ad blocking raises a range of future blockers. Mobile operating systems assert more control over research opportunities. applications than traditional desktop operating systems. Apple and Google have rules about apps that interfere with how other 1. It would be worthwhile to study how consumers trade apps work, making it difficult to offer blocking of in-app ads. off the intrinsic benefits of content consumption with Ad-supported publishers are perhaps most harmed by ad advertising quantity, advertising nuisance, and sub- blocking. Facebook and some other sites deliver ads and con- scription prices. Huang, Reiley and Riabov (2018) ran tent from the same servers, preventing ad blockers from iden- experiments that showed that increased advertising on tifying which page elements are ads. Other publishers seek to Pandora.com directly decreased consumers’ site usage Gordon et al. 11

and increased consumer subscriptions. It would be (Dukes and Gal-Or 2003) or increases the effects of interesting to measure similar effects in other contexts; ads that are not blocked. Future research could dig to condition on the nuisance level of the ads served; and deeper to examine how other dimensions of advertis- to estimate effects of ads on adoption dates of ad block- ingcontentareaffectedbyadblocking.Ifabrandhas ers, among other relevant outcome variables. It also to pay more to reach a consumer, the brand might would be interesting to study how design elements change how much it focuses on communicating a sin- within publisher requests for whitelisting might corre- gle piece of information (e.g., price, product feature) late with response metrics. or alter the mix of entertaining and informative content 2. There may be better ways for browsers and ad blockers contained in the ad. to improve mechanisms to allocate attention. Although 6. Platforms’ business models may incentivize them to set browsers that specialize in privacy and ad blocking ad blocking defaults or limits in mobile operating sys- have not realized high market shares, some notable tems and desktop browser software. Overall, it seems experiments are ongoing. For example, the browser likely that platforms selling ads will restrict ad blocking implements the “Basic Attention Token” to more than platforms that charge consumers directly for reward users for looking at ads and send money to devices and software, in order to limit harm to adver- publishers the user likes. New designs might take hints tising revenues. available from recent literature on privacy, which shows that consumer valuations of privacy respond Digital Advertising Fraud strongly to framing and context (Acquisti, John, and Loewenstein 2013, Winegar and Sunstein 2019). Description and Background 3. Recent work by Shiller, Waldfogel, and Ryan (2018) Digital advertising fraud is a collection of practices that mis- shows that, as the proportion of a site’s visitors who use represent advertising inventory or disguise machines as ad blockers increases, the site’s quality declines. Fur- humans to steal advertising budgets. It is fundamentally diffi- ther research is needed. How does a site’s revenue loss cult or impossible to measure, but it appears to be widespread.9 from ad blocking compare with payments to ad block- eMarketer (2019b) reported that recent estimates of fraud vary ers? When do sites switch from ads to paid subscription from $6.5 to $19 billion, and that fraudsters target high-price ad models, unblockable “,” or other impressions, new markets, and new media. Adobe reported in blocking-proof business models? Similar questions can 2018 that about 28% of website traffic showed “strong non- be asked for advertisers, especially those whose target human signals.” The IAB Tech Lab reported that “just 59.8% customer segments are most likely to use ad blockers. of clicks could be confirmed as human traffic” (Swant 2019). How do ad blocking and whitelisting affect ad place- Pixalate (2018) estimated that 10%–15% of programmatic ments and ad prices? desktop advertising was invalid traffic. Davies (2019) reported 4. It would be interesting to understand how ad blocking that “28% of global mobile media budgets are wasted on changes product market outcomes. A few recent theo- fraud.” Sweeney (2018) reported that 95% of marketing exec- retical pieces address the interaction between publishers utives surveyed said that digital media must become more and ad blockers (Gritckevich, Katona, and Sarvary reliable and 21% said they have cut ad spend due to inaccurate, 2019) and differentiation between publishers (Despota- questionable, or false reporting. kis et al. 2017), but there are more directions to explore. We know of six basic motivations to commit advertising For example, how do ad blockers compete with each fraud. other? Blocking more ads than a rival ad blocker may attract more consumers, and it may also impact the 1. Publishers may overreport or misrepresent audience value to publishers of paying to be whitelisted by metrics to increase ad revenues. default. It would be fruitful to quantify these trade- 2. Advertisers may click competitors’ ads to harm rivals’ offs, as they likely rely on the extent of consumer multi- market inferences, ad budget, or brand awareness, or to homing, both across publishers and across ad blockers. reduce competition in ad auctions by depleting a rival’s 5. There is also the question of how advertisers react to ad budget. blocking. Chen and Liu (2019) argue that ad blocking 3. A firm may commit detectable advertising fraud that can incentivize higher quality ads when ads send infor- ostensibly benefits a rival, with the goal of inducing a mative signals. Other fundamental questions should platform or other intermediary to punish the rival. also be addressed. If ad blocking reduces the overall supply of advertising space, this necessarily raises the overall price of reaching customers. The increase 9 Skepticism is appropriate when interpreting fraud measurements. Fraud is could be disproportionate for more ad-averse consu- defined by intention, which is not directly observable in ad data. Indirect measures of ad fraud trade off subjective risks of false positive observations mers. How does that affect advertisers? Intuitively, the of fraud against false negative detections of valid ad exposures. Fraud detection direct effect should be negative, but it is possible that firms might overreport fraud to attract business, whereas ad sellers might limited advertising also reduces price competition underreport fraud to reassure clients. 12 Journal of Marketing XX(X)

4. Advertising market intermediaries, including agencies, return those funds to defrauded advertisers and that Google’s ad networks, and demand- and supply-side platforms, advertiser contracts did not require the company to offer may misrepresent the inventory or the bids they have refunds to defrauded advertisers (AdTrader 2018). Facebook available in an effort to alter ad prices, ad sales, or states that “we believe fake accounts are measured correctly commissions. They also may deviate from contracts to within the limitations to our measurement systems.” However, manipulate ad auctions, as in the recent bid caching Facebook does not specifically define fake accounts, although scandal (Davies 2018). its community standards do prohibit “accounts that are fake” 5. Publishers or agencies may distribute advertisements sur- without elaboration. Facebook’s Community Standards reptitiously or insert false information into advertisers’ Enforcement Reports show that Facebook “took action on” conversion tracking systems to claim undue credit from 6.6 billion fake accounts in the 12 months ending in September advertisers who pay publishers per conversion achieved. 2019 but did not specify the actions taken. Facebook has 6. Firms or individuals may create fraudulent profiles on restated advertising metrics numerous times (Peterson 2017). social networks to falsify measures of influence, ad There are two types of efforts to address digital advertising clicks, or seemingly organic discussions (astroturfing). fraud. The first increases market transparency and accountabil- ity to reduce the incidence of advertising supply chain partici- Collectively, there are dozens of known schemes to commit ad pants from stealing from each other. For example, the IAB fraud. Tech Lab defined the ads.txt standard in 2017 to enable pub- Fraud techniques that use machines to mimic human beha- lishers to publicly identify authorized advertising sellers and vior are often enabled by “botnets,” which consist of a central resellers, so that buyers can audit ads.txt files to avoid server and a host of -infected computers. The server unauthorized sellers. Publisher adoption is widespread. Newer directs connected machines to take specific actions in ways that standards ads.cert, sellers.json, and the OpenRTB Supply resemble their owners’ organic behaviors, making this machine Chain Object enable similar disclosures by other market parti- activity difficult to distinguish from human activity. cipants. Anecdotally, it appears that these efforts have mean- Security researchers have publicized several ad hoc botnet ingfully reduced fraud borne by supply chain participants. discoveries. For example, the discovery of the 3ve botnet in The other type of antifraud effort helps advertisers avoid 2018 led to the first criminal charges filed for advertising fraud. paying for fraudulent ads ex ante and to seek reimbursement The indictment stated that “more than 1.7 million infected for fraud detected ex post. Most importantly, there are a range computers ...download[ed] fabricated webpages and load[ed] of firms that specialize in ad fraud detection, including some ads.”10 that partner with large digital advertising platforms to directly Another set of fraud techniques misrepresents advertising analyze some data. There are also industry collaborations, such inventory, as several publishers have shown. The Financial as the Trustworthy Accountability Group, which publishes a Times did not sell inventory in programmatic advertising mar- monthly Data Center IP List, a “common list of [Internet Pro- ketplaces, but it found fraudulent FT.com ad inventory offered tocol] addresses with invalid traffic coming from data centers in 25 ad exchanges, with fraudulent video inventory exceeding where human traffic is not expected to originate.” In addition, truly available inventory by a factor of 30. The Guardian pur- industry-standard contractual language has been developed chased intermediaries’ Guardian.com video inventory on open with the intent of helping advertisers obtain relief for fraudu- advertising exchanges and found that 72% of ads purchased lent ads. Measurement difficulties make it challenging to know falsely claimed to occur on Guardian.com. how advertiser-facing fraud has changed over time. Advertiser Large digital platforms have been inconsistent in their concern and awareness about fraud have increased, but at the claims about digital advertising fraud. Google states in its same time, fraudsters have grown more sophisticated and dis- advertiser-facing webpages that “we protect you from invalid covery of large-scale botnets has increased. activity and advertising fraud” and describes a variety of The issues underlying digital ad fraud are nontransparency approaches. However, internal communications filed as evi- and nonverifiability of human recipients of ads. Advertising dence in a 2017 lawsuit indicated that Google recalled fraudu- delivery systems seldom require proof of humanity; for exam- lent funds from publishers but lacked an internal system to ple, publishers do not require users to solve a reCAPTCHA before serving ads. Ads are delivered when a computer requests a page from a web server, placing ad exposures in the set of 10 The indictment further stated, “To create the illusion that real human activities that are trivial for botnets to perform programmati- internet users were viewing the advertisements loaded onto these fabricated websites, the defendants programmed the datacenter servers to simulate the cally. As Vinton Cerf, co-creator of the internet protocol, put it, internet activity of human internet users: browsing the internet through a fake “We didn’t focus on how you could wreck this system browser, using a fake mouse to move around and scroll down a webpage, intentionally.” Digital ad fraud is often described as a cat- starting and stopping a video player midway, and falsely appearing to be and-mouse game in which fraudsters develop new tactics when signed into Facebook. Furthermore, the defendants ...[made] it appear that previous tactics have been neutralized. the datacenter servers were residential computers belonging to individual human internet users who were subscribed to various residential internet We emphasize the role of botnets because evasion of detec- service providers” (https://www.justice.gov/usao-edny/pr/two-international- tion is of central importance to those who would commit adver- cybercriminal-rings-dismantled-and-eight-defendants-indicted-causing). tising fraud. Fraud that can be detected can be reversed, with Gordon et al. 13 charges reverting back to the client marketers and fraudsters order to better measure when ads are being delivered to being excluded from established advertising networks. There- people as opposed to bots. fore, competent ad fraud is, by definition, fraud that is suffi- 3. It may be interesting to adapt market designs and regu- ciently unlikely to be detected. Major advertising networks say lations from other contexts. For example, securities fraud that they seek to detect advertising fraud, but they do not fully may have been affected by government regulations or explain the specific techniques they use. Fraud detectors are dispersion of trading across competing stock markets. hamstrung in gaining advertisers’ trust: sharing their precise There may also be lessons available from other examples fraud detection algorithms could help unscrupulous actors of fraud, such as insurance fraud or identity theft. Suc- avoid fraud detection. There is also a credibility problem: Wil- cessful efforts to detect, prevent, reverse, or deter frau- bur and Zhu (2009) proved that, under general conditions, dulent activity in other settings may have implications digital advertising platforms’ revenues may rise when they fail for effective management of digital advertising fraud. to detect ad fraud, even when advertisers adjust their bids based 4. It is important to understand empirical predictors and on expected fraud levels. In summary, small advertisers are impacts of fraud. If institutional ethics guidelines allow, forced to trust in partners’ efforts to detect fraud, even though researchers could simply run experiments by purchasing the partners get paid every time their detection algorithms fail. various types of botnet actions, as they are available cheaply online, and then observe how fraudulent traffic changes market outcomes. Ideally, an approach might be Future Research Opportunities developed that enables individual agencies or advertisers The academic literature on ad fraud is relatively sparse. Only a to audit platforms’, publishers’ and fraud detection firms’ few theoretical studies examine how advertising market con- claims about audiences delivered and fraud prevented. ditions and business models affect various players’ incentives 5. Another approach could use certain events as disconti- to engage in fraud (e.g., Chen et al. 2015). A more sizable nuities in quasi-experimental designs, such as privacy literature in computer science proposes algorithms to detect regulation changes, publishers’ adoption dates of particular fraud schemes (e.g., Behdad et al. 2012). Finally, ads.txt or other transparency standards, or major public some empirical papers relate fraud prevalence to other market disclosures of botnets or ad fraud schemes. Examining conditions to offer guidance to firms. For example, Edelman how market data and indicators of fraudulent activity and Brandi (2015) relate marketers’ monitoring efforts to change around event times could show how fraud online affiliate fraud. They show that monitoring by outside responds to incentives and distorts markets. specialists is more effective at punishing clear violations of We close by highlighting two hypotheses related to ad fraud marketer rules, whereas internal marketing staff are more and allocative efficiency. First, there are trade-offs between effective at punishing borderline rule violations. consumer privacy, market transparency, and fraud detection. More research is needed, particularly the following five In particular, the more signals there are about human identity areas: and activity, the more information is available to potentially 1. Theoretical analyses have seldom considered the eco- distinguish humans from machines. Therefore, it seems possi- nomic antecedents and consequences of digital ad ble that increasing consumer privacy may also hinder detection fraud. It would be interesting to model digital advertis- of ad fraud and market transparency. Second, we think the ing as a credence good as many advertisers cannot reli- application of blockchain technology is promising in digital ably measure ad effects. Two open questions are how advertising markets. However, widespread adoption of new credence goods markets function when delivery of the standards requires coordinated action. Will the walled gardens service is only partially verifiable and how optimal con- adopt blockchain or other antifraud technologies? As an exam- tracting terms depend on the properties of the transac- ple, Ads.txt adoption was boosted significantly when Google tion. A related question is whether there are strategies adopted and promoted the standard, but Google did not actively buyers or market makers can adopt independently to support the standard until after a significant number of publish- improve market efficiency in such settings. Model- ers adopted it. To date, several blockchain-based solutions have based predictions could help to identify opportunities been proposed for digital ad markets, but we are not aware of to improve market regulations. any that have achieved significant traction. New standards may 2. Another area of opportunity is to help platforms and need to anticipate the chicken-and-egg problem to get ad pur- fraud detection firms design better approaches to detect chasers, sellers and intermediaries on board. and reverse ad fraud. In general, the topic of how to attack machine learning algorithms and how to resist attack is a topic of active research (e.g., Wang et al. Broader Discussion 2019). Consideration of adversarial methods may help firms anticipate and defend against future developments Interactions Between Inefficiencies in fraud technology. It also may help identify opportu- We have discussed each source of inefficiency in isolation, but nities to incentivize humans to prove their humanity in they may interact to exacerbate market-level inefficiencies. A 14 Journal of Marketing XX(X) prominent example is when advertising agencies’ private decisions than others, but we are aware of little public evidence incentives depart from marketing principals’ profit incentives about the financial returns to data-driven advertising decision to measure credible ad effects. This interaction could help making. Similar questions can be posed about estimating the explain the relative scarcity of experimental evidence about returns to using specific new tools in the advertising process, advertising effects and the frequent misinterpretation of obser- such as automated ad content generation, programmatic adver- vational methods as yielding causal ad effects. tising purchases, fraud detection, and brand safety monitoring. Other harmful interactions between inefficiencies appear Another open question is how different types of firms should likely. Agencies may misreport the extent of ad blocking or structure their external advertising agencies and what contract- ad fraud to their clients, especially if negative information ing mechanisms might be optimal in what situations. Differ- might reduce the client’s advertising budget and, consequently, ences in advertiser objectives, capabilities, ownership the agency’s commission. Another prominent example is that structures, and firm size may help explain why some firms ad fraud or ad blocking may contribute to the statistical chal- make more data-driven decisions than other firms. Overall, lenge of measuring ad effects by reducing the number of valid there is substantial scope to shed more light on the antecedents advertising exposures delivered to consumers in treatment and and consequences of advertiser behavior. control cells of an experiment. Industry-level questions. There are not many examples of success- ful regulation in multisided platform industries like digital Policy-Relevant Research Opportunities advertising markets. It is unclear how to optimally trade off conflicting goals among various players. One relevant trade-off Several high-profile regulatory reports have recommended new may occur between consumers’ interests, such as privacy, and regulations for digital advertising markets. Those reports have advertisers’ interests, such as fraud detection. Another relevant described many important topics extensively (e.g., antitrust, trade-off may occur between large platforms’ interests in brand safety, disinformation, consumer privacy, market trans- acquiring early-stage companies and potential competitors’ parency). However, they have covered the four themes treated incentives to enter and innovate. in this article in less detail. We now consider the question of It would also be interesting to consider what metrics regu- how research could help inform broader policy considerations, lators can use to measure the efficiency of digital advertising in addition to the four individual themes treated previously. markets; greater economic activity in advertising markets does Ultimately, digital advertising markets exist to create value not necessarily indicate greater economic welfare (Stigler for the two types of agents they aim to connect: consumers and Committee on Digital Platforms 2019). It seems likely that marketers. Consumers supply attention and realize utility from regulations that entrench incumbents may deter potential advertisements, media content, and advertised products. Mar- entrants, so entry and innovations might be potential indicators keters buy ads and realize profits from product sales. All other of well-functioning markets. It may be that ad blocking instal- relevant parties are intermediaries enabling interactions lations decrease with market performance for consumers and between consumers and marketers. Therefore, consumer wel- therefore might underpin relevant metrics. Perhaps advertiser fare and producer profits should be the two primary concerns in usage of experimentation platforms, and subsequent changes in designing government policy. advertising spending, could indicate how digital advertising markets are serving advertisers’ interests. Consumer-level questions. It is particularly important to under- Finally, a question remains whether regulatory regimes will stand how advertisements affect consumer search for informa- be robust to platforms’ active preparations to be regulated. For tion about products and prices. Some advertising may enhance example, some technology companies have been anticipating product-market outcomes for consumers, whereas other ads antitrust regulation for years (Athey and Luca 2019). If regu- may suppress product search and reduce competition in the latory policy is not robust to adversarial interference, it could product market. It is also important to uncover when advertis- exacerbate the problems it is meant to resolve. ing may have direct negative impacts on consumer utility, as is suggested by the substantial literature on advertising avoidance and the extensive use of ad blockers. There is existing literature Conclusion on these topics, but more comprehensive and systematic evi- dence is needed to guide policy due to the heterogeneity among Digital advertising markets offer numerous innovations over consumers, advertisers, and the variety of advertising channels traditional advertising. Perhaps most notable is the immediacy and messages. with which a small business can target and advertise to local consumers, along with the increased variety of advertising for- Marketer-level questions. The ad effect measurement section mats, relevance, market structures, and interactivity. However, explains some significant obstacles to profit-maximizing as with most technological innovations, the shift to digital advertising decisions. An important topic that is not well docu- advertising has produced costs as well as benefits. We have mented is the variability in how different types of firms make reviewed four common issues that are likely to hinder alloca- different types of advertising decisions. Some advertisers are tive efficiency in digital advertising markets. Most marketers known to be more sophisticated and make more data-driven either do not or cannot measure incremental ad effects; Gordon et al. 15 uncertainty about ad effects may distort market demand for analytics/how-incremental-lift-testing-uncovers-the-truth-behind- advertising. Numerous intermediaries separate marketers from your-results. publishers, each of which takes a cut of advertising expendi- AdTrader (2018), “Why AdTrader Is Suing Google and Why Most tures and has its own private information and incentives, lead- Advertisers Might Never Get that Chance Again,” https://medium. ing to asymmetric information and moral hazard. Consumers com/adtrader/why-adtrader-is-suing-google-and-why-most-adver use ad blocking software to passively prevent advertisements tisers-might-never-get-that-chance-again-38e688d53872. from being displayed, resulting in misappropriated advertising Adweek (2019), “Results of IAB Tech Lab’s Blockchain Pilot Shows revenues and reduced incentives to provide media content. How Many Digital Ads Are Seen by Humans,” https://web.archive. Advertising fraud misrepresents advertising opportunities and org/web/20190213033426/https://www.adweek.com/digital/ directs ad exposures to machines to steal advertising budgets. results-of-iab-tech-labs-blockchain-pilot-shows-how-many-digi This discussion is aimed to help academics, scientists tal-ads-are-seen-by-humans/. employed by advertising companies, policy makers, and exec- Association of National Advertisers (ANA) (2017), “Programmatic: utives as they seek to understand modern digital advertising Seeing through the Financial Fog,” http://www.ana.net/getfile/ markets. We do not claim that these four issues are necessarily 25070. more important than others; we have focused on them because ANA (2018), “The Continued Rise of the In-House Agency,” https:// we believe that they are less well understood than other topics www.ana.net/miccontent/show/id/rr-2018-in-house-agency. that have received more careful study. Such topics as antitrust, Athey, Susan and Michael Luca, (2019), “Economists (and Econom- brand safety, disinformation, consumer privacy, and market ics) in Tech Companies,” Journal of Economic Perspectives,33 transparency are also important policy considerations. We have (1), 209–30. not taken a position on whether or how digital advertising Australian Competition and Consumer Commission (2019), “Digital markets should be regulated, but we do recommend that policy Platforms Inquiry: Final Report,” white paper. makers have full information and hope this article helps them Barajas, Joel, Ram Akella, Marius Holtan, and Aaron Flores (2016), achieve this goal. Overall, we hope this survey of market inef- “Experimental Designs and Estimation for Online Display Adver- ficiencies in digital advertising markets helps develop scien- tising Attribution in Marketplaces,” Marketing Science,35(3), tific literature and better inform the policy-making process. 465–83. Many opportunities remain, and much work has yet to be done. Bawa, Kapil and Robert W. Shoemaker (1987), “The Effects of a Direct Mail Coupon on Brand Choice Behavior,” Journal of Mar- Acknowledgments keting Research, 24 (4), 370–76. The authors thank the associate editor and four anonymous reviewers. Becker, Gary S. and Kevin M. Murphy (1993), “A Simple Theory of They also received helpful comments from Ron Berman, Sundar Bhar- Advertising as a Good or Bad,” The Quarterly Journal of Econom- adwaj, Garrett Johnson, Mingyu Joo, Don Lehmann, Matt McGranaghan, ics, 108 (4), 941–64. Oded Netzer, Dave Reibstein, Catherine Tucker, Sri Venkataraman, Michel Wedel, and Yi Zhu. Any remaining errors are theirs alone. Behdad, Mohammad, Luigi Barone, Mohammed Bennamoun, and Tim French (2012), “Nature-Inspired Techniques in the Context Declaration of Conflicting Interests of Fraud Detection,” IEEE Transactions on Systems and Man,42 (6), 1273–90. The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this Benes, Ross (2019), “Agency Pros Say Fraud Is Biggest Threat to article: Brett R. Gordon was previously a contractor and is currently Their Budgets.” https://www.emarketer.com/content/agency- an employee at Facebook; he donates all resulting income to charity. pros-say-fraud-is-biggest-threat-to-their-budgets. The authors have no other conflicts of interest to report. Bergemann, Dirk and Stephen Morris, (2016), “Information Design, Bayesian Persuasion, and Bayes Correlated Equilibrium,” Ameri- Funding can Economic Review, 106 (5), 586–91. The author(s) received no financial support for the research, author- Berman, Ron (2018), “Beyond the Last Touch: Attribution in Online ship, and/or publication of this article. Advertising,” Marketing Science, 37 (5), 771–92. Berman, Ron and Elea McDonnell Feit (2018), “Enhancing Power of References Marketing Experiments Using Observational Data,” working Abhishek, Vibhanshu, Stylianos Despotakis, and R. Ravi (2017), paper, https://papers.ssrn.com/sol3/papers.cfm?abstract_id¼3 “Multi-Channel Attribution: The Blind Spot of Online Adver- 140631. tising,” working paper, Carnegie Mellon University. Biswas, Shirsho, Jean-Pierre Dube, and Andrey Simonov (2019), Abraham, Magid (2008), “The Off-Line Impact of Online Ads,” Har- “Channel Positions and TV Ad Effects,” unpublished manuscript, vard Business Review, 86 (4), 28. University of Chicago. Acquisti, Alessandro, Leslie K. John, and George Loewenstein (2013), Blake, Tom, Chris Nosko, and Steven Tadelis, (2015), “Consumer “What Is Privacy Worth?” The Journal of Legal Studies, 42 (2), Heterogeneity and Paid Search Effectiveness: A Large-Scale Field 249–74. Experiment,” Econometrica, 83 (1), 155–74. AdRoll (2019), “How Incremental Lift Testing Uncovers the Truth Borden, Neil H. (1964), “The Concept of the Marketing Mix,” Journal Behind Your Results.” https://www.adroll.com/blog/marketing- of Advertising Research, 4 (2), 2–7. 16 Journal of Marketing XX(X)

Brinker, Scott (2019), “Marketing Technology Landscape Super- Advertising,” working paper, https://papers.ssrn.com/sol3/papers. graphic (2019): Martech 5000 (actually 7,040),” https://web. cfm?abstract_id¼3428421. archive.org/web/20190801223536/https://chiefmartec.com/2019/ Despotakis, Stylianos, R. Ravi, and Kannan Srinivasan (2017), “The 04/marketing-technology-landscape-supergraphic-2019/. Beneficial Effects of Ad Blockers,” working paper, Carnegie Mel- Cambria, Erik, Soujanya Poria, Alexander Gelbukh, and Mike lon University. Thelwall (2017), “Sentiment Analysis Is a Big Suitcase,” IEEE Dinner, Isaac M., Harald J. Van Heerde, and Scott A. Neslin (2014), Intelligent Systems, 32 (6), 74–80. “Driving Online and Offline Sales: The Cross-Channel Effects of Cao, Xinyu and T. Tony Ke (2019), “Cooperative Search Adver- Traditional, Online Display, and Paid Search Advertising,” Jour- tising,” Marketing Science 38 (1), 44–67. nal of Marketing Research, 51 (5), 527–45. Chae, Inyoung, Hernan A. Bruno, and Fred M. Feinberg (2019), Du, Rex, Mingyu Joo, and Kenneth C. Wilbur (2019), “Advertising “Wearout or Weariness? Measuring Potential Negative Conse- and Brand Attitudes: Evidence from 575 Brands over Five Years,” quences of Online Ad Volume and Placement on Website Visits,” Quantitative Marketing and Economics, 17 (3), 257–323. Journal of Marketing Research, 56 (1), 57–75. Du, Rex, Xu Linli, and Kenneth C. Wilbur (2019), “Immediate Chalasani, Prasad, Ari Buchalter, Jaynth Thiagarajan, and Ezra Responses of Online Brand Search and Price Search to TV Ads,” Winston (2017), “Counterfactual-Based Incrementality Measure- Journal of Marketing, 83 (4), 81–100. ment in a Digital Ad-Buying PlatformZ,” arXiv preprint arXiv: Du, Ruihuan, Yu Zhong, Harikesh S. Nair, Bo Cui, and Ryan Shou 1705.00634. (2019), “Causally Driven Incremental Multi Touch Attribution Chan, David and Michael Perry (2017), “Challenges and Opportuni- Using a Recurrent Neural Network,” https://arxiv.org/abs/1902. ties in Media Mix Modeling,” technical report, Google Inc. https:// 00215. ai.google/research/pubs/pub45998. Dukes, Anthony and Esther Gal-Or (2003), “Negotiations and Exclu- Chapman, Randall G. (1986), “Assessing the Profitability of Retailer sivity Contracts for Advertising,” Marketing Science,22(2), Couponing with a Low-Cost Field Experiment,” Journal of Retail- 222–45. ing 62 (1), 19–40. Eastlack, Joseph O. and Ambar G. Rao (1989), “Advertising Experi- Chen, Min, Varghese S. Jacob, Suresh Radhakrishnan, and Young U. ments at the Campbell Soup Company,” Marketing Science, 8 (1), Ryu (2015), “Can Payment-per-Click Induce Improvements in 57–71. Identification Technologies?” Information Systems Edelman, Benjamin and Wesley Brandi (2015), “Risk, Information Research, 26 (4), 754–72. and Incentives in Online ,” Journal of Market- Chen, Yuxin and Qihong Liu (2019), “Signaling Through Adver- ing Research, 52 (1), 1–12. tising When Ad Can Be Blocked,” working paper, New York eMarketer (2019a), “Digital Ad Fraud 2019,” https://web.archive.org/ University. web/20190824032234/%20https://www.emarketer.com/content/ Choi, Hana, Carl F. Mela, Santiago Balseiro, and Adam Leary (2019), digital-ad-fraud-2019. “Online Display Advertising Markets: A Literature Review and eMarketer (2019b), “US Digital Ad Spending 2019,” https://www. Future Directions,” working paper, https://papers.ssrn.com/sol3/ emarketer.com/content/us-digital-ad-spending-2019. papers.cfm?abstract_id¼3070706. Feit, Elea McDonnell and Ron Berman (2019), “Test & Roll: Profit- Coviello, Lorenzo, Uri Gneezy, and Lorenz Goette (2017), “A Large- Maximizing A/B Tests,” Marketing Science, 38 (6), 1038–58. Scale Field Experiment to Evaluate the Effectiveness of Paid Fitz, Grant (2018), “It’s Time to Review the Advertising Budget,” Search Advertising,” working paper, https://papers.ssrn.com/sol3/ https://www.cfo.com/budgeting/2018/05/time-to-review-advertis papers.cfm?abstract_id¼3061698. ing-budget. Crouse, Megan Corinn (2010) Business Revolution: The Ad Agency. Forrester Consulting (2014), “Cross-Channel Attribution Is Needed to http://pabook2.libraries.psu.edu/palitmap/AdCo.html. Drive Marketing Effectiveness,” https://think.storage.googleapis. Dan, Avi (2014), “Rethinking the Agency-of-Record Model,” Forbes com/docs/forrester-cross-channel-attribution_research-studies. (April 15) https://www.forbes.com/sites/avidan/2014/04/15/ pdf. rethinking-the-agency-of-record-model/. Fossen, Beth L. and David A. Schweidel (2017), “Television Adver- Danaher, Peter J., Andre Bonfrer, and Sanjay Dhar (2008), “The tising and Online Word-of Mouth: An Empirical Investigation of Effect of Competitive Advertising Interference on Sales for Pack- Social TV Activity,” Marketing Science, 36 (1), 105–23. aged Goods,” Journal of Marketing Research, 45 (2), 211–25. Fuchs, William, Aniko O¨ ry, and Andrzej Skrzypacz (2015), Davies, Jessica (2018), “Ad Tech’s Bid-Caching Controversy, “Transparency and Distressed Sales Under Asymmetric Explained,” Pixalate accessed April 2020, https://digiday.com/ Information,” Theoretical Economics, 11 (3), 1103–44. media/ad-techs-bid-caching-controversy-explained. Gabzewicz, Jean J., Paolo G. Garelli, and Natalie Sonnac (2007), Davies, Jessica (2019), “Ghost Sites, Domain Spoofing, Fake Apps: A “Newspapers’ Market Shares and the Theory of the Circulation Guide to Knowing Your Ad Fraud,” https://web.archive.org/web/ Spiral,” Information Economics and Policy, 19 (3–4), 405–13. 20190206032634/https://digiday.com/media/ghost-sites-domain- Gordon, Brett R., Florian Zettelmeyer, Neha Bhargava, and Dan spoofing-fake-apps-guide-knowing-ad-fraud/. Chapsky (2019), “A Comparison of Approaches to Advertising Decarolis, Francesco and Gabriele Rovigatti (2019), “From Mad Men Measurement: Evidence from Big Field Experiments at to Maths Men: Concentration and Buyer Power in Online Facebook,” Marketing Science, 38 (2), 193–225. Gordon et al. 17

Gritckevich, Aleksandr, Zsolt Katona, and Miklos Sarvary (2019), Kalyanam, Kirthi, John McAteer, Jonathan Marek, and Lifeng Lin “Ad Blocking,” working paper, Columbia University. (2017), “Cross Channel Effects of Search Engine Advertising on Ha, Anthony (2018), “German Supreme Court Dismisses Axel Brick & Mortar Retail Sales: Meta-Analysis of Large Scale Field Springer Lawsuit, Says Ad Blocking Is Legal,” https://tech Experiments on Google.com,” Quantitative Marketing and Eco- crunch.com/2018/04/20/adblock-plus-v-axel-springer. nomics, 16 (6), 1–42. Harberger, Arnold C. (1954), “Monopoly and Resource Allocation,” Kamenica, Emir and Matthew Gentzkow (2011), “Bayesian The American Economic Review, 44 (2), 77–87. Persuasion,” American Economic Review, 126 (4), 2590–615. Hartmann, Wesley R. and Daniel Klapper (2018), “Super Bowl Ads,” Katona, Zsolt and Miklos Sarvary (2018), “Eyeo’s Adblock Plus: Marketing Science, 37 (1), 78–96. Consumer Movement or Advertising Toll Booth?” Berkeley- Hill, Daniel N., Robert Moakler, Alan E. Hubbard, Vadim Haas Case Series B5912. Tsemekhman, Foster Provost, and Kiril Tsemekhman (2015), Kim, Alex and Subramanian Balachander (2017), “Coordinating “Measuring Causal Impact of Online Actions Via Natural Experi- Traditional Media Advertising and Search Advertising,” working ments: Application to Display Advertising,” in 21th ACM paper, University of California, Riverside. SIGKDD International Conference. New York: Association for Laffont, Jean-Jacques and David Martimort (2001), The Theory of Computing Machinery. Incentives: The Principal-Agent Model. Princeton, NJ: Princeton Homburg, Christian, Ove Jensen, and Harlem Krohmer (2008), University Press. “Configurations of Marketing and Sales: A Taxonomy,” Journal Lambrecht, Anja and Catherine E. Tucker (2013), “When Does Retar- of Marketing, 72 (2), 133–54. geting Work? Information Specificity in ,” Horsky, Dan, Sharon Horsky, and Robert Zeithammer (2016), “The Journal of Marketing Research, 50 (5), 561–76. Modern Advertising Agency Selection Contest: A Case for Sti- Lewis, Randall A and David H. Reiley (2014), “Online Ads and Off- pends to New Participants,” Journal of Marketing Research,53 line Sales: Measuring the Effect of Retail Advertising via a Con- (5), 773–89. trolled Experiment on Yahoo!,” Quantitative Marketing and Horsky, Sharon (2006), “The Changing Architecture of Advertising Economics, 12 (3), 235–66. Agencies,” Marketing Science, 25 (4), 367–83. Lewis, Randall A. and Daniel Nguyen (2015), “Display Advertising’s House of Lords Select Committee on Communications (2019), Competitive Spillovers to Consumer Search,” Quantitative Mar- “Regulating in a Digital World,” white paper. keting and Economics, 13 (2), 93–115. Hu, Yu Jeffrey, Jiwoong Shin, and Zhulei Tang (2016), “Incentive Prob- Lewis, Randall A. and Justin M. Rao (2015), “The Unfavorable Eco- lems in Performance-Based Online Advertising Pricing: Cost per nomics of Measuring the Returns to Advertising,” The Quarterly Click vs. ,” Management Science, 62 (7), 2022–38. Journal of Economics, 130 (4), 1941–73. Huang, Jason, David H. Reiley, and Nickolai M. Riabov (2018), Lewis, Randall A. and David H. Reiley (2013), “Down-to-the-Minute “Measuring Consumer Sensitivity to Audio Advertising: A Field Effects of Super Bowl Advertising on Online Search Behavior,” in Experiment on Pandora Internet Radio,” http://www.davidreiley. EC ‘13 Proceedings of the fourteenth ACM Conference on Elec- com/papers/PandoraListenerDemandCurve.pdf. tronic Commerce. New York: Association for Computing Machin- Internet Advertising Bureau (IAB) (2019), “IAB Internet Advertising ery, 639–56. Revenue Report,” white paper. Lewis, Randall A., Justin M. Rao, and David H. Reiley (2011), “Here, Jerath, Kinshuk, T. Tony Ke, and Fei Long (2018), “The Logic and There, and Everywhere: Correlated Online Behaviors Can Lead to Management of ‘Digital Co-op’ in Search Advertising,” working Overestimates of the Effects of Advertising,” in Proceedings of the paper, Columbia University. 20th International Conference on , 157–66. Johnson, Garrett and Randall A. Lewis (2015), “Cost per Incremental Lewis, Randall A., Justin M. Rao, and David H. Reiley (2015), Action: Efficient Pricing of Advertising,” working paper, https:// “Measuring the Effects of Advertising: The Digital Frontier.” in ssrn.com/abstract¼2668315. Economic Analysis of the Digital Economy, Avi Goldfarb, Shane Johnson, Garrett, Randall A. Lewis, and Elmar I. Nubbemeyer M. Greenstein and Catherine E. Tucker, eds. Chicago: The Uni- (2017a), “Ghost Ads: Improving the Economics of Measuring versity of Chicago Press, 191–218. Online Ad Effectiveness,” Journal of Marketing Research,54 Lewis, Randall A. and Jeffrey Wong (2018), “Incrementality Bidding (6), 867–84. and Attribution.” http://causaleffect.io/lewis2018.pdf. Johnson, Garrett, Randall A. Lewis, and Elmar I. Nubbemeyer Liaukonyte, Jura, Thales Teixeira, and Kenneth C. Wilbur (2015), (2017b), “The Online Display Ad Effectiveness Funnel & Carry- “Television Advertising and ,” Marketing Sci- over: Lessons from 432 Field Experiments,” working paper, ence, 34 (3), 311–30. https://papers.ssrn.com/sol3/papers.cfm?abstract_id¼2701578. Lin, Xiliang, Harikesh S. Nair, Navdeep S. Sahni, and Caio Waisman Joo, Mingyu, Kenneth C. Wilbur, Bo Cowgill, and Yi Zhu (2014), (2019), “Parallel Experimentation in a Competitive Advertising “Television Advertising and Online Search,” Management Sci- Marketplace,” https://arxiv.org/abs/1903.11198. ence, 60 (1), 56–73. Liu, Liu, Daria Dzyabura, and Natalie Mizik (2018), “Visual Listening K2 Intelligence (2016), “An Independent Study of Media Transpar- In: Extracting Brand Image Portrayed on Social Media,” in The ency in the U.S. Advertising Industry.” https://www.ana.net/con Workshops of the Thirty-Second AAAI Conference on Artificial tent/show/id/industry-initiative-media-transparency. Intelligence, 71–77. 18 Journal of Marketing XX(X)

Lodish, Leonard M., Magid Abraham, Stuart Kalmenson, Jeanne Sahni, Navdeep N. (2015), “Effect of Temporal Spacing Between Livelsberger, Beth Lubetkin, Bruce Richardson, and Mary Ellen Advertising Exposures: Evidence from an Online Field Experi- Stevens (1995), “How TV Advertising Works: A Meta-Analysis of ment,” Quantitative Marketing and Economics, 13 (3), 203–47. 389 Real World Split Cable TV Advertising Experiments,” Jour- Sahni, Navdeep N., Sridhar Narayanan, and Kirthi Kalyanam (2019), nal of Marketing Research, 32 (2), 125–39. “An Experimental Investigation of the Effects of Retargeted Mayzlin, Dina and Jiwoong Shin (2011), “Uninformative Advertising Advertising: The Role of Frequency and Timing,” Journal of Mar- as an Invitation to Search,” Marketing Science, 30 (4), 666–85. keting Research, 56 (3), 401–18. McKinsey & Company (2013), “How CMOs Can Get CFOs on Their Schmidt, Susanne and Martin Eisend (2015), “Advertising Repetition: Side,” https://www.mckinsey.com/business-functions/marketing- A Meta-Analysis on Effective Frequency in Advertising,” Journal and-sales/our-insights/how-cmos-can-get-cfos-on-their-side. of Advertising Research, 44 (4), 415–28. Naik, Prasad A. and Kalyan Raman (2003), “Understanding the Schwartz, Eric M., Eric T. Bradlow, and Peter S. Fader (2017), Impact of Synergy in Multimedia Communications,” Journal of “Customer Acquisition via Display Advertising Using Multi- Marketing Research, 40 (4), 375–88. Armed Bandit Experiments,” Marketing Science, 36 (4), 500–22. Nair, Harikesh S., Sanjog Misra, William J. Hornbuckle IV, Ranjan Searls, David (2019), “Is Ad Blocking Past 2 Billion Worldwide?” Mishra, and Anand Acharya (2017), “Big Data and Marketing http://blogs.harvard.edu/doc/2019/03/23/2billion/. Analytics in Gaming: Combining Empirical Models and Field Shapiro, Bradley (2018), “Positive Spillovers and Free Riding in Experimentation,” Marketing Science, 36 (5), 699–725. Advertising of Prescription Pharmaceuticals: The Case of Anti- Nanigans (2019), “TIL: Ghost Ads,” https://www.nanigans.com/2019/ depressants,” Journal of Political Economy, 126 (1), 381–437. 01/25/til-ghost-ads/. Shiller, Benjamin, Joel Waldfogel, and Johnny Ryan (2018), “The Narayanan, Sridhar and Kirthi Kalyanam (2015), “Measuring Position Effect of Ad Blocking on Website Traffic and Quality,” RAND Effects in Search Advertising: A Regression Discontinuity Journal of Economics, 49 (1), 43–63. Simonov, Andrey, Chris Nosko, and Justin M. Rao (2018), Approach,” Marketing Science, 34 (3), 388–407. “Competition and Crowd-Out for Brand Keywords in Sponsored Neff, Jack (2015), “The CMO’s Guide to Agency Procurement,” Search,” Marketing Science, 37 (2), 200–15. AdAge, May 13. https://adage.com/article/cmo-strategy/cmo-s- Simonov, Andrey and Justin M. Rao (2019), “Firms’ Reactions to guide-agency-procurement/298536. Public Information on Business Practices: Case of Search Adver- Netzer, Oded, Ronen Feldman, Jacob Goldenberg, and Moshe Fresko tising,” Quantitative Marketing and Economics, 17 (2), 105–34. (2012), “Mine Your Own Business: Market-Structure Surveillance Sinkinson, Michael and Amanda Starc (2019), “Ask Your Doctor? Through Text Mining,” Marketing Science, 31 (3), 521–43. Direct-to-Consumer Advertising of Pharmaceuticals,” Review of Pathak, Shareen (2018), “Marketing 2019: The Year of Consolidation Economic Studies, 86 (2), 836–81. Arrives in Force,” https://digiday.com/marketing/marketing-2019- Sluis, Sarah (2019), “Marketers Are Going Straight to Exchanges for year-consolidation-arrives-force/. Ultimate Data Transparency,” https://adexchanger.com/data- P´erez-Rosas, Veronica, Rada Mihalcea, and Louis-Phillippe Morency exchanges/marketers-are-going-straight-to-exchanges-for-ulti (2013), “Utterance-Level Multimodal Sentiment Analysis,” in mate-data-transparency/#close-olyticsmodal. Proceedings of the 51st Annual Meeting of the Association for Smith, Timothy M., Srinath Gopalakrishna, and Rabikar Chatterjee Computational Linguistics, 1, 973–82. (2006), “A Three-Stage Model of Integrated Marketing Commu- Peterson, Tim (2017), “FAQ: Everything Facebook Has Admitted nications at the Marketing–Sales Interface,” Journal of Marketing About Its Measurement Errors,” MarketingLand.com, https://mar Research, 43 (4), 564–79. ketingland.com/heres-itemized-list-facebooks-measurement- Soleymani, Mohammad, David Garcia, Brendan Jou, Bjo¨rn Schuller, errors-date-200663. Shih-Fu Chang, and Maja Pantic (2017), “A Survey of Multimodal Pixalate (2018), “Ad Fraud (IVT) Benchmarks: 10%-15% of Desktop Sentiment Analysis,” Image and Vision Computing, 65, 3–14. Traffic Was Invalid in Q2 2018,” https://web.archive.org/web/ Stigler Committee on Digital Platforms (2019), “Final Report,” 20190207193901/https://blog.pixalate.com/ad-fraud-ivt-bench https://www.judiciary.senate.gov_imo_media_doc_market- marks-desktop-q2-2018. 2Dstructure-2Dreport-2520-2D15-2Dmay-2D2019.pdf. Poria, Soujanya, Erik Cambria, Devamanyu Hazarika, Navonil Swant, Marty (2019), “Results of IAB Tech Lab’s Blockchain Pilot Majumder, Amir Zadeh, and Louis-Phillippe Morency (2017), Shows How Many Digital Ads Are Seen by Humans,” AdWeek “Context-Dependent Sentiment Analysis in User-Generated (February 11), https://www.adweek.com/digital/results-of-iab- Videos,” in Proceedings of the 55th Annual Meeting of the Asso- tech-labs-blockchain-pilot-shows-how-many-digital-ads-are-seen- ciation for Computational Linguistics, 1, 873–83. by-humans/. Rafieian, Omid and Hema Yoganarasimhan, (2019), “Targeting and Sweeney, Erica (2018), “Study: 21% of Marketers Pull Back Ad Privacy in ,” Marketing Science, https://www. Spend due to Poor Digital Measurement,” https://web.archive. msi.org/reports/targeting-and-privacy-in-mobile-advertising/. org/web/20190730133601/https://www.marketingdive.com/news/ Rao, Justin M. and Andrey Simonov (2019), “Firms’ Reactions to study-21-of-marketers-pull-back-ad-spend-due-to-poor-digital- Public Information on Business Practices: The Case of Search measurement/522199/. Advertising,” Quantitative Marketing and Economics,17(2), Tuchman, Anna E., Harikesh S. Nair, and Pedro M. Gardete (2018), 105–34. “Television Ad Skipping, Consumption Complementarities and Gordon et al. 19

the Consumer Demand for Advertising,” Quantitative Marketing Wilbur, Kenneth C. and Yi Zhu (2009), “Click Fraud,” Marketing and Economics, 16 (2), 111–74. Science, 28 (2), 293–308. Vaver, Jon and Jim Koehler (2011), “Measuring ad Effectiveness Winegar, Angela G. and Cass R. Sunstein (2019), “How Much Is Data Using Geo Experiments,” technical report, Google Inc. Privacy Worth? A Preliminary Investigation,” JournalofCon- Villas-Boas, J. Miguel (1994), “Sleeping with the Enemy: Should sumer Policy, 42, 425–40. Competitors Share the Same Advertising Agency?” Marketing Zantedeschi, Daniel, Elea McDonnell Feit, and Eric T. Bradlow Science, 13 (2), 190–202. (2017), “Measuring Multichannel Advertising Response,” Man- Wang, Xianmin, Jing Li, Xiaohui Kuang, Yu-an Tan, and Jin Li agement Science, 63 (8), 2706–28. (2019), “The Security of Machine Learning in an Adversarial Set- Zhong, Zemin (2018), “A Model of Search Design,” working paper, ting: A Survey,” Journal of Parallel and Distributed Computing, University of Toronto. 130 (August), 12–23. Zhu, Shitong, Xunchao Hu, Zhiyun Qian, Zubair Shafiq, and Heng Yin Waisman, Caio, Harikesh S. Nair, Carlos Carrion, and Nan Xu (2019), (2018), “Measuring and Disrupting Anti-Adblockers Using Differ- “Online Inference for Advertising Auctions,” https://arxiv.org/abs/ ential Execution Analysis,” in Network and Distributed Systems 1908.08600. Security (NDSS) Symposium, February 18–21, San Diego, CA.