What Makes a “Bad” Ad? User Perceptions of Problematic Online Advertising Eric Zeng Tadayoshi Kohno Franziska Roesner Paul G. Allen School of Computer Paul G. Allen School of Computer Paul G. Allen School of Computer Science & Engineering Science & Engineering Science & Engineering University of Washington University of Washington University of Washington Seattle, WA, USA Seattle, WA, USA Seattle, WA, USA [email protected] [email protected] [email protected] ABSTRACT that they are interested in. Still, many web users dislike online ads, Online display advertising on websites is widely disliked by users, fnding them to be annoying, intrusive, and detrimental to their with many turning to ad blockers to avoid “bad” ads. Recent ev- security or privacy. In an attempt to flter such “bad” ads, many idence suggests that today’s ads contain potentially problematic users turn to ad blockers [5] — for instance, a 2016 study estimated content, in addition to well-studied concerns about the privacy and that 18% of U.S. internet users and 37% of German internet users intrusiveness of ads. However, we lack knowledge of which types used an ad blocker [69], a large percentage considering that it takes of ad content users consider problematic and detrimental to their some initiative and technical knowledge to seek out and install an browsing experience. Our work bridges this gap: frst, we create a ad blocker. taxonomy of 15 positive and negative user reactions to online ad- There are many drivers of negative attitudes towards online vertising from a survey of 60 participants. Second, we characterize ads. Some users fnd the mere presence of ads to be problematic, classes of online ad content that users dislike or fnd problematic, often associated with their (perceived) increasingly disruptive, in- using a dataset of 500 ads crawled from popular websites, labeled trusive, and/or annoying qualities [5] or their impact on the load by 1000 participants using our taxonomy. Among our fndings, we times of websites [92]. Users are also concerned about the privacy report that users consider a substantial amount of ads on the web impacts of ads: research in computer security and privacy has re- today to be clickbait, untrustworthy, or distasteful, including ads vealed extensive ecosystems of tracking and targeted advertising for software downloads, listicles, and health & supplements. (e.g., [9, 28, 30, 61, 62, 64, 76, 84, 97, 98]), which users often fnd to be creepy and privacy-invasive (e.g., [29, 96, 100, 101]). The specifc CCS CONCEPTS content of ads can also cause direct or indirect harms to consumers, ranging from material harms in the extreme (e.g., scams [1, 34, 72], • Information systems Online advertising; Display adver- ! malware [65, 74, 104, 105], and discriminatory advertising [3, 57]) tising; • Social and professional topics Commerce policy; ! to simply annoying techniques that disrupt the user experience • Human-centered computing User studies. ! (e.g., animated banner ads [16, 38, 45]). In this work, we focus specifcally on this last category of KEYWORDS concerns, studying people’s perceptions of problematic or “bad” online advertising, deceptive advertising, dark patterns user-visible content in modern web-based ads. Driving this ex- ACM Reference Format: ploration is the observation that problematic content in mod- Eric Zeng, Tadayoshi Kohno, and Franziska Roesner. 2021. What Makes ern web ads can be more subtle than fashing banner ads and a “Bad” Ad? User Perceptions of Problematic Online Advertising. In CHI outright scams. Recent anecdotes and studies suggest high vol- Conference on Human Factors in Computing Systems (CHI ’21), May 8–13, umes and a wide range of potentially problematic content, in- 2021, Yokohama, Japan. ACM, New York, NY, USA, 24 pages. https://doi.org/ cluding “clickbait”, advertorials or endorsements with poor dis- 10.1145/3411764.3445459 closure practices, low-quality content farms, and deceptively for- matted “native” ads designed to imitate the style of the hosting 1 INTRODUCTION page [4, 7, 22, 39, 52, 63, 68, 71, 75, 90, 93, 103, 106]. While re- Online display advertising is a critical part of the modern web: searchers and the popular press have drawn attention to these ads sustain websites that provide free content and services to con- types of ad content, we lack a systematic understanding of how sumers, and many ads inform people about products and services web users perceive these types of ads on the modern web in general. What makes an ad “bad”, in the eyes of today’s web users? What are Permission to make digital or hard copies of all or part of this work for personal or people’s perceptions and mental models of ads with arguably prob- classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation lematic content like “clickbait”, which falls in a grey area between on the frst page. Copyrights for components of this work owned by others than the scams and poorly designed annoying ads? What exactly is it that author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or causes people to dislike (or like) an ad or class of ads? For future republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from [email protected]. regulation and research attempting to classify, measure, and/or CHI ’21, May 8–13, 2021, Yokohama, Japan improve the quality of the ads ecosystem, where exactly should the © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM. line be drawn? ACM ISBN 978-1-4503-8096-6/21/05...$15.00 https://doi.org/10.1145/3411764.3445459 CHI ’21, May 8–13, 2021, Yokohama, Japan Zeng et al. We argue that such a systematic understanding of what makes an ad “bad” — grounded in the perceptions of a range of web users, not expert regulators, advertisers, or researchers — is crucial for two reasons. First, while some ads can clearly be considered “bad”, like outright scams, and others can be considered “benign”, like honest ads for legitimate products, there is a gray area where it is more nuanced and difcult to cleanly classify. For example, “clickbait” ads for tabloid-style celebrity news articles may not cross the line for causing material harms to consumers, but may annoy many users and use misleading techniques. While the U.S. Federal Trade Commission currently concerns itself with explicitly harmful ads like scams and deceptive disclosures [18, 33, 63], whether and how Figure 1: An overview of our work and contributions. to address “clickbait” and other distasteful content is more nuanced. As part of our work, we seek to identify ads that do not violate current regulations and policies, but do harm user experiences, Contributions. Figure 1 shows an overview of the diferent com- in order to inform improvements such as policy changes or the ponents of our work and our resulting outputs and contributions. development of automated solutions. Second, research interested Specifcally, our contributions include: in measuring, classifying, and experimenting on “bad” online ads will beneft from having detailed defnitions and labeled examples (1) Based on a qualitative survey characterizing 60 participants’ of “bad” ads, grounded in real users’ perceptions and opinions. For attitudes towards the content and techniques found in mod- example, our prior work measuring the prevalence of “problematic” ern online web ads, we distill a taxonomy of 15 reasons why ads on the web used a researcher-created codebook of potentially people dislike (and like) ads on the web, such as “untrustwor- problematic ad content; that codebook was not directly grounded thy”, “clickbait”, “ugly / bad style”, and “boring” (Section 3, in broader user experiences and perceptions [106]. answering RQ1). (2) Using this taxonomy, we generate a dataset of 500 ads sam- Research Questions. In this paper, our goal is thus to systemati- pled randomly from a crawl of popular websites, labeled with cally elicit and study what kinds of online ads people dislike, and 12,972 opinion labels from 1025 people (Section 4, towards the reasons why they dislike them, focusing specifcally on the user- answering RQ2). This dataset is available in the paper’s sup- visible content of those ads (rather than the underlying technical plemental materials1. mechanisms for ad targeting and delivery). We have two primary (3) Combining participant opinion labels with researcher con- research questions: tent labels of these 500 ads, and using unsupervised learning (1) RQ1 — Defning “bad” in ads: What are the diferent types techniques, we identify and characterize classes of ad content of negative (and positive) reactions that people have to online and techniques that users react negatively to, such as click- ads that they see? In other words, why do people dislike (or bait native ads, distasteful content, deceptive and “scammy” like) online ads? content, and politicized ads (Section 4, answering RQ2). (2) RQ2 — Identifying and characterizing “bad” ads: What Our fndings serve as a foundation for policy and research on specifc kinds of content and tactics in online ads cause peo- problematic online advertising: for regulators, advertisers, and ad ple to have negative reactions? In other words, which ads do platforms, we provide evidence on which types of ads are most people dislike (or like)? detrimental to user experience and consumer welfare, and for re- While ads appear in many places online — including in social searchers, we provide a user-centric framework for defning prob- media feeds and mobile apps — we focus specifcally on third party lematic ad content, enabling future research on the online advertis- programmatic advertising on the web [2], commonly found on ing ecosystem.