Framing Effect: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases TAL AUGUST, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA NIGINI OLIVEIRA, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA CHENHAO TAN, Department of Computer Science, University of Colorado Boulder, USA NOAH A. SMITH, Paul G. Allen School of Computer Science & Engineering, University of Washington & Allen Institute for Artificial Intelligence, USA KATHARINA REINECKE, Paul G. Allen School of Computer Science & Engineering, University of Washington, USA Online experimentation with volunteers relies on participants’ non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., “What is your thinking style?”). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of potential sample biases, we measured how different slogan frames affected which participants self-selected into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame froma ‘supporting science’ frame to a ‘comparing oneself to others’ frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers. CCS Concepts: • Human-centered computing → Empirical studies in HCI; Additional Key Words and Phrases: framing; study recruitment; volunteer-based online experimentation ACM Reference Format: Tal August, Nigini Oliveira, Chenhao Tan, Noah A. Smith, and Katharina Reinecke. 2018. Framing Effect: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases . Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 22 (November 2018), 19 pages. https://doi.org/10.1145/3274291 Authors’ addresses: Tal August, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, [email protected]; Nigini Oliveira, Paul G. Allen School 22 of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, [email protected]; Chenhao Tan, Department of Computer Science, University of Colorado Boulder, 430 UCB, Boulder, CO, 80309, USA, [email protected]; Noah A. Smith, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, & Allen Institute for Artificial Intelligence, USA, [email protected]; Katharina Reinecke, Paul G. Allen School of Computer Science & Engineering, University of Washington, 185 E Stevens Way NE, Seattle, WA, 98195-2350, USA, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Association for Computing Machinery. 2573-0142/2018/11-ART22 $15.00 https://doi.org/10.1145/3274291 Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 22. Publication date: November 2018. 22:2 T. August et al. 1 INTRODUCTION Researchers are increasingly conducting online studies with volunteers, such as on the platforms VolunteerScience [62], GamesWithWords [23], LabintheWild [47], and TestMybrain [19], as well as by simply posting study links on online social networks. Unlike compensated studies, which motivate participants through financial gain, studies with volunteers incentivize participation through intrinsic motivations, such as the desire to help science or to learn more about oneself. To leverage these motivations and attract participants, researchers often advertise studies using catchy slogans, such as “Shine a light on vision research!” Prior work in volunteer recruiting across a variety of domains shows that the use of slogan framing—where the way information is presented leads to differences in how it is perceived [13, 60]—can lead to surprisingly large differences in how many and which people are attracted [22, 33, 56, 60]. Similar effects of slogan framing could be expected in volunteer-based online studies. Infact, these studies often collect widely varied sample sizes [35], with markedly different makeups in terms of demographics and motivations to take studies (e.g., interest in supporting science versus interested in comparing to others) [29, 47]. These sample biases, i.e., recruiting a participant sample not representative of the entire population, can threaten study validity and generalizability [6]. The known differences in sample size and makeup of people that studies attract, coupledwith framing literature showing that slogans can attract differently depending on the individual, suggest that how a study is advertised (i.e., what slogan it uses) is important in collecting adequate and representative samples of participants. Are some slogan frames best at attracting participants? Do certain types of participants prefer specific slogan frames more than others? If so, what sortsof slogans will attract whom? The goal of this paper is to (1) identify how using slogan frames affects recruitment in online studies with volunteers and (2) determine whether different slogan frames recruit participants differently based on motivation and demographics. To find out how slogan preferences, participant motivations and demographics relate to one another, we collected preferences for different slogan frames and motivations for volunteering from 59 participants on LabintheWild, a volunteer-based online experimentation site [47]. We then explored how these preferences could affect sample size and composition in online studies by deploying these slogans on five LabintheWild studies, measuring how each affected subject recruitment with 2,670 participants. Our analyses contribute two main findings: (1) Slogans offering the incentive of participants learning about themselves or comparing themselves to others are preferred by significantly more participants than other frames. This preference leads to marked differences in recruitment: using a slogan that offers participants thebenefit of learning something new about themselves, such as personalized feedback or a social comparison, can lead to almost a 9% increase in recruitment over a slogan appealing to participants’ altruism to support science. (2) Slogans focusing on participants learning something new about themselves recruited participants who were significantly more interested to take studies out of boredom than other frames. This influence of participant motivation on slogan preference can lead to higher drop outrates and less attentive participants [29], impacting sample composition and threatening results. (3) There were no significant differences between the age or gender of participants recruited byeach frame. While past work has shown differences in the the age and gender of people attracted to different slogan advertisements, we did not find a similar result. Our results suggest that researchers can use frames to boost recruitment in online studies, but that it is important to recognize the possible sample biases a researcher introduces by using a specific frame. For example, advertising studies as a way for participants to learn somethingnew Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 22. Publication date: November 2018. 22:3 about themselves attracts participants more motivated by boredom than other frames. Participants motivated by boredom have previously been found to be more inclined to drop out or not pay attention to study instructions [29]. Based on our results, we recommend cycling through multiple different slogan frames to advertise an online study in order to both collect adequate numbersof participants while also ensuring participants with diverse motivations. 2 RELATED WORK Related to our research questions mentioned in the introduction is prior work on: (1) motivations to participate in online experiments and community-driven projects, (2) framing effects and target marketing, and (3) framing for recruitment of volunteers. 2.1 Motivation for Taking Online Studies and Contributing to Community-Driven Online Projects People take part in online studies around the world, yet their motivations for doing so differ. For example, people on Amazon Mechanical Turk (MTurk)—a large, general-purpose crowd-sourcing site that researchers use to collect participant samples by offering small financial compensation [38, 50]—complete tasks (whether