Who’s the Guinea Pig? Investigating Online A/B/n Tests in-the-Wild Shan Jiang John Martin Christo Wilson Northeastern University Northeastern University Northeastern University
[email protected] [email protected] [email protected] ABSTRACT users of a website are split into n groups, with one serving as the A/B/n testing has been adopted by many technology companies as control and the other n − 1 as treatments. The control group is a data-driven approach to product design and optimization. These shown the product as-is, while the treatments groups are shown tests are often run on their websites without explicit consent from variations. After some time has passed, the users’ behaviors are users. In this paper, we investigate such online A/B/n tests by using analyzed to determine which treatment, if any, had a desirable effect Optimizely as a lens. First, we provide measurement results of (e.g., more clicks, higher conversion rates, etc.). 575 websites that use Optimizely drawn from the Alexa Top-1M, Many large technology companies are open about their advocacy and analyze the distributions of their audiences and experiments. of Online Controlled Experiments (OCEs) like A/B/n testing. In 2000, Then, we use three case studies to discuss potential ethical pitfalls Google famously used A/B tests to experiment with the presentation of such experiments, including involvement of political content, of results in Google Search, and by 2011 Google engineers claimed to price discrimination, and advertising campaigns. We conclude with be running over 7,000 A/B tests per year [21].