STFU NOOB! Predicting Crowdsourced Decisions on Toxic Behavior in Online Games [Please cite the WWW’14 version of this paper] Jeremy Blackburn Haewoon Kwak University of South Florida Telefonica Research Tampa, FL, USA Barcelona, Spain
[email protected] [email protected] ABSTRACT game. With the advent of the free-to-play business model, where One problem facing players of competitive games is negative, or the game is given away for free and the developers are supported toxic, behavior. League of Legends, the largest eSport game, uses via micro-transactions, a game’s sustainability is directly related a crowdsourcing platform called the Tribunal to judge whether a to the strength of its community. A large, healthy community at- reported toxic player should be punished or not. The Tribunal is a tracts new players and keeps existing players engaged and willing two stage system requiring reports from those players that directly to spend money. Left unchecked, toxic behavior threatens to tear a observe toxic behavior, and human experts that review aggregated game’s community apart. reports. While this system has successfully dealt with the vague There have been many attempts to deal with toxic behavior. The nature of toxic behavior by majority rules based on many votes, it earliest required direct human intervention, usually from a game naturally requires tremendous cost, time, and human efforts. master/administrator. While accurate and decisive, this type of sys- In this paper, we propose a supervised learning approach for tem doesn’t scale when there are 10s of millions of active players. predicting crowdsourced decisions on toxic behavior with large- More recently, crowdsourced systems, which make use of the input scale labeled data collections; over 10 million user reports involved of many humans, have arisen.