Evaluating Changes to Fake Account Verification Systems

Evaluating Changes to Fake Account Verification Systems Fedor Kozlov†, Isabella Yuen†, Jakub Kowalczyk†, Daniel Bernhardt†, David Freeman†, Paul Pearce†‡§, and Ivan Ivanov† †Facebook, Inc ‡Georgia Institute of Technology §International Computer Science Institute Abstract of billions of people every day. The largest social networks— Facebook, Twitter, LinkedIn, and Instagram—provide a broad Online social networks (OSNs) such as Facebook, Twitter, set of features enabling more than two billion people to share and LinkedIn give hundreds of millions of individuals around news, media, opinions, and thoughts [12,49]. The scale and the world the ability to communicate and build communities. scope of these OSNs in turn attract highly motivated attackers, However, the extensive user base of OSNs provides consider- who seek to abuse these platforms and their users for political able opportunity for malicious actors to abuse the system, with and monetary gain [3]. fake accounts generating the vast majority of harmful actions The prevalence, impact, and media coverage of harmful and content. Social networks employ sophisticated detection social media accounts has increased commensurately with mechanisms based on machine-learning classifiers and graph the growth of the platforms [8,28]. A key contributor to this analysis to identify and remediate the actions of fake accounts. problem is fake accounts—accounts that do not represent an Disabling or deleting these detected accounts is not tractable authentic user, created for the express purpose of abusing the when the number of false positives (i.e., real users disabled) platform or its users. is significant in absolute terms. Using challenge-based verification systems such as CAPTCHAs or phone confirmation as Recent research estimates as much as 15% of all Twitter a response for detected fake accounts can enable erroneously accounts to be fake [51], and Facebook estimates as much as detected real users to recover their access, while also making 4% of their monthly active users to fall into this category [11]. it difficult for attackers to abuse the platform. These fake accounts post spam, compromise user data, gener- ate fraudulent ad revenue, influence opinion, or engage in a In order to maintain a verification system’s effectiveness multitude of other abuses [14, 15, 38, 44, 48]. over time, it is important to iterate on the system to improve the real user experience and adapt the platform’s response to The variety of behaviours exhibited by fake accounts— adversarial actions. However, at present there is no established especially those controlled by humans—makes building ac- method to evaluate how effective each iteration is at stopping curate detection systems a challenge. On a platform with fake accounts and letting real users through. This paper pro- billions of active users, a detection system with even 99% poses a method of assessing the effectiveness of experimental precision would incorrectly identify hundreds of thousands iterations for OSN verification systems, and presents an eval- of users every day as malicious. It follows that OSNs require uation of this method against human- labelled ground truth remediation techniques that can tolerate false positives with- data using production Facebook data. Our method reduces the out incurring harm, while still providing significant friction volume of necessary human labelled data by 70%, decreases for attackers. the time necessary for classification by 81%, has suitable pre- A common OSN remediation technique is to enroll fake cision/recall for making decisions in response to experiments, accounts detected by a detection system into a verification and enables continuous monitoring of the effectiveness of the system [17, 33] aimed at blocking access to the OSN for fake applied experimental changes. accounts and providing a way to recover an account for le- gitimate users. These systems are composed of challenges which prompt identified users to provide some additional 1 Introduction information such as phone numbers, recent activity, or iden- tity verification. These challenges—of which the best known Online Social Networks (OSNs) enable people to build com- example is a CAPTCHA [53]—take the form of challenge- munities and communicate effortlessly. With the proliferation response tests that are designed to be easy for real users to of social media usage, OSNs now play a role in the lives pass, but difficult for attackers to solve. Verification systems have numerous advantages over direct disabling of accounts. additional signals, after which they are evaluated against a They provide a soft response that is tolerant of false positives: continuously retrained machine-learned ensemble decision a real user classified as potentially fake has semi-automated tree of account behaviours. Using this model to evaluate test means of changing the classification result without substan- and control groups of accounts that pass the verification sys- tial impact on their engagement. The challenges themselves tem allows us to determine the change in post-verification provide an opportunity to collect additional signals about the fake/real distributions and ultimately how successful an in- user (e.g., time-to-solve), which can aid in further investiga- troduced change is at improving the system’s effectiveness. tion, re-classification, and remediation. The strength (friction) Section3 provides an overview of Facebook’s verification sys- of the challenge can be scaled based on initial classification tem and relevant background. Section4 discusses the design confidence of the detection system. of this model and several variants. We assess our approach Despite these advantages, attackers can adapt to overcome with experiments conducted on Facebook’s production veri- the friction posed by verification system challenges [27, 35, fication system, described in Section5. Our system: enables 41]. It follows that continuously iterating on the design of rapid A/B experimentation; supports an arbitrary number of those challenges and being able to measure the effectiveness backtests of the experimental changes, allowing us to con- of the iterations over time is an important component of im- tinuously monitor the effectiveness of the improvements and proving fake account defences, which has not yet been ad- adversarial response over time; supports a variety of verifica- dressed in the research literature. tion system challenges. We seek to understand iteration effectiveness: the degree We deployed our approach in a real-world setting at Face- to which a new or improved challenge is more successful in book to assess its potential effectiveness. Our approach, PAS, both stopping fake accounts and letting real users through. provided useful signal on whether accounts clearing the ver- To compare effectiveness, we subject pools of accounts to ification system were real or fake; it vastly out-performed two different experiences in an A/B experiment and compute random assignment, achieving precision over 70% and recall the change in the proportion of fake and real accounts that over 60% for all three classes. This approach reduced the vol- managed to successfully pass the verification process. This ume of human labelling for the life cycle of an experiment by computation is particularly challenging as it involves deter- 70%, and the labelling frequency from continuous to a single mining the true nature of a set of users that were already post-experiment operation. Practically, we showed that our identified as fake (with high-probability) by an in-production approach could reduce the time necessary for classification by detection framework. To aid in classification one could lever- up to 81%. This reduction in human effort allowed Facebook age human labelling of accounts at various stages within and to run more experiments in parallel, improving the agility and after the verification process. However, relying on human scale of their experimentation methods. labelling limits the scale and speed of experiments, especially Furthermore, the deployed model completely automated when we require that: many experiments can be run at the the backtests of successfully launched experiments. Thanks same time; we support backtests, a technique where some to automated backtesting, three instances of adversarial adap- flows are withheld from a small proportion of the popula- tation to the experimental changes were discovered, allowing tion after they have become the default experience, in order the Facebook team to quickly find appropriate mitigations. to gauge adversarial response; experiments must be power- ful enough to show results on different user segments (e.g., Out-of-scope: In this work, we focus on classification of platform, locale). fake and real accounts that were already detected by an in- To enable such classification at scale and across such re- production detection framework and were able to pass chal- quirements, our approach is to replace the majority of human lenges in OSN verification systems, such as CAPTHA and labelling with automated techniques having precision/recall phone confirmation. Automated classification of these ac- suitable for both making decisions on the experiments and counts enables an assessment of experimental iterations for continuously monitoring the effectiveness of the applied ex- OSN verification systems in order to improve real user experimental changes. perience and increase friction for fake accounts. Based on description above, we consider the following areas out of Our

Load more