Bias in Online Freelance Marketplaces: Evidence from Taskrabbit and Fiverr
Total Page:16
File Type:pdf, Size:1020Kb
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr Anikó Hannák Claudia Wagner David Garcia Northeastern University GESIS Leibniz Institute for the ETH Zürich, Switzerland [email protected] Social Sciences & U. of [email protected] Koblenz-Landau [email protected] Alan Mislove Markus Strohmaier Christo Wilson Northeastern University GESIS Leibniz Institute for the Northeastern University [email protected] Social Sciences & U. of [email protected] Koblenz-Landau [email protected] ABSTRACT INTRODUCTION Online freelancing marketplaces have grown quickly in re- Online freelance marketplaces such as Upwork, Care.com, cent years. In theory, these sites offer workers the ability and Freelancer have grown quickly in recent years. These to earn money without the obligations and potential so- sites facilitate additional income for many workers, and cial biases associated with traditional employment frame- even provide a primary income source for a growing works. In this paper, we study whether two prominent minority. In 2014, it was estimated that 25% of the online freelance marketplaces—TaskRabbit and Fiverr— total workforce in the US was involved in some form of are impacted by racial and gender bias. From these two freelancing, and this number is predicted to grow to 40% platforms, we collect 13,500 worker profiles and gather in- by 2020 [37, 34]. formation about workers’ gender, race, customer reviews, Online freelancing offers two potential benefits to workers. ratings, and positions in search rankings. In both market- places, we find evidence of bias: we find that perceived The first, flexibility, stems from workers’ ability to decide gender and race are significantly correlated with worker when they want to work, and what kinds of tasks they evaluations, which could harm the employment opportu- are willing to perform [33]. Indeed, online freelancing nities afforded to the workers. We hope that our study websites provide job opportunities to workers who may fuels more research on the presence and implications of be disenfranchised by the rigidity of the traditional labor discrimination in online environments. market, e.g., new parents who can only spend a few hours working on their laptops at night, or people with disabilities [66]. ACM Classification Keywords The second potential benefit of online freelance market- H.3.5 Online Information Services: Web-based services; places is the promise of equality. Many studies have un- J.4 Social and Behavioral Sciences: Sociology; K.4.2 covered discrimination in traditional labour markets [12, Social Issues: Employment 22, 8], where conscious and unconscious biases can limit the opportunities available to workers from marginalized groups. In contrast, online platforms can act as neutral intermediaries that preclude human biases. For example, Author Keywords when a customer requests a personal assistant from Fancy Gig economy; discrimination; information retrieval; Hands, they do not select which worker will complete the linguistic analysis task; instead, an algorithm routes the task to any avail- Permission to make digital or hard copies of all or part of this work able worker. Thus, in these cases, customers’ preexisting for personal or classroom use is granted without fee provided that biases cannot influence hiring decisions. copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM While online freelancing marketplaces offer the promise must be honored. Abstracting with credit is permitted. To copy of labor equality, it is unclear whether this goal is being otherwise, or republish, to post on servers or to redistribute to lists, achieved in practice. Many online freelancing platforms requires prior specific permission and/or a fee. Request permissions from [email protected]. (e.g., TaskRabbit, Fiverr, Care.com, TopCoder, etc.) are CSCW 2017, February 25–March 1, 2017, Portland, OR, USA. still designed around a “traditional” workflow, where Copyright is held by the owner/author(s). Publication rights licensed customers search for workers and browse their personal to ACM. ACM ISBN 978-1-4503-4335-0/17/02 ...$15.00. http://dx.doi.org/10.1145/2998181.2998327 profiles before making hiring decisions. Profiles often self-report gender or race,1 we infer these variables by contain the worker’s full name and a headshot, which having humans label their profile images. Additionally, allows customers to make inferences about the worker’s we also recorded each workers’ rank in search results gender and race. Crucially, perceived gender and race for a set of different queries. To analyze our dataset, may be enough to bias customers, e.g., through explicit we use standard regression techniques that control for stereotyping, or subconscious preconceptions independent variables, such as when a worker joined the marketplace and how many tasks they have completed. Another troubling aspect of existing online freelancing marketplaces concerns social feedback. Many freelancing Our analysis reveals that perceived gender and race have websites (including the four listed above) allow customers significant correlations with the amount and the nature to rate and review workers. This opens the door to of social feedback workers receive on TaskRabbit and negative social influence by making (potentially biased) Fiverr. For example, on both services, workers who collective, historical preferences transparent to future are perceived to be Black receive worse ratings than customers. Additionally, freelancing sites may use rating similarly qualified workers who are perceived to be White. and review data to power recommendation and search More problematically, we observe algorithmic bias in systems. If this input data is impacted by social biases, search results on TaskRabbit: perceived gender and race the result may be algorithmic systems that reinforce have significant negative correlations with search rank, real-world hiring inequalities. although the impacted group changes depending on which city we examine. In this study, our goal is to examine bias on online free- lancing marketplaces with respect to perceived gender Our findings illustrate that real-world biases can manifest and race. We focus on the perceived demographics of in online labor markets and, on TaskRabbit, impact workers since this directly corresponds to the experience the visibility of some workers. This may cause negative of customers when hiring workers, i.e., examining and outcomes for workers, e.g., reduced job opportunities and judging workers based solely on their online profiles. We income. We concur with the recommendations of other control for workers’ behavior-related information (e.g., researchers [23, 62, 58], that online labor markets should how many tasks they have completed) in order to fairly be proactive about identifying and mitigating biases on compare workers with similar experience, but varying their platforms. perceived demographic traits. In particular, we aim to investigate the following questions: Limitations. It is important to note that our study 1. How do perceived gender, race, and other demographics has several limitations. First, our data on worker demo- influence the social feedback workers receive? graphics is based on the judgement of profile images by human labelers. In other words, we do not know the true gender or race of workers. Fortunately, our methodology 2. Are there differences in the language of the reviews for closely corresponds to how customers perceive workers workers of different perceived genders and races? in online contexts. Second, although our study presents evidence that per- 3. Do workers’ perceived demographics correlate with ceived gender and race are correlated with social feedback, their position in search results? our data does not allow us to investigate the causes of these correlations, or the impact of these mechanisms These questions are all relevant, as they directly impact on workers’ hireability. Prior work has shown that sta- workers’ job opportunities, and thus their ability to earn tus differentiation and placement in rankings do impact a livelihood from freelancing sites. human interactions with online systems [49, 18], which As a first step toward answering these questions, we suggests that similar effects will occur on online freelance present case studies on two prominent online freelancing marketplaces, but we lack the data to empirically confirm marketplaces: TaskRabbit and Fiverr. We chose these ser- this. vices because they are well established (founded in 2008 Third, since we do not know customers’ geolocations, and 2009, respectively), and their design is representative we are unable to control for some location effects. For of a large class of freelancing services, such as Upwork, example, a customer may prefer to only hire workers who Amazon Home Services, Freelancer, TopCoder, Care.com, live in their own town for the sake of expedience, but if Honor, and HomeHero. Additionally, TaskRabbit and the racial demographics of that town are skewed, this Fiverr allow us to contrast if and how biases manifest in may appear in our models as racial bias. markets that cater to physical tasks (e.g., home clean- ing) and virtual tasks (e.g., logo design) [59]. Lastly, we caution that our results from TaskRabbit and Fiverr may not generalize to other freelancing services. For this study, we crawled data from TaskRabbit and This work is best viewed as a case study of two services Fiverr in late 2015, collecting over 13,500 worker profiles. at a specific point in time, and we hope that our findings These profiles include the tasks workers are willing to complete, and the ratings and reviews they have received 1We refer to this variable as “race” rather than “ethnicity” from customers. Since workers on these sites do not since it is only based on people’s skin color. will encourage further inquiry and discussion into labor thus facilities that enable customers to evaluate indi- equality in online marketplaces. vidual workers are critical (e.g., detailed worker profiles with images and work histories).