This article was downloaded by: [8.9.237.33] On: 06 March 2020, At: 11:24 Publisher: Institute for Operations Research and the Management Sciences (INFORMS) INFORMS is located in Maryland, USA
Management Science
Publication details, including instructions for authors and subscription information: http://pubsonline.informs.org Can Reputation Discipline the Gig Economy? Experimental Evidence from an Online Labor Market
Alan Benson, Aaron Sojourner, Akhmed Umyarov
To cite this article: Alan Benson, Aaron Sojourner, Akhmed Umyarov (2019) Can Reputation Discipline the Gig Economy? Experimental Evidence from an Online Labor Market. Management Science Published online in Articles in Advance 12 Sep 2019 . https://doi.org/10.1287/mnsc.2019.3303
Full terms and conditions of use: https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and- Conditions
This article may be used only for the purposes of research, teaching, and/or private study. Commercial use or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher approval, unless otherwise noted. For more information, contact [email protected]. The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or support of claims made of that product, publication, or service. Copyright © 2019, The Author(s)
Please scroll down for article—it is on subsequent pages
With 12,500 members from nearly 90 countries, INFORMS is the largest international association of operations research (O.R.) and analytics professionals and students. INFORMS provides unique networking and learning opportunities for individual professionals, and organizations of all types and sizes, to better understand and use O.R. and analytics tools and methods to transform strategic visions and achieve better outcomes. For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org MANAGEMENT SCIENCE Articles in Advance, pp. 1–24 http://pubsonline.informs.org/journal/mnsc ISSN 0025-1909 (print), ISSN 1526-5501 (online)
Can Reputation Discipline the Gig Economy? Experimental Evidence from an Online Labor Market
Alan Benson,a Aaron Sojourner,a Akhmed Umyarova a Carlson School of Management, University of Minnesota, Minneapolis, Minnesota 55455 Contact: [email protected], http://orcid.org/0000-0003-4256-3357 (AB); [email protected], http://orcid.org/0000-0001-6839-2512 (AS); [email protected], http://orcid.org/0000-0003-3731-9093 (AU)
Received: July 28, 2017 Abstract. Just as employers face uncertainty when hiring workers, workers also face Revised: July 3, 2018; December 17, 2018 uncertainty when accepting employment, and bad employers may opportunistically Accepted: December 28, 2018 departfromexpectations,norms,andlaws.However,priorresearchineconomicsand Published Online in Articles in Advance: information sciences has focused sharply on the employer’s problem of identifying good September 12, 2019 workers rather than vice versa. This issue is especially pronounced in markets for gig https://doi.org/10.1287/mnsc.2019.3303 work, including online labor markets, in which platforms are developing strategies to help workers identify good employers. We build a theoretical model for the value of Copyright: © 2019 The Author(s) such reputation systems and test its predictions on Amazon Mechanical Turk, on which employers may decline to pay workers while keeping their work product and workers protect themselves using third-party reputation systems, such as Turkopticon. We find that(1)inanexperimentonworkerarrival, a good reputation allows employers to operate more quickly and on a larger scale without loss to quality; (2) in an experimental audit of employers, working for good-reputation employers pays 40% higher effective wages because of faster completion times and lower likelihoods of rejection; and (3) exploiting reputation system crashes, the reputation system is particularly important to small, good-reputation employers, which rely on the reputation system to compete for workers against more established employers. This is the first clean field evidence of the
effectsofemployerreputationinanylabormarket and is suggestive of the special role that reputation-diffusing technologies can play in promoting gig work, in which con- ventional labor and contract laws are weak.
Open Access Statement: This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License. You are free to download this work and share with others commercially or noncommercially, but cannot change in any way, and you must attribute this work as “Management Science. Copyright © 2019 The Author(s). https://doi.org/10.1287/mnsc.2019.3303, used under a Creative Commons Attribution License: https://creativecommons.org/licenses/by-nd/4.0/.” History: Accepted by Chris Forman, information science. Funding: The authors thank the University of Minnesota Social Media and Business Analytics Col- laborative for funding. Supplemental Material: Data are available at https://doi.org/10.1287/mnsc.2019.3303.
Keywords: online labor markets • online ratings • employer reputation • labor markets • economics of information systems • job search • electronic markets and auctions • IT policy and management • contracts and reputation • information and market efficiency
1. Introduction and norms. To what extent can the threat of a bad Amazon Mechanical Turk (M-Turk), TaskRabbit, public reputation prevent employer opportunism and Upwork, Uber, Instacart, DoorDash, and other online the promise of a good reputation encourage respon- platforms have drastically reduced the cost of seek- sible employer behavior? To what extent can workers ing, forming, and terminating work arrangements. voluntarily aggregate their private experiences into This development has raised the concern that plat- shared memory to discipline opportunistic employers? forms circumvent regulations that protect workers. These questions are especially important to gig jobs and A U.S. Government Accountability Office (2015,p.22) online labor platforms, which are characterized by high- report notes that such “online clearinghouses for frequency matching of workers and employers, and on obtaining ad hoc jobs” are attempting “to obscure or which platform owners have moved fast and often eliminate the link between the worker and the busi- sought to break out of traditional regulatory regimes. ness . . . which can lead to violations of worker pro- Our setting focuses on M-Turk, which possesses tection laws.” Developers of these online platforms two useful and rare features for this purpose. First, no argue that ratings systems can help discipline em- authority (neither the government nor the plat- ployers and other trading partners that break rules form) disciplines opportunistic employers. In M-Turk,
1 Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 2 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) after workers put forth effort, employers may keep the indeed, that Yelp penetration in an area is associated with work product but refuse payment for any reason or the decline in chains that presumably rely on other forms no reason. Workers have no contractual recourse or of reputation. Nagaraj (2016) finds that digitization of appeal process. Because M-Turk also has no native magazine articles has a greater effect on Wikipedia entries tool for rating employers, many workers use third- for less-known individuals than for well-known in- party platforms to help them find the best em- dividuals for whom other information is readily available. ployers. Among the most popular of these platforms In three tests, we provide, to our knowledge, the is Turkopticon, which allows workers to share in- first clean field evidence of employer reputation ef- formation about employers. Second, such oppor- fects in a labor market. These tests follow the model’s tunism is observable to researchers with some effort. three predictions. Specifically, the first experiment In traditional labor markets, workers would pre- measures the effect of employer reputation on the sumablyrelyonanemployer’s reputation to protect ability to recruit workers. We create 36 employers on themselves against events not protected by a contract, M-Turk. Using a third-party employer rating site, such as promises for promotion or paying for over- Turkopticon, we endow each with (i) 8–12 good time. Unfortunately, these outcomes are especially dif- ratings, (ii) 8–12 bad ratings, or (iii) no ratings. We ficult to observe for both courts and researchers alike. then examine the rate at which they can recruit The literature on both traditional and online labor mar- workers to posted jobs. We find that employers with kets has instead focused on learning about opportun- good reputations recruit workers about 50% more ism by workers, not by employers (e.g., List 2006,Oyer quickly than our otherwise-identical employers with and Schaefer 2011, Moreno and Terwiesch 2014, Pallais no ratings and 100% more quickly than those with 2014, Stanton and Thomas 2015, Farronato et al. 2018, very bad reputations. Using M-Turk wage elastici- and Filippas et al. 2018). These two contextual fea- ties estimated by Horton and Chilton (2010), we es- tures enable our empirical study. timate that posted wages would need to be almost We begin by outlining a model of employer reputa- 200% greater for bad-reputation employers and tion. The model allows for two forms of employer rep- 100% greater for no-reputation employers to attract utation: a public reputation disseminated by a formal workers at the same rate as good-reputation em- rating system and an informal reputation governed by an ployers. Outside of M-Turk, one might think of the individual employer’s private visibility. The model attractiveness of the job as the firm’s ability to attract yields three key, testable predictions regarding the value applicants and reputation as a substitute for wage of reputation. First, employers with better reputations for that purpose. A better reputation shifts the firm’s are rewarded through an enhanced ability to attract labor supply curve to the right, allowing it to recruit workers at a given promised payment level, and in this and select from more workers for a given wage offer in sense, a better reputation acts as collateral against a given period of time. We also estimate that about future opportunism. Second, workers earn more when 55% of job searchers use Turkopticon, suggesting that they have information that enables them to work only more complete adoption would magnify these effects. for better-reputation employers. Third, the value of the We find evidence that Turkopticon is signaling em- public reputation system to employers depends on ployer characteristics rather than just task charac- their visibility: less-visible employers with good reputa- teristics. These results demonstrate that workers use tions on the system rely on it to attract workers, and better- reputation to screen employers and that reputation known, good-reputation employers are less reliant. This affects employers’ ability to recruit workers. third prediction has important implications for how a The second experiment tests the validity of online credible reputation system changes what types of trading reputations from the perspective of a worker. Repu- relationships a market with poor enforcement can bear. tation systems based on individual ratings are vulner- Interpreting this prediction, less well-known em- able to inflation and inaccuracy (Filippas et al. 2018). ployers, such as newer or smaller firms, struggle more to We act as a blinded worker to assess the extent to earn workers’ trust to recruit workers and grow. which other workers’ public employer ratings reflect Credible reputation systems especially help these less- real variation in employer quality. One research as- visible employers and, thereby, support the entry of sistant(RA)randomlyselectstasksfromemployers firms from the competitive fringe and promote economic who have good reputations, bad reputations, or no competition and dynamism. Our model is consistent reputation and sends each job to a second RA, who with the digitization literature’s findings that rating does the jobs while blind to employers’ reputations. systems are especially important to relatively unknown This experimental feature ensures that worker effort agents and not as important to agents with an otherwise is independent of employer reputation, and so any highly visible reputation. Luca (2016) finds that Yelp.com observed differences in subsequent employer be- reviews are especially important for smaller, inde- havior toward the worker are not a result of differ- pendent restaurants than for restaurant chains and, ences in worker effort. Consistent with a pooling Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 3 equilibrium, we observe no difference in the average Dellarocas and Wood 2008 and Nosko and Tadelis per-task payment promised up front by employers 2015) has studied sellers on eBay, on which buyers of different reputations. However, effective wages rate sellers on the accuracy of product descriptions while working for good-reputation employers are 40% and timeliness, but sellers only care that buyers sub- greater than effective wages while working for bad- mit payment. A similar case could be made for Yelp reputation employers. We decompose this difference (e.g., Luca 2016), on which the quality of the food and into the shares resulting from differences in job quality service is difficult to contract upon, but restaurants (how long the job takes) versus the probability of being serve all comers. paid at all. This experiment shows that, although the Although the vast majority of work has concerned reputation system aggregates ratings that are all vol- unilateral rating regimes, Airbnb (e.g., Fradkin et al. untarily provided and unverified, it contains useful 2015)andUber(e.g.,Rosenblatetal.2015) offer two information for workers. major exceptions: Airbnb hosts and guests rate each Finally, we focus on instances when Turkopticon other as do Uber drivers and passengers. Although servers stopped working by using a difference-in- bothsidesvaluetheeaseofplatformmatchingand differences design to study effects when the reputa- microcontracting, both also worry about opportu- tion system is removed temporarily. We match data nistic behavior by counterparties. These platforms try on M-Turk tasks completed across employers over to create healthy markets while avoiding regulation time from Ipeirotis (2010a) with each employer’s by developing two-sided online ratings systems. The contemporaneous Turkopticon ratings and compare digital labor platform Upwork has also used a two- the change in worker arrival rates for different types sided rating system, but research there has not fo- of employers when Turkopticon crashes. When Tur- cused on employer reputation. kopticon crashes, employers with bad reputations are Issues of trust and contract enforcement among largely unaffected. Taken with the prior evidence, this trading partners are especially important to online result suggests that employers with bad reputations spot markets and other forms of gig work. The nature do not recruit workers who use Turkopticon and rely of the employment relationship, especially in gig only on uninformed workers. However, the effect on work, features bilateral uncertainty. Despite this, the employers with good reputations on Turkopticon is literature on both traditional and online labor has heterogeneous by employer visibility, measured as the largely focused on the employer’s problem of eval- number of times an employer posted work in the past uating and rating employees rather than vice versa. and proxying for the likelihood that a worker would Labor economics has given great attention to how have encountered it before. Workers sharply with- employers interpret educational credentials, work draw their labor supply from less-visible, good- experience, or their experience at that firm to identify reputation employers, who presumably were benefit- themostproductiveworkers(forareview,seeOyer ing from Turkopticon informing workers of their good and Schaefer 2011). reputations. In contrast, worker arrival rates increase This focus on guarding against worker opportun- for more-visible, good-reputation employers, who are ism now extends to the burgeoning literature on online presumably already better known as safe bets. labormarkets.Studies,forinstance,haveexamined how online employers infer workers’ abilities from 2. Literature Review their work histories (Pallais 2014), oversubscription Most markets have information problems to some (Horton 2019), work through outsourcing agencies degree. For M-Turk workers, Turkopticon is the Dun & (Stanton and Thomas 2015), national origin (Agrawal Bradstreet of procurers, the Moody’sofbondbuyers, et al. 2013), or platform endorsements (Barach et al. the Fair Isaac of consumer lenders, and the Metacritic 2019). Although these studies present a sample of the of moviegoers. Each of these institutions offers ex- recent literature on online labor markets, they are tralegal protections against contractual incomplete- characteristic of its present focus on rating workers ness based on information sharing and the implicit rather than firms. As such, the literature offers little threat of coordinated withdrawal of trade by one side evidence on the risks that workers face when selecting of a market. employers or how platforms can design systems that A large literature studies unilateral ratings of sup- promote trade absent regulation. pliers in online markets. In principle, ratings should Our work builds on a handful of studies that have be unilateral when suppliers (including workers, sought to empirically examine employer reputation. sellers of goods, and service providers) vary in ways Turban and Cable (2003)providethefirst correla- that are difficult to contract upon, creating problems tional evidence that companies with better reputa- for buyers. In contrast, suppliers do not face many tions tend to attract more applicants using career- issues in choosing customers; buyers are essentially services data from two business schools. Hannon and money on the barrel. For example, prior work (e.g., Milkovich (1996) find mixed evidence that news of Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 4 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) prominent employer rankings affects stock prices. 3. Theory and Hypotheses Using a similar methodology, Chauvin and Guthrie We offer a formal model of job search in which there (1994) find small but significant effects. Brown and is no contract enforcement. Under certain conditions, Matsa (2015) find that distressed financial firms at- a “public relational contract” emerges, whereby the tract fewer and lower-quality applicants. List and threat of losing future workers discourages employer Momeni (2017), also in an experiment on M-Turk, opportunism. In doing so, our model illustrates the fi nd that employers that promise to donate wages insight from List (2006), that reputational concerns to a charity attract more work, albeit with more can lead economic agents to act in a manner that cheating, suggesting that workers take moral license appears more prosocial and fair. We explore how when performing work for good causes. As such, the workers and different kinds of employers rely on an prior evidence on employer reputation is either corre- employer-reputation system and test the model’s lational or concerns uncertainty of other characteristics hypotheses in the online market. fi ’ of the employer (e.g., the rm s future success or al- The model begins from a basic sequential search truism), which workers may value in themselves or model. Workers search for a job to receive a wage offer because they also view these as correlated with their from a random employer and choose whether to fi fi quality as an employer. We provide the rst eld ex- accept each offer and put forth effort. Then, the em- ’ perimental evidence regarding how employers rep- ployer chooses ex post whether to renege on pay- utation for treating workers affects their ability to ment. Workers’ decision to work or to forego an offer attract work. depends on their beliefs as to whether the employer Theoretically, a better public reputation would al- will pay, which depends on two exogenous factors: low trading partners to extract greater work or higher whether the worker is informed by a reputation system prices (Klein and Leffler 1981). Moreno and Terwiesch 1 and whether the employer is visible. Specifically, there (2014) find that online service providers leverage are two types of workers: workers informed by the better reputations to either charge more or increase reputation system see all employers’ past payment their probability of being selected for a project. Ba and histories, and uninformed workers do not. Being in- Pavlou (2002), in a market “similar to eBay,” find that formed, in this sense, signifies access to the collective buyers are willing to pay a price premium to deal experience of prior workers as would be diffused with trusted sellers. Similarly, Banerjee and Duflo by a public reputation system. Second, uninformed (2000) find that supplier reputation is important in workers may nonetheless observe the pay history of the Indian software market, in which postsupply fi an offering employer with a probability that depends service is important but dif cult to contract. McDevitt 2 (2011) finds evidence that residential plumbing firms on a property of the employer: its visibility. The visi- with high records of complaints are more likely to bility of an employer can be thought of simply as it change their name, suggesting that firms seek to is treated in the model: an employer characteristic purge bad reputations. However, in their study of that makes its history known to workers even without eBay sellers, Bajari and Hortacsu (2003) find only a the reputation system. Perhaps the most obvious em- small effect of reputation on prices. Although these pirical proxy for visibility is simply the size of the fi fi studies once again focus on rating the supply side, our employer, which, by de nition, signi es the extent of ’ theory and empirics speak to the demand side by workers past experience with the employer. presenting evidence that employers with good rep- Although the single-shot model would devolve utations can extract lower prices and greater quan- into the classic hold-up problem in which employers tities of work. never pay and workers never put forth effort, we We also consider how public ratings of employers characterize an interesting, nonunique, steady-state ’ substitute for more traditional markers of being equilibrium in which an employer s good reputation an established organization. Stinchcombe (1965)fa- acts as collateral against future wage theft. They con- mously theorized that new organizations face a tinue to pay so that they can attract workers and get credibility problem when attracting trading part- work done in the future. We explore how the existence ners, a phenomenon they refer to as the “liability of of the public reputation system (informed workers) newness.” Luca (2016) offers perhaps the most closely differentially affects highly-visible and less-visible related test for the relative importance of online employers. Employers with a good reputation (“high- ratings versus being established; he finds that Yelp road employers”) continue to pay as long as the share ratings are more important to independent restau- of informed workers is sufficiently high, and employers rants than to chains. As we discuss, our setting offers with a bad reputation (“low-road employers”) never a relatively clean opportunity to exploit exogenous pay.3 We describe conditions that avoid making high- removals of a reputation system, to do so for the road employers’ reneging temptation too great, which demand side, and to do so in a labor market. would then cause all workers to exit the labor market. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 5
This temptation is disciplined by the expected fu- 3. Employer j decides whether to pay w or to renege ture flow of workers, which depends on the extent of and pay zero. Employers discount future periods at workers’ ability to identify high-road employers. As a rate δ. result, we endogenously characterize the scope of We provide a set of parametric restrictions, prove economic activity that this environment can bear and the existence of an equilibrium, and explore its prop- how it depends on the ability of workers to screen erties. First, promised wages exceed the value of work employers through either a public reputation system effort: w − e > 0. This is a trivial precondition for any- or private knowledge.4 one wanting to work. Second, an uninformed worker The model makes three key claims. First, the public will not accept an unknown employer’sofferbecause reputation system deters employer opportunism by the expected payoff for doing so does not justify the creating a credible threat that employers’ ability to certaincostofeffort,sw < e.Thisassuresnoteveryone attract workers in the future will erode (tested in works for anyone. Third, for uninformed workers, study 1). Good-reputation employers can attract promised wages (w) and the chance of being matched more workers to the same job offer than either no- to a visible, high-road employer (vs) times its payoff reputation or bad-reputation employers; a better (w − e) exceed the certain cost of search, vs(w − e)≥c. reputation shifts the labor supply curve to the right. The uninformed worker’s market participation con- Second, it is incentive-compatible for workers to straint is met. It implies a lower bound on wage, wmin − screen out low-road employers in their job search, e +(vs) 1c, necessary to keep uninformed workers a strategy that boosts workers’ hourly earnings from dropping out of the market. It increases in (study 2). Third, the reputation system is a substitute thecostofeffortandsearchanddecreasesinthe for the visibility of individual employers. The repu- share of high-road employers and average employer tation system matters especially for smaller, less- visibility. visible high-road employers, whose good payment The fourth parameter restriction is less trivial and histories would otherwise be unknown to job seekers more interesting. For high-road employers, it requires (study 3). that the gains from high-road trade (y − w),employer Formally, assume a job search environment with farsightedness δ,andtheflow of workers (p + v − pv) measure one of workers indexed by i ∈[0, 1] and outweigh the one-time payoff of reneging (y)and − measure one of risk-neutral employers indexed by continuing as a low-road employer: (1 − δ) 1(p + v − j ∈[0, 1].Ashare,s ∈[0, 1], of employers have a high- pv)(y − w)≥y. We refer to this as the “reputation road history of making promised payments. Workers diffusion” criterion, noting that an employer’spast with i ≤ p ∈(0, 1) are fully informed to all employers’ play must be observed either through worker inform- past play via a public reputation system. Employers edness or employer visibility. As both p and v approach ’ differ in their history’s visibility vj ∈(0, 1) to other zero, workers can t screen employers well enough, ’ fi workers, and high values of vj represent “highly and employers don textractsuf cient rents on a good visible” employers. Let v ≡ E[vj].Workerswhoare reputation to justify maintaining a high-road repu- indifferent between accepting and rejecting offers tation.6 Otherwise, high-road employers would re- choose to accept. Employers indifferent between nege, the value of market participation for all workers paying and reneging choose to pay. Normalizing, we becomes negative, no work is performed, and the labor assume each earns zero if they do not participate in market unravels. this market (e.g., nonwork or the labor market char- An adapted Lerner index, (y − w)/y,measuresthe acterized by this environment). The timing of a period share of worker productivity kept as economic rent by of job search is high-road employers in this equilibrium. The reputation- 1. Worker i chooses whether to search. Those who diffusion criterion implies this share cannot fall be- − do incur cost c and receive a wage promise w from a low [(p + v − pv) 1] * (1 − δ). Better information for random employer j. Fully informed workers observe j’s workers, p or v rising to one, creates space for high- past decisions to pay or renege. Other workers ob- road employers to raise wages and to retain a smaller serve employer j’s history with probability vj.Non- share of worker productivity because they have re- searching workers receive zero and proceed to the liable access to a larger share of workers. This property next period of job search. Zero represents the value reflects the reputation system literature’sviewthat of not participating in the online labor market. betterreputationsystemscanexpandthescopeof 2. Worker i decides whether to accept or reject economic activity that can be completed online by employer j’s offer. If the worker accepts, the worker arms-length trading partners (Jøsang et al. 2007). incurs a cost of effort e, and employer j receives work First, consider workers who vary in their inform- product with value y.5 If the worker rejects, the edness. Informed workers encounter a high-road em- worker receives zero and proceeds to the next period ployer in any period with probability s.Theyaccept of job search. high-road employers’ offers because w − e − c ≥−c, Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 6 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) which follows from the assumption that trade is prof- informed workers decreases (i.e., the employer rep- itable: w − e > 0. They reject low-road employers’ of- utation system crashes, sending p → 0), a high-road fers because −c > −c − e.Therefore,thepresentvalue employer’sworkerflow converges to vj.Themost −1 of this strategy for fully informed workers is (1 − δ) · visible employers (vj → 1) are unaffected, and the [s(w − e)−c]. Uninformed workers encounter an em- arrival rate to the least-visible employers (vj → 0) falls ployer with an observable pay history with proba- more.Putanotherway,thereputationsystem(the bility v. In this case, they face the same incentives and mass of informed workers) is least valuable to the makethesamedecisionsasfully informed workers. most-visible employers and most valuable to the least- Uninformed workers who encounter a nonvisible visible employers. In our model, change in the repu- employer reject the offer because of the parameter tation system p does not affect low-road employers. restriction sw < e. The present value of this strategy They get the same arrival rate regardless of inform- to uninformed workers is (1 − δ)−1[vs(w − e)−c].Both edness or visibility. informed and uninformed workers’ payoffs satisfy The relative value of the reputation system to more- their participation constraint under the condition, versus less-visible employers is a chief contribution vs(w − e)−c ≥ 0. of this model, at least beyond labor markets. Con- Next, consider employers, which vary in their vis- ceptually, we can think of the reputation system as ibility. Low-road employers are supplied no labor be- any technology that makes it less costly for one cause workers only accept offers from revealed, high- trading party to observe the past behaviors of poten- road employers.7 High-road employers receive workers tial partners. In our setting, we observe relatively brief, ’ at a rate of p +(1 − p)vj, receiving all informed workers unexpected instances when the market s public repu- tation system crashes. In these instances, workers and the share vj of uninformed workers. In equilib- rium, it is incentive-compatible for employers to pay who remain in the market must rely on other mech- ’ if the present value payoff of paying exceeds the anisms to screen employers. Because our setting s −1 reputation system is hosted through a third party, its immediate temptation of reneging, (1 − δ) (p + vj − pv)(y − w)≥y. This follows from the reputation- outages present a special opportunity to study how diffusion criterion discussed earlier. The high-road workers adapt and how this change in search be- ’ havior affects employers of varying visibility. employer s participation constraint is simply that ’ y − w ≥ 0. Before continuing, it s also important to note some The first two hypotheses concern the model’skey other interesting features of the model. One relates to predictions regarding why it is incentive-compatible the substitutability of informed workers and visible employers. As p → 1, v no longer affects search or the for employers to maintain a good reputation and for fl → workers to screen for jobs on a good reputation. ow of workers; as v 1, p no longer affects search or the flow of workers. In either case, workers al- Hypothesis 1. For a given job offer, employers with a better ways accept jobs from known high-road employers. reputation attract workers more quickly. In this sense, technologies that give workers a col- lective memory serve as a scalable substitute for the Hypothesis 2. Workers have higher average pay when personal experience that markets have traditionally working for employers with better reputations. relied upon to avoid opportunistic trading partners. Our final hypothesis follows from an interesting A second insight relates to the upper bound on wage property of the model: that the reputation system that this environment can bear before the market that informs workers and employer visibility are sub- breaks down from high-road employer defection. stitutes. This is the key feature of the reputation- Rearranging the reputation diffusion criterion yields − diffusion criterion, and yields the surprising result w ≤ y[1 −(1 − δ)(p + v − pv) 1]≡wmax, which increases that, when the share of informed-type workers is sent in worker informedness p and average employer vis- to zero (e.g., because of the crash of the reputation ibility v. In other words, stronger public reputation system), the only employers affected are those with systems (higher p’s) and better private sources of in- good reputations and imperfect visibility. formation (higher v’s) both increase the upper bound on wages that are supportable in the absence of en- Hypothesis 3. If the reputation system is disabled, sending forceable contracts. the share of informed workers to zero, then (a) highly visible employers with good reputation lose a small or no share of 4. Setting workers, (b) less-visible employers with good reputation lose 4.1. M-Turk a larger share of workers, and (c) employers with bad rep- M-Turk is an online labor market that allows em- utation are unaffected regardless of visibility. ployers (these purchasers of labor are called re- To see this, recall that the flow of workers to high- questers) to crowdsource human intelligence tasks road employers is given by p + vj − pvj.Astheshareof (HITs) to workers over a web browser. Common HITs Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 7 include audio transcription, image recognition, text Horton and Golden 2015). Anyone can post any re- categorization, and other tasks not easily performed by view on Turkopticon. It has no revenue and is main- machines. M-Turk allows employers to process large tained by volunteers. batches of HITs with greater flexibility and at generally When an employer posts a task, it appears to much greater speeds and lower costs than traditional workers on a list of available tasks. This list specifies a employment. short description of the task, the number of tasks Amazon does not generally publish detailed usage available in the batch, the promised pay per task, the statistics; however, in 2010, it reported that more time allotted for workers to complete the task once than 500,000 workers from more than 190 countries they accept it, and the name of the employer. The were registered on M-Turk.8 In 2014, Panos Ipeirotis’ employer also may restrict eligibility to workers with web crawler found that the number of available HITs asufficiently high approval rating, which requires a fluctuated between 200,000 and 800,000 from January history of having submitted work approved and paid to June 2014.9 Ross et al. (2009) found that a majority for by past employers. Workers may preview the task of workers were female (55%) and from the United before accepting. Upon acceptance, a worker has States (57%) or India (32%). Horton and Chilton the allotted time to submit the task. The employer (2010) estimated that the median reservation wage then has a predetermined period to approve or reject was $1.38 an hour. M-Turk’s platform revenue comes the task with or without an accompanying note. from 10% brokerage fees paid for by employers. If the employer approves the task, the employer pays M-Turk specifically has many features making it the posted rate and broker fees to Amazon. The condi- attractive for studying how workers navigate em- tions for approval are not contractible; if the employer ployer heterogeneity using public employer reputa- rejects the task, the worker’s submitted work remains tion.10 First, there is no variation in the terms of in the employer’s possession but no payment is made. contracts. In most labor markets, relationships em- Moreover, the worker’s approval rate declines, re- body a mix of enforceable and unenforceable ele- ducing the worker’s eligibility for other tasks in the ments and the nature of the mix is unknown to the future. There is no process for appealing a rejection. researcher; observed differences between employers Opportunism takes many forms in this market. may reflect differences in workers’ contracts and Employers may disguise wage theft by posting un- access to legal recourse. In M-Turk, workers put paid trial tasks, implicitly with the promise that forth effort, employers acquire the work product, and workers who submit work that matches a known, then employers choose whether to pay workers. correct answer will receive work for pay when, in Employers may refuse payment for any reason or no fact, the trial task is the task itself, and the employer reason, and workers have no contractual recourse. rejects all submitted work for being defective. In This complete lack of contract enforcement is rare and addition to nonpayment, employers may also ad- valuable for research although potentially madden- vertise that a task should take a set amount of time ing for workers. Here, one can be sure that all em- when it is likely to take much longer. Therefore, al- ployer behavior is discretionary and is performed though the promised pay for accepted submissions absent the possibility of enforcement. Second, M-Turk is known, the effective wage rate, depending on the does not have a native employer-reputation system, a time it takes to complete the task, is not. Employers feature it shares with off-line labor markets but unlike can also delay accepting submitted work for up to other online labor markets. This also proves useful by 30 days. Employers may or may not communicate allowing us to decouple worker effort from employer with workers. Employers can also differ in how gen- reputation in the audit study. erous they are to workers. Some might not really do To help avoid employer opportunism, many M-Turk much quality control and pay for all work regardless of workers use Turkopticon, a third-party browser plug- how bad it is or pay a lot for very easy tasks. in that allows workers to review and screen em- ployers (Silberman and Irani 2016). There are several 4.2. Reputation on M-Turk reasons these ratings may be uninformative. First, the Within M-Turk, there is no tool allowing workers to system is unnecessary if workers face no information review employers, and workers cannot observe em- or enforcement problems. Second, the system relies on ployers’ effective wages or payment histories. How- workers voluntarily contributing accurate, private in- ever, several third-party resources allow workers to formation to a common pool, which costs time and share information voluntarily regarding employer directs other workers to scarce, high-paying tasks. This quality. These include web forums, automatic notifi- distinguishes labor markets from consumer markets cation resources, and public-rating sites.11 in which trade is nonrival. Third, ratings systems vary We test our hypotheses regarding the value of the widely in their informativeness because of reputation employer reputation system using Turkopticon, a inflation and other issues (Nosko and Tadelis 2015, community ratings database and browser plugin that Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 8 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) we estimate is used by a slight majority of M-Turk 2. We endowed 12 employers with good reputa- job seekers.12 The plugin adds information to the tions, 12 employers with bad reputations, and left 12 worker’s job search interface, including community employers with no ratings or reputation. We created ratings of an employer’s communicativity, generos- accounts on Turkopticon and posted numerical at- ity, fairness, and promptness. Ratings take integer tribute ratings and long-form text reviews. Reviews values from one to five. As of November 2013, Tur- for our bad (good)-reputation employers are taken as kopticon included 105,909 reviews by 8,734 workers a sample of actual bad (good) reviews of bad (good)- of 23,031 employers. The attributes have a mean of reputation employers on Turkopticon.16 Good- and 3.80 and a standard deviation of 1.72.13 Workers can bad-reputation employers receive 8–12 reviews each. click on a link to read text reviews of an employer. These reviews make our good and bad reputations These reviews typically further recommend or warn not unusual with regard to their mean reputations against doing work for a given employer. Figure 1 although the bad-reputation employers do have an provides an illustration. unusual degree of rater consensus about their bad- Figures 1 and 2 illustrate an M-Turk worker’sjob ness.17 Because M-Turk workers may sort tasks alpha- search process. Figure 1 shows how workers search betically by employers’ names, we balance reputations fortasksforpay.Figure2 shows a preview of the task by the first name of the employer so that reputation that we use for this study. is random with respect to the alphabetical order of Turkopticon is remarkable because it relies on vol- the employer. untary feedback from a community of anonymous 3. Our employer identities took turns posting tasks workers to provide a signal of employer quality. These on M-Turk. They did so in 72 one-hour intervals, reviews are costly in terms of the worker’stime,and posting new tasks on the hour. At the start of each the content of the review is unverifiable to other hour, the employer posted hundreds of tasks, more workers. More importantly, there is wide variation than were ever done within the hour. Each worker in the effective pay rate of individual tasks. Because was allowed to do only one task, and this was ap- employers typically post tasks in finite batches and parent when browsing the job listing. Posts began at allow workers to repeat tasks until the batch is com- 12:00a.m.onTuesday,July7,andendedat11:59p.m. pleted, the wage-maximizing behavior would be to onThursday,July9.Forexample,theemployer hoard tasks posted by good employers by misdirecting named Mark Kelly, who was endowed with a good other workers.14 Because reviews are anonymous, di- reputation on Turkopticon, posted tasks at 12:00 a.m. rect reciprocity and punishment is limited. As such, and ceased accepting new submissions at 12:59 a.m., sharing honest reviews could be thought of as a pro- thereafter disappearing from workers’ search results. social behavior that is costly to the worker in terms of At 1:00 a.m., Joseph Warren, who had no reputation time and valuable private information and in which on Turkopticon, posted new tasks. social recognition or direct reciprocity is limited. Other We balanced the intervals so that (1) in each hour, over studies of online reputation systems suggest that re- three days, the three reputation types are represented viewers are primarily motivated by a “joy of giving” once, and (2) in each hour, over each six-hour partition of and fairness (Cornes and Sandler 1994,Resnickand a day, the three reputation types are represented twice. Zeckhauser 2002). We chose the final schedule (Table 1)atrandomfromthe set of all schedules that would satisfy these criteria. 5. Experiment 1 The tasks consisted of image-recognition exer- 5.1. Setup cises. Workers were asked to enter the names, The first experiment examines the value of the rep- quantity, and prices of alcoholic items from an image utation system to employers. Specifically, we exam- of a grocery receipt that we generated. Receipts were ine whether a good reputation helps employers attract 20 items long and contained three to five alcoholic workers. We do so by creating employers on M-Turk, items.18 Workers could only submit one task in exogenously endowing them with reputations on any one-hour interval. The pay rate was $0.20, and Turkopticon, and then testing the rate at which they workers had 15 minutes to complete the task once attract work. they accepted it. 1. We created 36 employer accounts on M-Turk. 4. Simultaneously, we created three employers The names of these employers consistofpermutations that posted 12-cent surveys requesting informa- of three first names and 12 last names.15 We used tion from workers’ dashboards. These employers multiple employers to protect against the evolution of posted new batches of tasks each hour for 24 hours ratings during the experiment. We chose these names each. Their reputation did not vary. The purpose of because they are common, Anglo, and male (for first this task was to determine a natural baseline arrival names), and our analysis of Turkopticon ratings find rate that could be used as a control in the main that these names are not generally rated high or low. regressions. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 9
Figure 1. (Color online) M-Turk Worker’s Job Search Process: Turkopticon
Notes. Screen capture of an M-Turk worker’s job search interface. The tooltip box left of center is available to workers who have installed Turkopticon and shows color-coded ratings of the employer’s communicativity, generosity, fairness, and promptness. It also offers a link to long- form reviews.
5. We recorded the quantity and quality of completed obvious mechanism is that the workers would just tasks. We did not respond to communications and did consult the public reputation of the employer di- not pay workers until the experiment concluded. rectly each time and pick or reject the jobs accord- Note that employer reputation may affect the la- ingly. The other mechanism would be that some bor supply through multiple causal mechanisms. One workers who discovered a “good” employer would Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 10 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s)
Figure 2. (Color online) M-Turk Worker’s Job Search Process: Previewing, Accepting, and Submitting Tasks
Notes. Screen capture of an M-Turk worker’s job search interface. From the list of tasks, workers must choose to preview a task before accepting the task. They then enter data into the web form and submit their work. invite others to work for this “good” employer as limited to just the first-order effect but rather is the well.19 Accordingly, some workers who discovered overall effect on labor supply. a “bad” employer would warn others not to work for Because our experiment was three full days long, the “bad” employer. In other words, the effect of we also considered the possibility that the reputation employer reputation as examined in our study is not of our employers would start evolving as workers Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 11
Table 1. Balanced, Random Allocation of Employer Identities to Time Slots with Reputation
Tuesday Wednesday Thursday
0:00 (g) Mark Kelly (b) Thomas Jordan (n) Mark Jordan 1:00 (n) Joseph Warren (g) Joseph Jordan (b) Mark Warren 2:00 (g) Thomas Warren (n) Mark Jordan (b) Joseph Kelly 3:00 (n) Thomas Kelly (b) Thomas Jordan (g) Thomas Warren 4:00 (b) Mark Warren (n) Joseph Warren (g) Mark Kelly 5:00 (b) Joseph Kelly (g) Joseph Jordan (n) Thomas Kelly 6:00 (g) Joseph Lewis (n) Thomas Lewis (b) Mark Lewis 7:00 (n) Mark Roberts (g) Thomas Roberts (b) Thomas Clark 8:00 (b) Thomas Clark (n) Thomas Lewis (g) Mark Clark 9:00 (g) Mark Clark (b) Mark Lewis (n) Joseph Clark 10:00 (n) Joseph Clark (b) Joseph Roberts (g) Joseph Lewis 11:00 (b) Joseph Roberts (g) Thomas Roberts (n) Mark Roberts 12:00 (b) Thomas Martin (n) Joseph Johnson (g) Joseph Martin 13:00 (n) Thomas Adams (b) Joseph Adams (g) Mark Adams 14:00 (n) Mark Martin (g) Mark Adams (b) Mark Johnson 15:00 (g) Thomas Johnson (n) Thomas Adams (b) Joseph Adams 16:00 (b) Mark Johnson (g) Thomas Johnson (n) Mark Martin 17:00 (h) Joseph Martin (b) Thomas Martin (n) Joseph Johnson 18:00 (n) Thomas Miller (b) Joseph Robinson (g) Thomas Robinson 19:00 (g) Thomas Robinson (n) Mark Robinson (b) Thomas Owens 20:00 (g) Mark Owens (b) Joseph Robinson (n) Mark Robinson 21:00 (n) Joseph Owens (g) Joseph Miller (b) Mark Miller 22:00 (b) Mark Miller (n) Thomas Miller (g) Joseph Miller 23:00 (b) Thomas Owens (g) Mark Owens (n) Joseph Owens
Note. Parentheses denote employers endowed with good (g), no (n), and bad (b) reputations. would eventually start adding their own true feed- shift and one excluding this last six-hour shift. As can back into the reputation system or might eventually be seen and is discussed in Table 2,ourresultsare start noticing the similarities and patterns between qualitatively identical in both cases. the employers. To account for this, we switched be- tween employers rather quickly: every hour. In ad- 5.2. Results dition to that, we also used a balanced randomization Summarizing the results of the experiment, Figure 3 procedure, such as every six-hour slot20 of the day has shows the cumulative distribution of arrivals across at least two good employers, two bad employers, and the three employer-reputation types. As can be seen two neutral employers as presented in our schedule in Figure 3, our results are qualitatively consistent in Table 1. This way, none of the treatment groups throughout the time of the experiment starting from is affected disproportionally by any potential time evolution of the experiment, and each six-hour slot the very beginning owing to the balanced time allo- offers equal exposure to all treatment groups. cation schedule that we discussed. By the conclusion As we were monitoring the progress of the experi- of each of the 12 six-hour partitions, the employer ment, indeed, on Thursday21 at 4:14 p.m., an obser- with good ratings had attracted more work than the vant worker compiled and publicly announced a list employer with neutral ratings, and the employer with of our 24 employers with good and bad ratings on a neutral ratings had attracted more work than the Reddit forum, noting their similarities and suggesting employer with poor ratings. that the reviews were created by fake accounts. On Table 2 shows results from a negative binomial Thursday at 5:22 p.m., we reached out to this worker and model. In all samples except for the six-hour parti- to a concerned group of other workers on a Turkopticon tions, employers with good reputations attract work discussion board to address the concerns regarding more quickly than employers with poor reputations our employers falsifying reviews with the intent of with p < 0.01. However, if comparing only against no- defrauding workers. We disclosed to this group of reputation employers at a 5% significance level, em- workers that all workers would be paid. On Thursday ployers with a good reputation do not receive sub- at 6:14 p.m., the description of the experiment was mitted work significantly faster than those with no cross-posted on Reddit. As this possibility was con- reputation, and employers with a poor reputation re- sidered in our randomization procedure, we present ceive submitted work significantly slower only in the two sets of the results: one including this last six-hour full samples. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 12 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s)
Table 2. Negative Binomial Regression for Arrival of Submitted Tasks and Other Events
Good reputation No reputation
Sample β Standard error β Standard error Periods Events
Event: submitted tasks Full sample (1) All submitted tasks 2.053* (0.500) 1.503 (0.368) 72 1,641 Subsamples (2) Day 1 only 4.104* (1.969) 2.135 (1.030) 24 695 (3) Days 1–2 only 2.424* (0.766) 1.76 (0.559) 48 1,125 (4) 12 a.m.–6 a.m. 1.679 (0.823) 1.393 (0.689) 18 114 (5) 6 a.m.–12 p.m. 2.843* (1.201) 2.157 (0.915) 18 534 (6) 12 p.m.–6 p.m. 1.096 (0.267) .978 (0.239) 18 415 (7) 6 p.m.–12 a.m. 2.694* (0.955) 1.648 (0.589) 18 577 Excluding last 12 hours (8) No controls 2.466* (0.704) 1.803* (0.516) 60 1,313 (9) Controls for baseline rate 2.523* (0.719) 1.808* (0.515) 60 1,313 (10) Day fixed effects 2.294* (0.654) 1.778* (0.498) 60 1,313 (11) Hour fixed effects 1.858* (0.274) 1.374* (0.205) 60 1,313 (12) Day and hour fixed effects 1.836* (0.262) 1.364* (0.196) 60 1,313 Event: other (13) Task previews 2.314* (0.571) 1.495 (0.370) 72 1,837 (14) Task accepts 2.141* (0.529) 1.551 (0.384) 72 1,799 (15) Error-free submissions 2.018* (0.548) 1.5 (0.410) 72 1,012 (16) First submissions 2.871* (0.804) 1.644 (0.465) 72 899 (17) Error-free first submissions 2.88* (0.928) 1.641 (0.536) 72 508
Notes. Each row is a regression. Coefficients are incidence rate ratios with bad reputation as the omitted category. Standard errors in parentheses. *p < 0.05.
Although finding workers at a slower pace may not experiment as described), the bad employers would seem like such a major punishment for employers indeed get the work done in approximately twice the with a poor reputation at first, we examine this finding time. However, in the labor market in which almost very carefully: the only people who are working for everyone is informed, the already slow pace of work the bad employers are “uninformed” workers. In may become too slow for the bad employer to survive other words, if the labor market has approximately in the market. 50% informed workers and 50% uninformed workers In addition, we also examine differences in esti- (such as Amazon Mechanical Turk as of the time of the mated effort and quality. The mean time spent per
Figure 3. (Color online) Cumulative Accepted Jobs by Employer Reputation
Note. Bold points represent active job listings. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 13 task for good reputation, no reputation, and poor Table 3 also provides evidence about the effects reputation employers were respectively 136, 113, and of reputation on various steps in the matching pro- 121 seconds. The difference between good reputation cess. Conditional on a worker previewing a task, the and no reputation employers is statistically signifi- probability of accepting the task is not significantly cant with p < 0.01. For each of the three groups, the different by treatment. If information received by pre- error-free rates were between 61% and 63%, and the viewing a task (e.g., the type of the task, the intuitiveness major-error rates (e.g., no alcoholic items identified) of the user interface) were a substitute for reputation were between 3.0% and 5.2%. Differences in the error- information, then good-reputation employers would free rates and major-error rates are not statistically lose fewer workers during the preview stage than no- significant.22 Mason and Watts (2010)alsofoundthat reputation employers. In the former, but not latter, higher payments raise the quantity but not quality of workers would already have received the signal prior submitted work. to previewing the task. This evidence suggests that In the full sample, 45.2% of the submitted tasks observable task characteristics do not substitute for were not the first tasks submitted by an individual reputation information. The reputation system adds worker, and 9.7% of the submitted tasks were the information above what workers can otherwise observe. sixth task or greater. The high incidence of repeat Turkopticon is not native to the M-Turk interface submissions may be for a number of factors, including andmustbeinstalledbytheworker.Assuch,the power-users, correlated task search criteria (e.g., in- reputations we endow are visible only to a fraction of dividuals continuously search using the same crite- workers, and so only part of the “treated” population ria), automated alerts (e.g., TurkAlert), or purposely actually receives the treatment. To estimate the searching for the same task across hours. share of M-Turk job seekers who use Turkopticon, Table 3 shows results from our preferred specifi- we posted a one-question, free-response survey asking, cation of the negative binomial regressions to estimate “How do you choose whether to accept HITs from a the arrival rates of task previews, acceptances, sub- requester you haven’t worked for before? Please missions, first submissions (by worker), and correct describe any factors you consider, any steps you take, first submissions. These specifications omit the last and any tools or resources you use.” Becauseweposted 12 hours in which the experiment was disclosed and the survey from a requester account that did not have also include day and hour fixed effects. Arrival rates a Turkopticon rating and because we require workers for good-reputation employers are significantly great- to identify Turkopticon specifically, we expected this er than no-reputation employers for all outcomes, and procedure to yield a conservative estimate of the true arrival rates for no-reputation employers are signifi- portion of job seekers who use Turkopticon. Of these, cantly greater than bad-reputation employers for all 55 of the 100 responses mention Turkopticon ex- outcomes except correct first submissions with p < 0.05. plicitly, and seven other responses mention other or Results provide evidence that good reputations pro- unspecified websites.23 duce more previews, acceptances, submissions, first Experiment 1 also offers three additional pieces submissions, and correct first submissions. of evidence that Turkopticon provides information of The point estimates in column (3) suggest that employer type rather than task type. First, we find arrival rates for employers with good and no reputa- that the observed probability of accepting a task con- tions, respectively, exceed those of employers with bad ditional on previewing a task does not vary signifi- reputations by 84% and 36%. cantly by employer type. Second, we find that the
Table 3. Preferred Specification: Negative Binomial Regression of Arrival Rates in the First 60 Hours with Day and Hour Fixed Effects
Previews (1) Acceptances (2) Submissions (3) First submissions (4) Correct first submissions (5)
Good reputation 1.964* 1.909* 1.836* 2.488* 1.855* (0.280) (0.277) (0.262) (0.426) (0.405) No reputation 1.403* 1.387* 1.364* 1.608* 1.261 (0.204) (0.203) (0.196) (0.277) (0.278) Constant 16.56* 14.10* 13.31* 8.024* 3.54* (4.907) (4.300) (4.002) (2.788) (1.729) Day fixed effects Yes Yes Yes Yes Yes Hour fixed effects Yes Yes Yes Yes Yes Observations 60 60 60 60 60
Notes. Standard errors in parentheses. Bad reputation is the omitted category. All coefficients for good employers are significantly different from coefficients for bad employers with p < 0.05. *p < 0.05. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 14 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) elapsed time that workers spend previewing tasks raters disagree, then the average cost per item rises prior to accepting the task does not vary significantly to $0.071, and the probability an item is coded cor- by reputation type. Third, our survey of 100 M-Turk rectly rises to 0.966.24 The elasticity estimates imply workers featured no workers who reported a belief that reducing price per worker-task to hold average that certain tasks were inherently more fairly or costs per completed task constant reduces quantity highly compensated although nearly all cited ob- of work completed by 23.7%: less than the quantity servable employer characteristics from past experi- gained by a good reputation relative to no reputation. ence or tools such as Turkopticon. These suggest that In other words, a good-reputation employer could workers screened on Turkopticon ratings and not on implement a majority-rules process, cut the price per information (e.g., task type) gathered during the task worker-tasksoastoachievethesamecostpercom- previews. This, along with ratings criteria used by pleted task, improve accuracy from 0.890 to 0.966, and Turkopticon and the test in Experiment 1, lead us to still get work done more quickly than an employer conclude that workers use Turkopticon to get in- with no reputation although doing so may compro- formation about employers that wouldn’tbeacces- mise the employer’s good reputation (especially sible until after they would have otherwise exerted “generosity” ratings) in the long run. effort (e.g., time to completion and nonpayment) rather than getting information on task type. 6. Experiment 2 6.1. Setup 5.3. Alternative Dependent Variables Our model assumes that the reputation system pro- Employers, especially on Mechanical Turk, want to vides accurate information to workers. In reality, the get work done quickly, cheaply, and accurately. Our cheap talk, voluntarily-contributed ratings could be results suggest that better reputations help employers biased or very noisy, such that the “informed” who attract more workers at a given price and quality, giving make decisions based on the reputation system are no an advantage in speed. What if employers wished to more informed than others. Our second experiment use their reputation to achieve lower prices or greater validates this assumption empirically and examines quality? the value of the reputation system to workers. Horton and Chilton (2010)estimatethatM-Turk Specifically, we examined whether Turkopticon workers have a median wage elasticity for recruit- ratings are informative of three employer character- ment of 0.43. If this point elasticity holds for our istics that workers value but about which they face sample, a bad-reputation employer that pays $0.59, a uncertainty during the search process: the likelihood no-reputation employer that pays $0.37, and a good- of payment, the time to payment, and the implicit reputation employer that pays $0.20 would attract wage rate. As reflected in the literature on online work at the same rate. This is a conservative esti- ratings, informedness shouldn’t be taken for granted. mate. Horton and Chilton’s(2010) estimate of the mean Horton and Golden (2015) show that oDesk, an online elasticity is lower (0.24), implying that generating as labor market with a native bilateral rating system, ex- large a difference in worker arrival rates as reputation periences extensive reputation inflation as employers generates would require even larger differences in and workers strategically, rather than truthfully, report promised payments. Dube et al. (2018)synthesize experiences. Others report similar biases on eBay existing evidence, including Horton and Chilton (2010) (Dellarocas and Wood 2008,NoskoandTadelis2015), and subsequent experiments, and new evidence they Airbnb (Fradkin et al. 2015), and Yelp (Luca 2016). generate to estimate that the mean recruitment elas- The validity of Turkopticon ratings may be even more ticity is even lower (0.06), implying that reputation surprising given that tasks offered by revealed good is even more valuable as a substitute for higher wages employers are rival (unlike, for example, good prod- in generating changes in recruitment and worker ar- ucts on retail markets). rival rates to an employer for a given job posting. We followed the following procedure: Moreno and Terwiesch (2014)alsofind that online 1. We produced a random ordering of three rep- service providers on vWorker.com substitute be- utationtypes:good,bad,andnone. tween higher prices and greater volume. Our study 2. The nonblind research assistant (RA1), using a focuses on recruitment, leaving effects on retention browser equipped with Turkopticon, screened the list for future work. of tasks on M-Turk until finding one that met the re- To estimate the value of a good reputation for getting quirements of the next task on the random ordering.25 work of better quality, consider moving to a majority- • If the next scheduled item was good, RA1 rules process. In particular, each alcoholic item data- searched the list for a task posted by an employer collection costs an average of $0.03 in our study, and in which all attributes are green (all attributes are each item was coded correctly with a probability of greater than 3.0/5); 26.3% of the 23,031 employers p 0.890. If a third rater is used only if the first two reviewed on Turkopticon meet this criterion. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 15
• If the next scheduled item was bad, RA1 divided by the time to complete the task; they are not searched the list for a task posted by an employer zero if the work is rejected.27 Employers with good repu- with no green attributes and a red rating for pay (all tations have significantly lower rejection rates and attributes are less than 3.0/5, and pay is less than faster times to decisions. They do not have statistically 2.0/5); 21.6% of employers reviewed on Turkopticon different posted pay rates. This distinction is important meet this criterion. because the pay for accepted tasks is contractible but • If the next scheduled item was none, RA1 the task’s acceptance criteria and realistic time require- searched the list for a task posted by an employer ments are not. with no reviews. In principle, the ratings on Turkopticon could be 3. RA1 sent the task to the blinded RA2, who used a orthogonal to employer type and instead be pro- browser not equipped with Turkopticon. vidinginformationontasktypes(e.g.,surveyorphoto 4. RA2 performed and submitted the task. RA2 categorization) rather than employer types. We do was instructed to perform all tasks diligently.26 not find evidence that this is the case. First, Turkop- 5. RA1 and RA2 repeated steps 2–4. A web crawler ticon requests workers to rate employers on fairness, recorded payments and rejections by employers to communicativity, promptness, and generosity; unlike RA2’s account with accuracy within one minute of task type, these are revealed only after workers have actual payment or rejection. invested effort and are subject to hold-up. Textual The blinding procedure decouples the search comments also emphasize information that would process from the job performance process, thereby only be revealed to prospective workers after investing protecting against the risk that RA2 inadvertently effort. Second, the RA’s task classifications in exper- conditions effort on the employer’s reputation. iment 2 are not significantly correlated with em- ployers’ Turkopticon scores. We also found evidence 6.2. Results against workers screening on task rather than em- Figure 4 shows results for rejection rates and time to ployer in experiment 1. payment by the employer’s reputation type. Rejection Given the low cost of creating new employers, it is rates were 1.4% for employers with good reputations, puzzling that employers with poor reputations per- 4.3% for employers with no reputation, and 7.5% for sist rather than creating new accounts. When the study employers with bad reputations. was conducted, the only cost to creating a new em- Table 4 presents further results and significance ployer was the time filling forms and awaiting ap- tests for rejection rates, time to payment, and realized proval. Since then, the cost of producing new aliases hourly wage rates. We define realized wage rates to be has grown.28 If creating new accounts were perfectly payments divided by the time to complete the task if costless and employers were informed, we would ex- the work is accepted and zero if the work is rejected. pect there to be no active employers with poor repu- We define promised wage rates to be posted payments tations. However, Turkopticon’s textual reviews also
Figure 4. (Color online) Time to Payment and Rejection by Employer Reputation
Notes. Whiskers represent standard errors. p-values for a χ2 test that shares are independent of reputation are, respectively, 0.002, 0.011, 0.805, 0.012, and 0.007. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 16 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s)
Table 4. Rejection and Time to Payment by Employer Reputation
Paired test p-values
Mean Standard error N Good None Bad
Main outcomes 1. Rejection rates Good reputation 0.013 (0.008) 223 0.073 0.003 No reputation 0.043 (0.016) 164 0.073 0.246 Bad reputation 0.071 (0.018) 211 0.003 0.246 2. Days to decision Good reputation 1.679 (0.146) 223 0.132 0.001 No reputation 2.296 (0.433) 164 0.132 0.03 Bad reputation 3.715 (0.467) 211 0.001 0.03 3. Realized wage rates Good reputation 2.834 (0.228) 173 0.011 0.043 No reputation 1.957 (0.259) 141 0.011 0.949 Bad reputation 1.986 (0.352) 168 0.043 0.949 Other outcomes 4. Days to decision, accepts only Good reputation 1.643 (0.144) 220 0.083 0.001 No reputation 2.368 (0.451) 157 0.083 0.023 Bad reputation 3.943 (0.499) 196 0.001 0.023 5. Promised wage rates Good reputation 2.834 (0.228) 173 0.017 0.098 No reputation 2.011 (0.257) 141 0.017 0.771 Bad reputation 2.142 (0.352) 168 0.098 0.771 6. Advertised pay Good reputation 0.277 (0.025) 223 0.001 0.938 No reputation 0.159 (0.024) 164 0.001 < 0.001 Bad reputation 0.28 (0.022) 211 0.938 < 0.001
7. RA log-seconds to complete Good reputation 5.737 (0.228) 173 0.372 < 0.001 No reputation 5.639 (0.085) 141 0.372 0.001 Bad reputation 6.368 (0.069) 168 < 0.001 < 0.001
Notes. Rejection rate p-values are from a χ2 test that rejection rates are the same between the row and column. Time-to-pay p-values are from a two-sample t-test that the mean times to pay are the same between the row and column. Standard errors in parentheses. suggest that workers are aware that employers with 7. Natural Experiment bad reputations may create new identities. Experiments 1 and 2 demonstrate the effects of the We conclude that the longer work times and lower reputation system on employers and workers on the acceptance rates validate Turkopticon’sratings.Inother online market. So far, these results suggest that words, Turkopticon is informative about employer dif- workers can earn substantially more by screening ferences that would be unobservable (or at least more employers with good reputations, and employers with costly to observe) in the absence of the reputation system. better reputations attract workers more quickly (or, To provide an intuition for the magnitude of the alternatively, for a given speed, more cheaply) than value of employer-reputation information to workers, those with no or poor reputations. In this section, we note that our results imply that following a strategy of address the question of what happens to the job market doing jobs only for good-reputation employers would when the reputation system suddenly disappears. yield about a 40% higher effective wage than doing jobs for only no-reputation or bad-reputation em- 7.1. Ideal Experiment ployers: $2.83 versus just under $2.00 per hour. Re- Following Rubin (1974), we begin by describing an sults suggest about 20% of the gap in effective pay is ideal experiment that would identify the causal explained by nonpayment, and 80% is explained by partial-equilibrium effect of the reputation system on longer tasks. However, this calculation understates the market. Assume a researcher had the ability to the penalties when an employer rejects tasks because (1) shut down the reputation system at will and for the rejected worker is penalized in two ways: non- any periods of time and (2) monitor the entire market, payment and a lower approval rating. The latter re- including which jobs are being taken, how fast they duces the worker’s eligibility for future tasks from are finished, and so on. One could randomly assign other employers. thetimewhenthereputationsystemisremovedand Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 17
Table 5. Summary of Turkopticon Downtime Data respect to whether the reputation system was active at the moment. We measure work being done as the Variable Value “promised” payrateofagiventaskmultipliedbythe Number of downtime episodes 7.00 number of tasks that were done (rewards earned); we Average length of a downtime episode, hours 10.53 prefer this to the number of tasks alone because quick Average time between downtime episodes, days 61.54 taskstendtobecheap.Wecontrolfortimeofday,day Total range spanned by downtime episodes, days 369.27 of week, employer, and episode using fixed effects. More specifically, we use the following model: randomly decide for how long it is absent.29 Because log(1 + Rewards Earned )β + β DOWN + β H such a treatment assignment would be independent it 0 1 t 2 t + β + β +β + ε , of market conditions, one could conclude that any 3Dt 4Ri 5Et it changes observed in the market were caused by the (1) treatment. Acknowledging that it is infeasible and where RewardsEarned is the total promised pay to unethical to shut down the reputation system website it all workers working on task i at time t, DOWN is the purposefully, we use a natural experiment with ob- t indicator variable for whether the reputation system servational data, which serves as an approximation to is down at time t (DOWN 1) or not (DOWN 0), H the ideal experiment described. t t t is the fixed effect for the hour of the day at time t, Dt is fi 7.2. Observational Data the xed effect for the day of the week at time t, Ri is the fixed effect of the employer who requested task i, To explore the partial-equilibrium effect of reputation fi system absences, we exploit the seven instances when and Et is the xed effect for the downtime episode. the Turkopticon servers went down. To accomplish Theanalysissampleisrestrictedtoobservationsoc- that, we collected the following data: curring between two weeks before and after the start • Turkopticon downtime. We assembled data on of a downtime episode. Table 7 presents results. Turkoption’s downtime using time stamps from AsshowninTable7, the overall job consumption worker and Turkopticon administrative posts on on the market actually increasesasthereputation the Turkopticon website, Reddit, Twitter, and Google system shuts down. From this result, we conclude Groups. These are summarized in Table 5.Thechief that the workers tend to stay in the market when the concern is that Turkopticon’s downtimes are correlated reputationsystemisshutdownatleastintheshort with one of our variables, for example, because of term. There are a number of possible explanations for especially heavy traffic. However, all administrative this, not all of which necessarily correspond to higher posts attributed crashes to unrelated technical issues, pay among workers or better allocation of work. For example, workers might speed up work because such as software updates. ’ • Individual-task–level data on the entire market they re spending less time screening and reviewing that was collected by the web crawler M-Turk employers. In the short term, this might raise the Tracker (Ipeirotis 2010b, Difallah et al. 2015)andis amount of promised pay earned, but less of this summarized in Table 6.M-TurkTrackerscansthe promised pay may be realized (given experiment 1). In the long term, the lack of a reputation system may M-Turk market every six minutes and records the ’ fi status of all HITs that it observes, such as the number impair workers ability to nd good but small em- of tasks left in a particular HIT, the task description, ployers and may discourage smaller employers from andtherewardofferedbythetask.Bystudyingthe investing in a good reputation. changes in the number of tasks still left in each HIT, Table 7. Overall Effect of Reputation System Shutdown on we can explore how fast the jobs are taken and, thus, Job Consumption explore shifts in the supply of labor in this market. fi Our rst goal is to study the total effect of the Dependent variable: reputationsystemshutdownonthelabormarket.To log(Rewards Earned) do that, we examinee the amount of work done by DOWN 0.0034* (0.0004) M-Turk workers at any given moment in time with Hour of day fixed effect Yes Day of week fixed effect Yes fi Table 6. Summary of MTracker Data Employer xed effect Yes Downtime episode fixed effect Yes Variable Value Observations 5,572,840 R2 0.1422 Number of hit status observations 504,840,038 Adjusted R2 0.1419 Number of distinct requesters 65,243 Residual standard error 0.1109 (df = 5,570,882) Number of distinct crawls 267,319 Note. Average time between the crawls, minutes 12.07 Robust standard errors in parentheses. *p < 0.01. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 18 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s)
To examine whether workers’ job search changes, 2. Shift the percentile window by 5%; that is, pick we study the heterogeneity of the treatment effect. all good-reputation employers between the fifth and We want to separate reputation into two dimensions: bottom 30th percentiles of visibility. Estimate a new how good an employer’s public reputation is and how DOWN coefficient using only jobs for these employers, widely known or visible an employer is outside the denoted by DOWN5%,good.Plottheestimatedco- public reputation system. We measure the quality of efficient at 5% on the x-axis and in green. an employer’s reputation using Turkopticon reviews. 3. Shift by 5% again and repeat the procedure We measure the visibility of employer i at time t as the until DOWN75%,good is estimated, corresponding to number of times the MTurk-Tracker web crawler the top quartile by visibility of good-reputation em- encountered that employer across all time periods ployers, that is, the most-visible good-reputation before t.Thisisdesignedtocaptureworkers’ general employers. familiarity with the employer. Some employers (such 4. Repeat the entire procedure for bad-reputation , , as the brokers that use M-Turk to subcontract tasks on employers to estimate DOWN0%,bad DOWN5%,bad... behalf of their clients) become well known among DOWN75%,bad. M-Turk workers. However, many employers post These results suggest that the instantaneous effect jobs only infrequently. Independent of Turkopticon, of the reputation system varies by employer. Em- few workers have private knowledge of these less- ployers with bad reputations are relatively unaffected visible employers’ past behavior. An employer fre- by the downtime, consistent with these employers quently encountered by workers in their day-to-day attracting only workers who do not use the reputation browsing and work history would also tend to be system. Less-visible employers with good reputa- frequently encountered by the web crawler. On the tions are the most adversely affected. Results are other hand, if the web crawler (which runs every few consistent with these employers no longer being dis- minutes) encountered a particular employer only a covered by workers who use Turkopticon as a screen. handful of times then this employer would generally Most-visible employers with good reputations are posi- not be familiar to workers. tively affected as though workers using Turkopticon stop We performed a semiparametric test to examine screening for less-visible good-reputation employers and heterogeneity by employer reputation and visibility instead use the best-known, good-reputation employers based on the following procedure and plot the results as a fallback option. In other words, these results sug- in Figure 5: gest that Turkopticon aids in workers’ discovery of 1. Pick all the good-reputation employers in the small high-road employers and provides these em- lowest quartile of visibility, whose visibility is in the ployers an incentive to invest in their reputation. To first 25 percentiles.30 These are the least-visible good- extrapolate, we might expect that the reputation sys- reputation employers. Estimate the DOWN coeffi- tem promotes competition and prevents the market cient using only jobs for these employers. Denote it from devolving into a small, oligopsonistic set of well- fi DOWN0%,good.Plottheestimatedcoef cient at 0% on known employers because newer and smaller em- the x-axis with a green marker. ployers require substantial reputational investments to
Figure 5. (Color online) The Effect of Downtime Depends on Visibility of the Requester Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 19 become sufficiently well known to attract new workers flexible renegotiation provisions or subjective out- reliably. come standards. Ipeirotis (2010a)notesthatM-Turk’s In contrast to the ideal experiment, this study has poor enforcement likely limits the scope of work some caveats. Although we would like to study performed on the platform to very narrow tasks. Since whether the market would reach a new equilibrium our experiment, Amazon has begun requiring unique (e.g., it would collapse or become an oligopsony) in tax identification numbers, making it more difficult the absence of a reputation system, we only observe for employers to reset a bad reputation. relatively short, expected-to-be-temporary down- times that surely don’t allow employers to adapt their 8.2. Why Do Workers Contribute to the Public payment strategies endogenously or workers to ad- Reputation System? just their labor market response to such changes. We The coexistence of public and private reputation also cannot observe whether workers are actually systemsalsobegsthequestion:whydoworkers paid less for their work. We only observe promised pay- contribute to a collective memory when they can ments and the number of tasks performed. Nonethe- instead hoard private knowledge of the best em- less,theresultsproviderelatively clean evidence for ployers? Prior studies have also noted the collective the instantaneous effects of the reputation system on action problem that this entails (see, e.g., Levine and how workers search for jobs and how public and Prietula 2013 and Gao et al. 2015). Again, this liter- private reputation substitute for one another. ature almost exclusively focuses on ratings of sellers and service providers by buyers and clients. 8. Discussion Nonetheless, online labor markets and ratings of 8.1. Types of Reputation and Market Design employers are also a unique and instructive setting Our experimental design focuses on the value of a for understanding contributions to public ratings. In a publicly available employer reputation system and typical product market, goods are nonrival: when a not on workers’ private signals. To test Hypothesis 1, buyer favorably rates a product or seller on Amazon, we allowed workers to complete only one task. To that buyer’s ability to get future products or ser- test Hypothesis 2, we submitted only one task for each vicesisnothinderedbyanincrease in other buyers. employer. To test Hypothesis 3,weexaminedhow However, employers post finite numbers of tasks, completed work varied by public ratings and total and favorable reviews for smaller employers could visibility. Even so, private information remains im- lead other workers to consume those tasks. In this portant in this context, and other tools (e.g., Turk sense, worker reviews are costly to workers not only Alerts and DCI New HIT Monitor) are available to intermsoftime,butalsointhattheyattractother notify workers when a employer that they privately workers to a rival “good.” The ability of Turkopticon flag posts a job. Likewise, employers can privately to attract large numbers of raters shows that altruism invite workers to apply for future work. and volunteerism survive even under these conditions. Thecoexistenceofthesemoretraditionalmatches, Moreover, our results from our second experiment con- which rely on private information and repeated con- firm that these reviews are useful and informative. tracting, is arguably a reminder of the shortcomings of current rating systems. Indeed, the chief value prop- 8.3. Specific Puzzles from Our Empirical Results osition of such crowdsourcing platforms is to provide Each study features some empirical results that war- aquickandefficient method of connecting employers rant future attention. In experiment 1, we found that and workers to large numbers of trading partners. The good-reputation employers attract work more quickly inability to do so is a key concern for both employers with no loss of quality. However, good-reputation em- and workers. ployers might also get a reputation for paying regardless Amazon might do more to encourage workers and of work quality, leading workers to flock to these em- employers to use public ratings.Forinstance,Amazon ployers and then exert minimal effort. Although the could give workers access to historical information on analysis of workmanship in experiment 1 finds no each employer, such as average past wage and re- evidence for this margin of behavior, it remains an jection rates. It could also create a native, subjective open question whether employer reputation systems rating system, as Upwork has done and as Amazon has can also invite moral hazard. done for consumer products. The lack of information In experiment 2, why did effective wages for good- about employer reputations coupled with the lack of reputation employers exceed those for bad-reputation contract enforcement may be limiting the market to the employers? As noted in our review of the literature, small size that a reputation can discipline and to small studies have generally (although not always) found the tasks that are relatively short and well defined. For opposite result: good reputations allow trading part- instance, Guo et al. (2018) find that workers on online ners to extract more favorable terms, such as the ability markets avoid jobs that are difficult to codify or have to attract workers at lower pay. Such compensating Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 20 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) differentials may be impossible in this setting: al- Krueger (2017,p.13)reportsthataboutathirdof though pay-per-task is specified, the time to complete American workers spent some time in the prior week the task is not. Ratings may also capture other aspects “working or self-employed as an independent contractor, of the employer. Following Bartling et al. (2013), some independent consultant, or freelance worker,” including employers may be more altrustic; these employers “working on construction jobs, selling goods or ser- pay higher wages and also have better reputations. vices in their businesses, or working through a digital Indeed, Turkopticon ratings include an item for gen- platform, such as Uber, Upwork, or Avon,” and 84% erosity, which intends to capture expected wages. of these workers report self-employment as their A second alternative is that ratings implicitly capture main job. Among these workers, over a third report employers’ preferences for getting work done more “having an incident in the last year where someone quickly; impatient employers pay higher wages and hired you to do a job or project and you were not paid maintain good reputations to get work accomplished on time.” Over a quarter reported at least one incident quickly. of being unable to collect the full amount owed for a Finally, in the natural experiment, what would job or project that the worker completed. The Freelancers happen to the market if the reputation system remained Union has used both reputation and regulatory solu- down? Turkopticon’s downtimes suggest that smaller, tions to address client nonpayment, including its “client good-reputation employers are especially dependent scorecard” and a successful effort to lobby New York on Turkopticon to get work done quickly. Following City to pass the Freelance Isn’tFreeAct. the logic of our formal model, we may hypothesize that As on M-Turk, workers in the broader labor market the long-term loss of a reputation system would lead strive to distinguish which employers will treat them themarkettobecomeconcentratedamongthemost well or ill. Workers have always made decisions visible of the good-reputation employers although with partial information about employer quality, and smaller employers are deterred by the cost of estab- so these forces have always shaped labor markets. lishing a good reputation and may need to go through Contracts and bilateral relational contracting are third-party brokers (such as CrowdFlower) that have important forces disciplining employer opportunism, established reputations. but they are certainly incomplete. Workers have al- ways relied on public employer reputations propa- 8.4. Lessons for Reputation Systems in Off-Line gated through informal, decentralized, word-of-mouth Labor Markets conversations. Although economists have had theories What relevance do these findings have for other about how employer reputation would work, the in- markets? M-Turk workers are unconventional in that formal system has operated largely outside our view, they are contracted for very small tasks and have yielding a very thin empirical literature. As the cost of minimal interaction with firms. However, the issues communications and data storage fell in recent years, that they confront are more general. As Agrawal et al. employer reputation has become more centralized, (2015, p. 219) describe, “the growth of online markets systematic, and measurable, showing up in general for contract labor has been fast and steady.” labor market matching sites, such as Glassdoor.com Uber, TaskRabbit, DoorDash, and other online and Indeed.com, and in more specialized contexts, platforms are also blurring the boundaries between such as ProjectCallisto.org, which allows workers to off-line employment and entrepreneurship (Apte and share information about sexual harassment and abuse Mason 1995,Weil2014, Harris and Krueger 2015). at work, and Contratados.org, which allows migrant Although these platforms are also drawing increasing workers to review recruiters and employers. scrutiny from regulators, wage theft and other forms Attention to the worker’s information problem also of opportunism are also pervasive in other settings in suggests innovative directions for policy and in- which legal enforcement is weak, including among stitution building. Can more be done to improve the independent contractors, undocumented immigrants, functioning of the gig economy through helping workers misclassified employees, and low-wage employees overcome their information problem with respect to (Bobo 2011,Rodgersetal.2014). Wage theft has employer heterogeneity? Can we improve institutional prompted the U.S. Department of Labor’sWageand designs to better elicit, aggregate, and disseminate Hour Division to award back pay to an average of information about employers? Platform design affects 262,996 workers a year for the past 10 years, and far workers’ willingness to voluntarily contribute their more cases go unremedied (Bernhardt et al. 2009, private information to the public pool (Marinescu 2013;Bobo2011;Lifsher2014; U.S. Department of et al. 2018). A policy example of this kind of logic Labor Wage and Hour Division 2016). The value of in action is that, in 2009, the U.S. Occupational Safety stolen wages restored to workers through enforcement and Health Administration began systematically is- actions is larger than the total value stolen in all bank, gas suing press releases to notify the public about large station, and convenience store robberies (Lafer 2013). violations of workplace safety laws. This effort attempts Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 21 to influence employer reputation, to improve the flow and Chris Stanton; as well as workshop participants at the of information about employer quality, and to create ASSA Meetings, MEA-SOLE meetings, Minnesota Applied incentives for providing safer workplaces. Johnson Microeconomics Workshop, MIT Sloan Organizational Eco- (2019) found that it also induces competing employers nomics Lunch, MIT Conference on Digital Experimentation, to improve compliance with worker protection laws al- MIT Sloan IWER seminar, LERA meetings, Michigan Ross, Northwestern Law School, Organization Science Winter though the Trump administration rolled back this ef- Conference, Barcelona GSE Digital Economics Summer fort (Meier and Ivory 2017). Workers have traditionally Workshop, IZA World Labor Conference, NBER Personnel used labor unions and professional associations as a summer institute, GSU, the Advances in Field Experiments venue for exchanging information about working con- conference at BU, and Tennessee-Knoxville. Authorship is ditions and coordinating collective withdrawal of trade equal and alphabetical. This paper is an updated version to discipline employers. The rise of new institutions that of IZA Discussion Paper 9501. facilitate information sharing may be taking up some of this role. Endnotes Platforms can better deliver on their promise to 1 As discussed more later, we estimate that at least one half of the reduce matching frictions and increase efficiency to M-Turk labor supply installs Turkopticon although it remains a the extent that they help workers distinguish reliable puzzle why not all workers do so. Exogenous employer visibility employers. If that goal can be addressed, then the is also interesting: one might imagine that, in small communities, all market participants would have a well-established reputation falling costs of information processing and diffusion without the aid of any formal reputation system. M-Turk features may move labor markets closer to the competitive both a few large brokerages (e.g., Crowdflower) that constitute a ideal.31 Fulfilling this promise requires designing plat- substantial share of labor demand and a long tail of smaller forms that help workers to find great employers and requesters. avoid bad ones. 2 Perfect revelation of past payment simplifies the exposition. Board and Meyer-ter Vehn (2013) consider reputation building when learning is imperfect. Their model also yields ergodic shirking with 9. Conclusion increasing incentives for noncontractible investments as reputation Online platforms are making it cheaper to connect becomes noiseless. 3 trading partners, but issues of trust and reliability Workers may face the two standard kinds of information problems with respect to unobserved employer heterogeneity: adverse selec- remain. The empirical literature has focused almost tion and moral hazard. Employers’ technologies or product markets exclusively on sellers, including sellers of products may differ in ways that make low-road practices more or less (e.g., eBay brokers), services (e.g., restaurants), and profitable. In this adverse-selection setting, it is trivial to understand labor (e.g., contract workers on gig platforms). Labor why variation in employment practices emerges. An alternative markets have always faced bilateral uncertainty al- theory is that there is no essential heterogeneity between employers. thoughtherelativeabsenceofregulationhasmade Differences in strategic employment practices appear between competing employers (Osterman 2018). We focus on this more in- gig and online labor markets especially prone to teresting case. In all labor markets, both mechanisms are almost opportunistic employers. certainly empirically relevant. Cabral and Hortacsu (2010) did such This study provides a theoretical and empirical foun- an accounting in a consumer-goods market, baseball cards on EBay. dation to better understand how employer reputation We know of no analogous accounting in any labor market. That systems can partially substitute for legal and other remains for future work. 4 third-party contract enforcement. Moreover, the ex- Other studies show how reputation systems and credentials can improve efficiency in other online markets including eBay (Nosko perience of M-Turk and Turkopticon suggests that and Tadelis 2015, Hui et al. 2016) and Airbnb (Fradkin et al. 2015). reputation systems may have an important role to 5 Our model abstracts away from the real but very well-studied play in providing employers with incentives to treat employer information problem of dealing with workers who may workers well, giving lesser-known employers direct differ in quality or shirk. access to workers, and ultimately expanding the scope 6 Another approach is to let employers exogenously vary in their of work that can be completed online. Institutions and discount rates, in which case farsighted employers become high-road policies can combat opportunistic employers, but given employers. 7 the complexities of the employment relationship, it We could allow low-road employers to receive some work and some fi seems implausible that opportunism will ever be fully pro t by allowing visibility to yield an imperfect signal. Then s could arise endogenously by allowing employers to exogenously vary in eliminated. their discount rates; cheap and patient employers renege, rich and impatient employers pay. We omit this complexity because it adds Acknowledgments little insight, and our empirics are concerned with whether the signal The authors thank their excellent research assistants Harshil is valuable, not the degree to which it is precise. Chalal, Sima Sajjadiani, Jordan Skeates-Strommen, Rob 8 Available online at https://archive.fo/FaVE. Vellela, and Qianyun Xie. They also thank Panos Ipeirotis for 9 Available online at http://www.mturk-tracker.com (accessed June sharing M-Turk Tracker data. For their useful feedback, the 14, 2014). authors thank John Budd, Eliza Forsythe, Mitch Hoffman, 10 In legal terms, M-Turk is a brokerage that facilitates relationships John List, Colleen Manchester, Mike Powell, David Rahman, between two contracting parties: one that seeks work for pay and Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 22 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) another that performs work. We use “employer” as shorthand for the 23 Otherwise, responses emphasize estimated pay, estimated time to former. completion, and perceived trustworthiness (e.g., from a known or- 11 Popular resources include Reddit’s HitsWorthTurkingFor, ganization). To the extent one is interested in the effect of reputation CloudMeBaby.com, mturkforum.com, mturkgrind.com, turkalert.com, among informed workers, this treatment-on-treated effect is 82% . −1 turkernation.com, turkopticon.ucsd.edu. (from 0 55 ) larger than the estimated effect in the observed equi- librium. The estimated effect is a weighted average of a larger effect 12 For details on our estimates, see the end of the study 2 results among workers who use the reputation system and a zero effect section. Silberman et al. (2010), Irani (2012), and Silberman (2013) among those who don’t. provide background on Turkopticon. 24 Assuming errors are independent, the expected cost is 2c[p2 + 13 These statistics are based on our analysis of data scraped from the (1 − p)2]+3c[1 − p2 −(1 − p)2]. The probability of a correct decision site. Attribute ratings are determined by the mean from the following is p2 + 2(p2(1 − p)). questions: (i) for communicativity, “How responsive has this re- 25 ’ quester been to communications or concerns you have raised?”; To hold skill constant, the RA omitted any tasks requiring master s qualification. The task classifications were uncorrelated with Tur- (ii) for generosity, “How well has this requester paid for the amount kopticon scores. of time their HITs take?”; (iii) for fairness, “How fair has this requester 26 been in approving or rejecting your work?”; (iv) for promptness, RA2 was not able to complete all jobs sent by RA1. Some expired ’ “How promptly has this requester approved your work and paid?” quickly. Also, bad-reputation employers jobs were more likely to be Their means (standard deviations) are respectively 4.01 (1.68), 3.98 so dysfunctional as to be unsubmittable. 27 (1.62), 3.71 (1.68), and 3.18 (1.91), suggesting that ratings are mean- Counts are lower for wage rates because the blinded RA lost track ingfully spread. Their number of reviews are 93,596, 93,025, 99,437, and of time to completion for some tasks. 44,298. Reviews are somewhat consistent across dimensions; the cor- 28 On July 27, 2014, Amazon began requiring employers to post a relation between any one dimension and the mean value of the other legal personal or company name, physical address, and Social Se- three dimensions is 0.57. On workers’ displays, average ratings are curity number or employer identification number. color coded; scores less than two are red, scores between two and three 29 The control group is the time when the reputation system is up. are yellow, and scores greater than three are green. 30 Reputations are defined as in experiment 2. 14 This competition between workers to get the best jobs is the basis of 31 According to Manning (2011, p. 978), “If one thinks of frictions as resources such as TurkAlert.com, which allows workers to receive an being caused by a lack of awareness of where vacancies are . . . then alert whenever employers of their choosing post new tasks. one might have expected a large effect of the Internet. But if . . . one 15 The first names are Joseph, Mark, and Thomas. The last names are thinks of frictions as coming from idiosyncracies in the attractive- Adams, Clark, Johnson, Jordan, Kelly, Lewis, Martin, Miller, Owens, ness of different jobs . . . then one would be less surprised that the Roberts, Robinson, and Warren. effects of the Internet seem to be more modest.”
16 For this purpose, we define bad reviews as those giving a score of 1/5 on all rated attributes and a good review as giving a 4/5 or 5/5 on References all rated attributes. The text reviews clearly corroborate the numerical rankings; an RA given only the text reviews correctly identified the Agrawal A, Horton J, Lacetera N, Lyons E (2015) Digitization and employer type in 285 of the 288 reviews. the contract labor market: A research agenda. Goldfarb A, Greenstein SM, Tucker CE, eds. Economic Analysis of the Digital 17 At the time of the experiment, of the 23,095 employers rated on Economy (University of Chicago Press, Chicago), 219–250. Turkopticon, 22.9% met our criteria for being bad-reputation and Agrawal AK, Lacetera N, Lyons E (2013) Does information help or 48.1% met our definition of being good-reputation. Many of the hinder job applicants from less developed countries in online bottom ratings come from employers with few reviews. Of the 1,564 markets? NBER Working Paper No. 18720, National Bureau of – fi employers with 8 12 reviews, only 1.1% met our de nition of bad and Economic Research, Cambridge, MA. fi “ ” 41.5% met our de nition of good. In this sense, our good employers Apte UM, Mason RO (1995) Global disaggregation of information- had good ratings but not uncommonly so. However, the mean ratings intensive services. Management Sci. 41(7):1250–1262. of our bad employers are especially bad given the consensus across so Ba S, Pavlou PA (2002) Evidence of the effect of trust building many raters that they merit one on all dimensions. As such, our technology in electronic markets: Price premiums and buyer estimate of the effect of bad reputation (relative to good and no behavior. MIS Quart. 26(3):243–268. reputation) might be interpreted as an upper bound at which about Bajari P, Hortacsu A (2003) The winner’s curse, reserve prices, and half of workers are using this reputation system to screen employers. endogenous entry: Empirical insights from eBay auctions. 18 Alcoholic items came from a list of 25 bestselling beers. This task, RAND J. Econom. 34(2):329–355. therefore, features simple image recognition, abbreviation recogni- Banerjee AV, Duflo E (2000) Reputation effects and the limits of tion, and domain knowledge. contracting: A study of the Indian software industry. Quart. J. – 19 In both of these cases, the workers have no information of their own Econom. 115(3):989 1017. and, thus, base their decisions only on the available public reputation Barach M, Golden J, Horton J (2019) Steering buyers to selected that they see sellers: The role of platform incentives and credibility. Man- agement Sci. Forthcoming. 20 There are four six-hour slots in each day—12 a.m.–5 a.m., 6 a.m.–11 Bartling B, Fehr E, Schmidt KM (2013) Use and abuse of authority: a.m., 12 p.m.–5 p.m., 6 p.m.–11 p.m.—that roughly correspond to A behavioural foundation of the employment relation. J. Eur. night, morning, day, and evening shifts. Econom. Assoc. 11(4):711–742. 21 Thursday is the last day of the three days of the experiment. Bernhardt A, Spiller MW, Theodore N (2013) Employers gone rogue: 22 Differences are for a two-sample t-test for equal means of the log- Explaining industry variation in violations of workplace laws. work time with α < 0.1. Error-free receipts are those in which all Indust. Labor Relations Rev. 66(4):808–832. alcoholic items were identified, no nonalcoholic items were identi- BernhardtA,MilkmanR,TheodoreN,HeckathornD,AuerM, fied, and the prices were entered correctly. Major-error receipts are DeFilippis J, Gonzalez AL, et al. (2009) Broken Laws, Unpro- those in which no alcoholic items were identified or more than six tected Workers (NELP National Employment Law Project, New items are listed. York). Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s) 23
Board S, Meyer-ter Vehn M (2013) Reputation for quality. Econo- Klein B, Leffler KB (1981) The role of market forces in assuring metrica 81(6):2381–2462. contractual performance. J. Political Econom. 89(4) 615–641. Bobo K (2011) Wage Theft in America (New Press, New York). Krueger A (2017) Independent workers: What role for public policy? Brown J, Matsa DA (2016) Boarding a sinking ship? An investigation Ann. Amer. Acad. Political Soc. Sci. 675(1):8–25. of job applications to distressed firms. J. Finance 71(2):507–550. Lafer G (2013) The legislative attack on American wages and labor Cabral L, Hortacsu A (2010) The dynamics of seller reputation: Ev- standards, 2011–2012. Economic Policy Institute Briefing Paper idence from ebay. J. Indust. Econom. 58(1):54–78. 364, Economic Policy Institute, Washington, DC. Chauvin KW, Guthrie JP (1994) Labor market reputation and the Levine SS, Prietula MJ (2013) Open collaboration for innovation: value of the firm. Managerial Decision Econom. 15(6):543–552. Principles and performance. Organ. Sci. 25(5):1414–1433. Cornes R, Sandler T (1994) The comparative static properties of the Lifsher M (2014) California cracks down on wage theft by employers. impure public good model. J. Public Econom. 54(3):403–421. Los Angeles Times (October 23), https://www.latimes.com/ Dellarocas C, Wood CA (2008) The sound of silence in online feed- business/la-fi-wage-theft-action-20141024-story.html. back: Estimating trading risks in the presence of reporting bias. List J, Momeni F (2017) When corporate social responsibility back- Management Sci. 54(3):460–476. fires: Theory and evidence from a natural field experiment. Difallah DE, Catasta M, Demartini G, Ipeirotis PG, Cudré-Mauroux P NBER Working Paper No. 24169, National Bureau of Economic (2015) The dynamics of micro-task crowdsourcing: The case of Research, Cambridge, MA. Amazon MTurk. Gangemi A, Leonardi S, Panconesi A, eds. Proc. List JA (2006) The behavioralist meets the market: Measuring social – 24th Internat. Conf. World Wide Web (ACM, New York), 238 247. preferences and reputation effects in actual transactions. J. Poli- Dube A, Jacobs J, Naidu S, Suri S (2018) Monopsony in online labor tical Economy 114(1):1–37. markets. NBER Working Paper No. 24416, National Bureau of Luca M (2016) Reviews, reputation, and revenue: The case of Economic Research, Cambridge, MA. Yelp.com. Harvard Business School Working Paper No. 12-016, Farronato A, Fradkin A, Larsen B, Brynjolfsson E (2018) Consumer Harvard University, Cambridge, MA. protection in an online world: When does occupational licensing Manning A (2011) Imperfect competition in the labor market. matter? Working paper, Harvard University, Cambridge, MA. Card D, Ashenfelter O, eds. Handbook of Labor Economics, vol. 4, fl Filippas A, Horton JJ, Golden J (2018) Reputation in ation. Proc. 2018 part B (Elsevier, Amsterdam), 973–1041. – ACM Conf. Econom. Comput. (ACM, New York), 483 484 Marinescu I, Klein N, Chamberlain A, Smart M (2018) Incentives can Fradkin A, Grewal E, Holtz D, Pearson M (2015) Bias and reciprocity reduce bias in online reviews. NBER Working Paper No. 24372, fi in online reviews: Evidence from eld experiments on Airbnb. National Bureau of Economic Research, Cambridge, MA. Proc. 16th ACM Conf. Econom. Comput. Roughgarden T, ed. (ACM, Mason W, Watts DJ (2010) Financial incentives and the performance New York), 641. of crowds. ACM SigKDD Explorations Newsletter 11(2):100–108. Gao GG, Greenwood BN, Agarwal R, McCullough JS (2015) Vocal McDevitt RC (2011) Names and reputations: An empirical analysis. minority and silent majority: How do online ratings reflect Amer. Econom. J. Microeconom. 3(3):193–209. population perceptions of quality? MIS Quart. 39(3):565–589. Meier B, Ivory D (2017) Federal rules on worker safety and re- Guo X, Gong J, Pavlou P (2018) Enhancing the “call for bids” to cord keeping are likely targets for rollbacks. New York Times improve matching efficiency in online labor markets: Capturing (March 13), https://www.nytimes.com/2017/03/13/business/ the meaning of unstructured textual content with machine learn- us-worker-safety-rules-osha.html. ing. Working paper, Temple University, Philadelphia. Moreno A, Terwiesch C (2014) Doing business with strangers: Reputation Hannon JM, Milkovich GT (1996) The effect of human resource in online service marketplaces. Inform. Systems Res. 25(4):865–886. reputation signals on share prices: An event study. Human Re- source Management 35(3):405–424. Nagaraj A (2016) Does copyright affect reuse? Evidence from the Google books digitization project. Working paper, University Harris SD, Krueger AB (2015) A proposal for modernizing labor laws for twenty-first-century work: The independent worker. The of California, Berkeley, Berkeley. Hamilton Project Discussion Paper 10, Brookings Institution, Nosko C, Tadelis S (2015) The limits of reputation in platform fi Washington, DC. markets: An empirical analysis and eld experiment. NBER Horton J (2019) Buyer uncertainty about seller capacity: Causes, con- Working Paper No. 20830, National Bureau of Economic Re- sequences, and a partial solution. Management Sci.,ePubaheadof search, Cambridge, MA. print May 6, https://doi.org/10.1287/mnsc.2018.3116. Osterman P (2018) In search of the high road: Meaning and evidence. – Horton JJ, Chilton LB (2010) The labor economics of paid crowd- ILR Rev. 71(1):3 34. sourcing. Proc. 11th ACM Conf. Electronic Commerce (ACM, New Oyer P, Schaefer S (2011) Personnel economics: Hiring and incentives. York), 209–218. Card D, Ashenfelter O, eds. Handbook of Labor Economics, vol. 4, – Hui X, Saeedi M, Shen Z, Sundaresan N (2016) Reputation and part B (Elsevier, Amsterdam), 1769 1823. fi regulations: Evidence from eBay. Management Sci. 62(12): Pallais A (2014) Inef cient hiring in entry-level labor markets. Amer. – 3604–3616. Econom. Rev. 104(11):3565 3599. Ipeirotis PG (2010a) A plea to Amazon: Fix Mechanical Turk. Accessed Resnick P, Zeckhauser R (2002) Trust among strangers in internet July 9, 2019, https://www.behind-the-enemy-lines.com/2010/10/ transactions: Empirical analysis of eBay’s reputation system. plea-to-amazon-fix-mechanical-turk.html. Econom. Internet E-commerce 11(2):23–25. Ipeirotis PG (2010b) Analyzing the Amazon Mechanical Turk mar- Rodgers WM, Horowitz S, Wuolo G (2014) The impact of client ketplace. XRDS Crossroads ACM Magazine Students 17(2):16–21. nonpayment on the income of contingent workers: Evidence Irani L (2012) Microworking the crowd. Limn (2):https://escholarship from the Freelancers Union independent worker survey. Indust. .org/uc/item/7vc0r3sh. Labor Relations Rev. 67(3 suppl):702–733. Johnson MS (2019) Regulation by shaming: Deterrence effects of Rosenblat A, Levy KE, Barocas S, Hwang T (2017) Discriminating publicizing violations of workplace safety and health laws. tastes: Uber’s customer ratings as vehicles for workplace dis- Working paper, Duke University, Durham, NC. crimination. Policy Internet 9(3):256–279. Jøsang A, Ismail R, Boyd C (2007) A survey of trust and reputation Ross J, Zaldivar A, Irani L, Tomlinson B (2009) Who are the Turkers? systems for online service provision. Decision Support Systems Worker demographics in Amazon Mechanical Turk. Technical 43(2):618–644. report, University of California, Irvine, Irvine. Benson, Sojourner, and Umyarov: Can Reputation Discipline the Gig Economy? 24 Management Science, Articles in Advance, pp. 1–24, © 2019 The Author(s)
Rubin DB (1974) Estimating causal effects of treatments in ran- Stanton C, Thomas C (2015) Landing the first job: The value of in- domized and nonrandomized studies. J. Educational Psych. 66(5): termediaries in online hiring. Rev. Econom. Stud. 83(2):810–854. 688–701. Stinchcombe AL (1965) Social structure and organizations. March JG, Silberman MS (2013) Dynamics and governance of crowd work ed. Handbook of Organizations (Rand McNally, Chicago), 142–193. markets. Presentation, International Workshop on Human Turban DB, Cable DM (2003) Firm reputation and applicant pool Computation and Crowd Work, September 30, Karlsruhe, characteristics. J. Organ. Behav. 24(6):733–751. Germany. U.S. Department of Labor Wage and Hour Division (2016) Working for Silberman MS, Irani L (2016) Operating an employer reputation afairday’s pay. Accessed July 9, 2019, https://archive.fo/DGVX1. system: Lessons from Turkopticon, 2008–2015. Comparative Labor U.S. Government Accountability Office (2015) Contingent work- Law Policy J. 37(3):472–505. force: Size, characteristics, earnings, and benefits. Accessed Silberman MS, Ross J, Irani L, Tomlinson B (2010) Sellers’ problems July 9, 2019, https://www.gao.gov/products/GAO-15-168R. in human computation markets. Proc. ACM Sigkdd Workshop Weil D (2014) The Fissured Workplace (Harvard University Press, Human Comput. (ACM, New York), 18–21. Cambridge, MA).