Subcontracting Microwork CHI 2017 Submission

Subcontracting Microwork Meredith Ringel Morris1, Jeffrey P. Bigham2, Robin Brewer3, Jonathan Bragg4, Anand Kulkarni5, Jessie Li2, and Saiph Savage6 1Microsoft Research, Redmond, WA, USA 2Carnegie Mellon University, Pittsburgh, PA, USA 3Northwestern University, Evanston, IL, USA 4University of Washington, Seattle, WA, USA 5University of California, Berkeley, CA, USA 6West Virginia University, Morgantown, WV, USA [email protected], {bigham, jessieli}@cs.cmu.edu, [email protected], [email protected], [email protected], [email protected] ABSTRACT scientific enterprises [10, 11], and community or volunteer Mainstream crowdwork platforms treat microtasks as ventures [5]. Mainstream crowdwork platforms treat indivisible units; however, in this article, we propose that microtasks (also called human intelligence tasks or HITs) as there is value in re-examining this assumption. We argue that indivisible units; however, we propose that there is value in crowdwork platforms can improve their value proposition re-examining this assumption. We argue that crowdwork for all stakeholders by supporting subcontracting within platforms can improve their value proposition for all microtasks. After describing the value proposition of stakeholders by supporting subcontracting within subcontracting, we then define three models for microtask microtasks, i.e. outsourcing one or more aspects of a subcontracting: real-time assistance, task management, and microtask to additional workers. task improvement, and reflect on potential use cases and implementation considerations associated with each. Finally, After describing the value proposition of subcontracting, we we describe the outcome of two tasks on Mechanical Turk then define three models for subcontracting microtasks: real- meant to simulate aspects of subcontracting. We reflect on time assistance, task management, and task improvement, the implications of these findings for the design of future and reflect on potential use cases and implementation crowd work platforms that effectively harness the potential considerations associated with each. Finally, we describe the of subcontracting workflows. outcome of two tasks on Mechanical Turk meant to simulate aspects of subcontracting; these simple HITs demonstrate the Author Keywords potential of subcontracting for improving aspects of Crowdsourcing; microwork; subcontracting; human crowdwork such as making workers and/or requesters aware computation; task selection; task design. of accessibility bugs, and for better matching task ACM Classification Keywords components to workers’ skills and interests. We reflect on H.5.3. Group and Organization Interfaces. the implications for the design of future crowd work platforms that effectively harness the potential of INTRODUCTION subcontracting workflows. Microwork, small tasks performed on crowd work platforms (e.g., Amazon’s Mechanical Turk, Upwork, LeadGenius, Value Proposition SamaSource, and many others [37]), is an increasingly We propose that enabling subcontracting of microwork, if important form of digital labor, representing both a valued, properly implemented, can enhance crowdwork processes flexible job opportunity for workers [39] and a viable means and outcomes. Subcontracting can provide value from the of accomplishing work at scale for businesses [17, 31], perspective of requesters, workers, and platform owners. Value for Workers Permission to make digital or hard copies of all or part of this work for From the perspective of a subcontracting-enabled crowd personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies platform, there are two classes of workers, whom we will bear this notice and the full citation on the first page. Copyrights for refer to as the primary worker (the original worker accepting components of this work owned by others than the author(s) must be a task, who may initiate subcontracting), and secondary honored. Abstracting with credit is permitted. To copy otherwise, or workers (the workers who perform the subcontracted republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. microwork). CHI 2017, May 06 - 11, 2017, Denver, CO, USA Microtaskers performing primary worker roles in a Copyright is held by the owner/author(s). Publication rights licensed to ACM. subcontracting-enabled system would have added ACM 978-1-4503-4655-9/17/05…$15.00 responsibilities that bring opportunities to begin to learn new DOI: http://dx.doi.org/10.1145/3025453.3025687 skills, such as management and task design, providing Note that it is not necessarily the case that subcontracting scaffolding for career growth, a dynamic currently lacking crowd work results in workers interacting with sub- from most mainstream crowd platforms [20]. The added components of an original microtask/HIT (e.g., working on responsibility associated with the role of primary workers a component that has been doled out by a primary worker in should result in increased wages for these roles, in an the task management model). For instance, in some cases economically fair system [1]. Subcontracting also secondary workers might simply support a complete, original fundamentally enables new work types within platforms, HIT (e.g., in the real-time assistance model). Subcontracting such as task labeling, headhunting, management, etc.; these could even enlarge the original HIT to be more than it first roles might encourage more diverse audiences to engage embodied (e.g., the addition of metadata in the task with crowd work by offering tasks that better match their improvement model). skills and/or interests. While not explicitly focused on the need for subcontracting, Secondary (subcontracted) work can offer low-reputation several prior systems and studies illustrate scenarios in which workers (e.g., platform newcomers, workers lacking skills one or more of our three subcontracting models may be with tasks of certain categories, etc.) a path to building skills useful and/or have implemented a specific workflow that and reputation. Secondary work can also lower the barrier for exemplifies these different types of subcontracting. We novice or casual workers by allowing them to invest less time discuss this prior work in the context of each of the three and effort in task selection, by instead following a trusted subcontracting models. primary worker or by benefiting from task decomposition Real-Time Assistance and metadata added by other secondary workers. Real-time assistance encompasses a model of subcontracting Value for Requesters in which the primary worker engages one or more secondary One potential benefit of a subcontracting-based system for workers to provide real-time advice, assistance, or support task requesters is that it may allow them to reduce the time during a task. Such interactions might occur with the and effort necessary for optimizing the deconstruction of integration of widgets for real-time assistance, and might tasks into microtasks, or of adding metadata or instructions include easy integration of video-conferencing, screen to tasks, as such tasks may be delegated to workers sharing, VOIP calls, live-streaming, instant-messaging, or themselves through the subcontracting structure (see the other types of real-time communications for task sharing. “Models of Subcontracting” section for more in-depth For instance, Kobayashi et al. [21] and Brewer et al. [6] explanations of these processes). postulate IM or video chat interactions with experienced Additionally, for requesters the addition of a layer of crowd workers during HIT execution may be necessary for indirection through subcontracting mitigates the risk and supporting older adults (or other novice or less tech-savvy workload involved in working with a full crowd of workers) in understanding complex task interfaces or potentially unreliable or untrusted workers, as some of this workflows. Such chat support would constitute a type of real- risk is now delegated to primary workers. This benefit is time assistance subcontracting, since the support-giver is not analogous to the benefit of using a contracting or temp the original worker who accepted the HIT, but is facilitating agency in the traditional economy – the temp agency its completion. Similarly, real-time assistance could help manager is analogous to the primary worker, while facilitate task completion (or better task quality) by individual temps map to secondary workers. connecting workers with people with whom they can discuss and clarify interpretations of task instructions, or with people Value for Platforms Subcontracting can benefit platform operators by enabling a who may have domain expertise or other skills relevant to more efficient and productive work system. Via completion of a task. subcontracting’s potential to enhance task design, better Zyskowski et al. [39] noted that some crowd workers with match workers with tasks, and reduce rates of partially- disabilities have to leave tasks incomplete due to difficulties completed tasks, platforms can expect to see better encountered mid-way through (e.g., a blind participant throughput and higher-quality work outcomes. By discovering that one sub-component of a task requires empowering workers and offering them more flexibility and examining an uncaptioned image). Real-time assistance initiative within their work, platforms may

Load more