Eliciting Worker Preference for Task Completion Mohammad Esfandiari #1, Senjuti Basu Roy #1, Sihem Amer-Yahia ∗2 # NJIT, USA 1 [email protected], [email protected] ∗ Univ. Grenoble Alpes, CNRS, LIG, France 2 [email protected] Abstract—Current crowdsourcing platforms provide little sup- In this paper, our objective is to design a framework that port for worker feedback. Workers are sometimes invited to advocates for explicit preference elicitation from workers to post free text describing their experience and preferences in develop a model that guides task completion. That differs completing tasks. They can also use forums such as Turker Nation1 to exchange preferences on tasks and requesters. In fact, from developing solutions for task assignment. The objective crowdsourcing platforms rely heavily on observing workers and behind preference elicitation is to use obtained feedback to inferring their preferences implicitly. In this work, we believe effectively maintain a Worker Model. Given a task t that a that asking workers to indicate their preferences explicitly improve worker w undertakes (either via self-appointment or via an their experience in task completion and hence, the quality of their assignment algorithm), there could be one of two possible contributions. Explicit elicitation can indeed help to build more accurate worker models for task completion that captures the outcomes : 1. the task is completed successfully. 2. otherwise. evolving nature of worker preferences. We design a worker model In reality, worker preferences are latent, i.e., they are to whose accuracy is improved iteratively by requesting preferences be inferred through task factors and task outcomes. Popular for task factors such as required skills, task payment, and task platforms, such as Mechanical Turk or Prolific Academic,3, relevance. We propose a generic framework, develop efficient have characterized tasks using factors, such as type, payment solutions in realistic scenarios, and run extensive experiments that show the benefit of explicit preference elicitation over implicit and duration. While our framework is capable to consume any ones with statistical significance. available task factors, our effort nevertheless is to propose a generic solution that characterizes workers by understanding I. INTRODUCTION their preferences for a given set of task factors [3], [4]. Our overarching goal is to seek feedback from workers to The main actors of a crowdsourcing platform are tasks and effectively maintain a Worker Model for task completion. To workers who complete them. A range of studies point out the achieve this goal, the first challenge is to define an accurate importance of designing incentive schemes, other than finan- model that predicts, per worker, how much each task factor is cial ones, to encourage workers during task completion [1], responsible for the successful completion of that task or for [2]. In particular, it is expected that a crowdsourcing system its failure. We propose to bootstrap this model by selecting should “achieve both effective task completion and worker a small set of tasks a worker needs to complete initially to satisfaction”. The ability to characterize the workforce with learn her model. An equally important challenge is to update factors that influence task completion is recognized to be the Worker Model, as workers complete tasks. Indeed, unless of great importance in building such a system [3], [4], [5], that model is updated periodically, it is likely to become [6], [7], [8], [9], [10], [11]. Those efforts have focused on outdated, as worker’s preferences evolve over time. To update implicitly observing workers and inferring their preferences. In the model, we advocate the need to explicitly elicit from a this paper, we argue that solely relying on implicit observations worker her preferences. That is a departure from the literature does not suffice and propose to elicit preferences from workers where workers are observed and their preferences computed explicitly, as they complete tasks. Any computational model, arXiv:1801.03233v1 [cs.DB] 10 Jan 2018 implicitly. We claim that the explicit elicitation of preferences designed for the workers to understand their task completion results in a more accurate Worker Model. Preferences are likelihood needs to consume worker preference. The evolving elicited via the Question Selector that selects a set of k task nature of worker preference requires to periodically ask work- factors and asks a worker w to rank them. For example, a ers and refine such models. To the best of our knowledge, this worker may be asked “Rank task relevance and payment”.A work is the first to examine the benefit of explicit preference higher rank for payment will indicate the worker’s preference elicitation from workers and its impact on effective task for high paying tasks over those most relevant to her profile. completion. Our proposed approach of explicit preference Once the worker provides her preference, the Worker Model elicitation does not incur additional burden to the workers; is updated with the help of the Preference Aggregator. in fact, platforms such as, Amazon Mechanical Turk2 already Question Selector and Preference Aggregator constitute the seek worker feedback in the form of free text. Our effort is to two computational problems of our framework. judiciously select questions for elicitation in a stuctured and holistic fashion. Worker Model. Our natural choice is to use a graphical model [12], such as a Bayesian Network where each node 1http://turkernation.com/ 2https://www.mturk.com/ 3https://www.prolific.ac/ is a random variable (task factors/worker preference/task out- tion. We present an innovative formulation to bootstrap come) and the structure of the graph expresses the conditional the model. An important aspect of the model is that it dependence between those variables. The observed variables could be easily adapted to other crowdsourcing processes, are task factors and task outcomes and the Worker Model such as, task assignment or worker compensation. We contains worker preferences in the form of latent variables present two core problems around the model: Question that are inferred through that model. It is however known Selector that asks a worker to rank the k task factors that structure learning in Bayesian Network is NP-hard [12] that cause the highest error in the model, and Preference and that the parameters could be estimated through methods Aggregator that updates the model with elicited prefer- such as Expectation Maximization, that also are computa- ences. tionally expensive. As both Question Selector and Prefer- • Technical Results (Section III): We study the hardness ence Aggregator have to invoke the model many times, it of our problems and their reformulation under realistic becomes prohibitively expensive to use it in real time. We assumptions, as well as design efficient solutions with therefore propose a simplified model that has a one-to-one provable guarantees. correspondence between task factors and worker preference. • Experimental Results (Section IV): We present extensive The preference of a worker for a task factor is construed as a experiments that corroborate that explicit preference elic- weight and the Worker Model becomes a linear combination itation outperforms implicit preferences [13], and that our of the task factors. This simplification allows us to design framework scales well. efficient solutions. Question Selector. The question selector intends to select II. FORMALISM AND FRAMEWORK the k-task factors whose removal maximizes the improvement We present our formalism, following which we provide an of the Worker Model F. The idea is to present those uncertain overview of the proposed framework and the problems we factors to the worker and seek her explicit preference. We tackle. prove that optimally selecting k questions, i.e., k task factors Example 1: We are given a set of tasks, where each task is for a worker, is NP-hard, even when the Worker Model is characterized by a set of factors (e.g., type, payoff, duration are linear. We develop an efficient alternative using an iterative some examples). A task could be of different types, such as, greedy algorithm that has a provable approximation bound. image tagging, ranking, sentiment analysis. Payoff determines Preference Aggregator. The second technical problem is to the $ value the workers receives as payment, whereas, duration update the Worker Model with the elicited preference. Given is an indication of the time a worker needs to complete that a set of k task factors, worker w provides an absolute order task. Generalizing this, one can imagine that each task could on these factors. The obtained ranking is expressed as a set be described as a vector of different factors and a set of tasks of k(k − 1)=2 pairwise linear constraints, as i > j, i > l, together gives rise to a task factor matrix T f . One such matrix etc. We design an algorithm that updates the Worker Model of 6 tasks is presented below: using the same optimization function as the one used to build it 8id tagging ranking sentiment payoff duration outcome9 initially, modified by adding those constraints. With a Bayesian > > >t1 1 0 0 high long 1 > Network as the underlying model, the addition of dummy <>t2 1 0 0 low short 0 => variables would encode the constraints aptly. However, with t3 0 1 0 low short 1 >t4 0 1 0 low long 0 > one variable per constraint, the solution would not scale. For > > :>t5 0 0 1 high short 1 ;> the simplified linear Worker Model, we add them as pairwise t6 0 0 1 high long 0 linear constraints. The problem then becomes a constrained Given a task t that a worker w undertakes (either via self- least squares problem that could be solved optimally in poly- appointment or via an assignment algorithm), there could be nomial time. one of two possible outcomes : 1. the task is completed We run experiments that measure the accuracy of our successfully (denoted by 1).