Achieving Reliability and Fairness in Online Task Computing Environments

UNIVERSIDAD CARLOS III DE MADRID TESIS DOCTORAL ACHIEVING RELIABILITY AND FAIRNESS IN ONLINE TASK COMPUTING ENVIRONMENTS Autor Evgenia Christoforou IMDEA Networks Institute & Universidad Carlos III de Madrid Co-Director Antonio Fernandez´ Anta IMDEA Networks Institute Co-Director & Tutor Angel Sanchez´ Universidad Carlos III de Madrid DOCTORADO EN INGENIERIA´ MATEMATICA´ DEPARTAMENTO DE MATEMATICAS´ Leganes´ (Madrid), Junio de 2017 UNIVERSIDAD CARLOS III DE MADRID PH.D. THESIS ACHIEVING RELIABILITY AND FAIRNESS IN ONLINE TASK COMPUTING ENVIRONMENTS Author Evgenia Christoforou IMDEA Networks Institute & Universidad Carlos III de Madrid Co-Director Antonio Fernandez´ Anta IMDEA Networks Institute Co-Director & Tutor Angel Sanchez´ Universidad Carlos III de Madrid DOCTORATE IN MATHEMATICAL ENGINEERING DEPARTMENT OF MATHEMATICS Leganes´ (Madrid), June 2017 Achieving Reliability and Fairness in Online Task Computing Environments A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Prepared by Evgenia Christoforou, IMDEA Networks Institute & Universidad Carlos III de Madrid Under the advice of Antonio Fernandez´ Anta, IMDEA Networks Institute Angel Sanchez,´ Universidad Carlos III de Madrid Departamento de Matematicas,´ Universidad Carlos III de Madrid Date: Junio, 2017 Web/contact: [email protected] This work has been supported by IMDEA Networks Institute and the Spanish Ministry of Educa- tion grant FPU2013-03792. VI TESIS DOCTORAL ACHIEVING RELIABILITY AND FAIRNESS IN ONLINE TASK COMPUTING ENVIRONMENTS Autor Evgenia Christoforou, IMDEA Networks Institute & Universidad Carlos III de Madrid Co-Director Antonio Fernandez´ Anta, IMDEA Networks Institute Co-Director & Tutor Angel Sanchez,´ Universidad Carlos III de Madrid Firma del tribunal calificador: Presidente: Alberto Tarable Vocal: Juan Julian´ Merelo Guervos´ Secretario: Jose´ A. Cuesta Calificacion:´ Leganes,´ 25 de Mayo de 2017 To my father, a great man and a great scientist... Θάρρος Στα δύσκολα εις την ζωήν, να μεν λιποτακτήσεις, ούτε τζιαι την προσπάθειαν, στην μέση να αφήσεις. Μες τη ζωήν την δύσκολην, ποττέ μεν σταματήσεις, τζι’ αν κάμεις τζιαι διακοπήν, πάλαι να ξεκινήσεις. Αφού εγεννηθήκαμεν, μπροστά μας να θωρούμεν, ούτε με σι!ννια τζιαι βροσιές, πρέπει να σταματούμεν. #ζι’ αν αιστανθε$ς τον κ$ντ%νον, εσού να μεν κολόσεις, ούτε τζιαι τα πιστεύω σο%, ποττέ σο% να προδ&σεις. Θάρρος, Χαράλαμπος Χριστοφόρου, 2013 Acknowledgements Before presenting this work, I would like to take a moment of the reader’s time to thank the people who have supported and guided me through this process. First and foremost I would like to thank my parents, without them nothing of this would have been possible. They gave me a great opportunity in life by giving me an education, which is my “fortune” as they keep saying, and for that I am forever grateful. I would like to thank my mother, one of the strongest women I have ever met in my life, for her endless support, her endless careering and her endless love. Words are not enough to describe my father’s support and guidance throughout my life. He is the kindest person I ever met and losing him was the hardest moment of my life. Dedicating this thesis to him is the least I can do for the person that taught me how to have courage in life. Moreover, I would like to thank my brother and my sister for being there and sharing every step of our life together. Moving on, I would like to acknowledge my thesis directors. It is not an exaggeration to say that without them this work would not have been possible. They have helped mature as a scientist in every aspect, but one of the greatest lessons that Antonio gave me was to think out of the box, while Angel (Anxo) taught to have right-thinking. I would like to give a special thanks to Chryssis Georgiou, for being my first supervisor and showing me how to be a researcher. Moreover, I would like to thank my co-authors Miguel Mosteiro for the excellent collaboration all these years; Agustin Santos´ for the amazing energy that he brings into the work; Kishori Konwar for sharing his amazing ideas and last but not least Nicolas Nicolaou for amount of time we spent working together and for the endless brainstorming. A especial thanks goes out to four very dear friends of mine that are always there when I need them to support me and take my tension away. Voula with her infinite passion for life being always an optimist; Nuri always being a realist; Kalia with her absolute kindness; and Andri always there to listen and support. I would also like to thank Dario, Alessia, Gaia, Maurizio and Paolo for the great dinner company, enjoying together the Madrid tapas experience and Miguel for joining us to the adventurous salsa classes. I could not help but saying a great thanks to all my IMDEA current and former colleagues for sharing a coffee break and having a great team attitude. A particular thanks goes out to Pablo that although he is far he never stopped caring. I would like to take a final moment to thank my partner in life Christian, that has been my pillar of strength. He has supported me through out this process not only by being my best friend but also by being my most sever critic. He has been encouraging me to continue through my toughest moments with every possible mean, without him this work would have never been completed. XI Abstract We consider online task computing environments such as volunteer computing platforms run- ning on BOINC (e.g., SETI@home) and crowdsourcing platforms such as Amazon Mechanical Turk. We model the computations as an Internet-based task computing system under the master- worker paradigm. A master entity sends tasks across the Internet, to worker entities willing to perform a computational task. Workers execute the tasks, and report back the results, completing the computational round. Unfortunately, workers are untrustworthy and might report an incorrect result. Thus, the first research question we answer in this work is how to design a reliable master- worker task computing system. We capture the workers’ behavior through two realistic models: (1) the “error probability model” which assumes the presence of altruistic workers willing to provide correct results and the presence of troll workers aiming at providing random incorrect results. Both types of workers suffer from an error probability altering their intended response. (2) The “rationality model” which assumes the presence of altruistic workers, always reporting a correct result, the presence of malicious workers always reporting an incorrect result, and the presence of rational workers following a strategy that will maximize their utility (benefit). The rational workers can choose among two strategies: either be honest and report a correct result, or cheat and report an incorrect result. Our two modeling assumptions on the workers’ behavior are supported by an experimental evaluation we have performed on Amazon Mechanical Turk. Given the error probability model, we evaluate two reliability techniques: (1) “voting” and (2) “auditing” in terms of task assignments required and time invested for computing correctly a set of tasks with high probability. Considering the rationality model, we take an evolutionary game theoretic approach and we design mechanisms that eventually achieve a reliable computational platform where the master receives the correct task result with probability one and with minimal auditing cost. The designed mechanisms provide incentives to the rational workers, reinforcing their strategy to a correct behavior, while they are complemented by four reputation schemes that cope with malice. Finally, we also design a mechanism that deals with unresponsive workers by keeping a reputation related to the workers’ response rate. The designed mechanism selects the most reliable and active workers in each computational round. Simulations, among other, de- pict the trade-off between the master’s cost and the time the system needs to reach a state where the master always receives the correct task result. The second research question we answer in this work concerns the fair and efficient distribution of workers among the masters over multiple XIII XIV computational rounds. Masters with similar tasks are competing for the same set of workers at each computational round. Workers must be assigned to the masters in a fair manner; when the master values a worker’s contribution the most. We consider that a master might have a strategic behavior, declaring a dishonest valuation on a worker in each round, in an attempt to increase its benefit. This strategic behavior from the side of the masters might lead to unfair and inefficient assignments of workers. Applying renown auction mechanisms to solve the problem at hand can be infeasible since monetary payments are required on the side of the masters. Hence, we present an alternative mechanism for fair and efficient distribution of the workers in the presence of strategic masters, without the use of monetary incentives. We show analytically that our designed mechanism guarantees fairness, is socially efficient, and is truthful. Simulations favourably compare our designed mechanism with two benchmark auction mechanisms. Table of Contents Acknowledgements XI Abstract XIII List of Tables XVII List of Figures XXI List of Acronyms 1 1. Introduction 3 2. Background and Related Work 11 2.1. Background . 11 2.2. Related Work . 16 3. Evaluation of Reliability Techniques 25 3.1. Introduction . 25 3.2. Model . 27 3.3. Exact Worker Behavior ( = 0)........................... 30 3.4. Probabilistic Worker Behavior ( > 0)....................... 32 3.4.1. Algorithm MWMIX: Auditing and Voting . 33 3.4.2. Algorithm MWVOTE: Use Voting Alone . 36 3.5. Algorithm E1: Tightly Estimating fa and .................... 38 4. A Reinforcement Learning Approach 43 4.1. Introduction . 43 4.2. General Model . 45 4.3. Presence of Rational Workers . 48 4.3.1. Introduction . 48 4.3.2.

Load more