Pando: Personal Volunteer Computing in Browsers Erick Lavoie Laurie Hendren School of Computer Science School of Computer Science McGill University McGill University Montreal, Quebec, Canada Montreal, Quebec, Canada [email protected] [email protected] Frederic Desprez Miguel Pupo Correia Antenne Inria Giant INESC-ID INRIA Grenoble Rhône-Alpes Lisboa, Portugal Grenoble, France [email protected] [email protected] Abstract to own many unused laptops, tablets, and cell phones, many The large penetration and continued growth in ownership of of which could still make useful computing contributions. personal electronic devices represents a freely available and Technical, financial, and administrative barriers in exist- largely untapped source of computing power. Moreover, the ing systems prevent a large part of the general programmer large environmental and social impact of producing these population, especially in humanities and developing coun- devices suggests we should better use those that already tries, from using these resources for parallelizing the execu- exist. We aim to make these devices available for parallel tion of many possible personal projects. This suggests a vast computations to both scientists and other programmers of under-explored design space for various personal tools that the general public, for their personal projects, and in the use alternative computing resources. In parallel, the wide simplest way possible to program and to deploy. We named popularity of social applications makes it easier than ever for our approach to distributed computing personal volunteer individual users to leverage their personal social network for computing. help, but this possibility has little been used so far to meet We designed, implemented, and tested Pando, a new dis- their computing needs. Personal devices, personal projects, per- tributed computing tool based on a declarative concurrent sonal tools, and personal social networks form together a new programming model, organized around the pull-stream de- viewpoint from which new tools may be developed, which sign pattern, and implemented using JavaScript, WebRTC, we call personal volunteer computing. and WebSockets. This tool enables a dynamically varying The major challenges in designing personal volunteer com- number of failure-prone personal devices contributed by puting tools for general programmers are analogous to those volunteers to parallelize the application of a function on a that have prompted the articulation of intermediate technolo- stream of values, by using the devices’ browsers. gies [94] in the 1970s for promoting economic development in To illustrate Pando’s capabilities, to show its benefits as developing countries. Then, the limited access to high-speed well as its limitations, we implemented a variety of applica- reliable infrastructure, specialists, capital, and world-wide tions including crypto-currency mining, hyper-parameter resources had promoted a development vision based on local optimization in machine learning, crowd computing, and knowledge, local resources, and simple reliable designs that open data processing and tested it using diverse devices we may be implemented, maintained, and improved by their have accumulated over the years. Pando, both as a tool and a users for their personal needs. Translated to the design of dis- reference design, should therefore be a useful addition to the tributed computing tools for today, the major challenges are parallel toolbox of a multitude of users and a complemen- to find simple designs that are applicable for a wide rangeof tary approach to existing parallel and distributed computing applications with minimum needs for hosted infrastructure alternatives. or dedicated hardware, and explain them in a way to foster local appropriation and replication in many programming 1 Introduction environments. The declarative concurrent programming paradigm [101] More than 1.5 billion cell phones were sold in the world greatly simplifies reasoning about concurrent processes: it in 2017 [21] and the computing power of the highest-end abstracts the non-determinism in the execution by making devices today rivals that of desktops and laptops [60]. They it non-observable. This paradigm has already enjoyed great collectively represent an immense source of largely untapped practical successes with the popular MapReduce [44] and computing power. At the same time, the large environmental Unix pipelining [64] programming models. Could it also be and social impact of producing these devices suggests we useful for building personal volunteer computing tools? This should better use those that already exist. It is now common 1 paper answers positively through the design of Pando, a tool Previous parallelism approaches such as GPU program- that enables a dynamically varying number of failure-prone ming and heterogeneous computing [78] have typically fo- personal devices contributed by volunteers to parallelize the cused on maximizing the performance on newer hardware application of a function on a stream of values by using the architectures. Acquiring the targeted devices may be too devices’ browsers. expensive for many use cases, the newer techniques may This paper makes the following contributions: (1) it in- not work on the devices most people already own, and are troduces and articulates the personal volunteer computing usually sufficiently complicated to implement to require ded- viewpoint within the context of other parallel and distributed icated experts for the task. Most of the parallelism solutions computing approaches; (2) it presents the design of Pando are therefore better suited as reusable libraries and runtime through both high-level design principles and a concrete systems rather than general-purpose coordination tools. working implementation, itself organized around the pull- Cloud computing platforms [36, 40] introduce a financial stream design pattern and based on JavaScript [25], Web- barrier for those that do not have access to financial instru- Sockets [7], and WebRTC [17] to enable its execution inside ments, such as a bank account or a credit card. Existing open browsers; (3) it presents our novel StreamLender abstrac- source cloud solutions such as Open Compute [6], Open tion that encapsulates the key properties of Pando’s pro- Stack [4], and Hadoop [9] are designed for making avail- gramming model necessary to distribute a single stream to able co-located dedicated devices to many users in a shared a dynamically varying number of failure-prone processing platform, with a complexity and resource requirements that devices; (4) it reports on the application of Pando to 7 appli- limits their deployment on an ad hoc set of personal devices. cations, including crypto-currency mining, crowd comput- Edge [95] and gray [85] computing platforms extend ex- ing, machine learning hyper-parameter optimization, and isting clouds to use personal devices for computation and open data processing in combination with other peer-to- therefore inherit their financial and technical barriers for peer data distribution protocols; (5) it presents real-world usage and deployment. Moreover, they require trust in the measurements on the benefits that can be obtained showing operator of the platforms that sensitive personal data from that both Pando and personal volunteer computing are a participating users will be correctly managed. useful and complementary addition to existing parallel and Grid computing platforms [51, 55, 56] are designed to distributed computing approaches. The implementation of make available devices from many collaborating organiza- Pando is open source [32] and its individual components tions available to researchers with a unified interface. Access may be repurposed in many different applications. to the existing platforms requires administrative permissions, The rest of this paper is organized as follows. We introduce which are typically reserved to researchers from public insti- personal volunteer computing by studying the limitations tutions and not available to programmers from the general of other approaches for our intended context in Section 2. public. The tools themselves include facilities to manage We present the overall design of Pando in Section 3. We pro- complex access control policies that are not necessary when vide the key properties and behaviour of the StreamLender using personal devices for computations. abstraction in Section 4. We present the different applica- Previous peer-to-peer computing solutions [27, 46, 59, 66, tions in Section 5 and evaluate the benefits and limitations 67, 82, 88, 99, 107] have typically focused on building global of parallelizing them in real-world deployments in Section 6. platforms, shared by many users, This introduces the need We compare the specificities of our design to related work for load-balancing and multiplexing multiple tasks on the in Section 7. We conclude with a brief recapitulation of the same set of participating devices, which complicates their paper and future work in Section 8. implementation. Moreover, in a volunteer context, maintain- ing the system while no tasks are being executed has a cost 2 Personal Volunteer Computing in time and attention, with no immediate benefit, which disincentivises voluntary contributions. Personal volunteer computing aims to address the needs of Volunteer computing tools [30, 93] have historically fo- scientists and programmers of the general population, for cused