CAPTCHA-based Image Labeling on the Soylent Grid Peter Faymonville1,∗ Kai Wang1, John Miller2, Serge Belongie1 † 1Department of Computer Science and Engineering and 2Calit2 University of California, San Diego [email protected], [email protected], [email protected], [email protected] ABSTRACT sets and/or tasks exist. Therefore, the surrounding infras- We introduce an open labeling platform for Computer Vision tructure around a Captcha doesn’t have to be implemented researchers based on Captchas, creating as a byproduct la- each time a research group decides to try a new Captcha beled image data sets while supporting web security. For the method and long-term relationships to Captcha-dependent two different tasks of annotation and detection, we provide websites can be established, reducing the integration costs a security analysis and explore usability issues. We present for these websites. the interfaces used by researchers, website administrators and users and experimental results obtained using the plat- This research was motivated through the GroZi project, form. Finally, we discuss system sustainability issues in the which aims at creating a technological solution for grocery context of a broader “ecosystem” for the platform. shopping for blind and visually impaired users. Part of this solution will be a mobile device with a camera running a Categories and Subject Descriptors text-recognition algorithm. Providing training data for this algorithm was the initial goal of this research project. K.6.5 [Computing Milieux]: Management of Computing and Information Systems—Security and Protection 2. RELATED WORK This paper builds on [6], which described the general Soy- 1. INTRODUCTION lentGrid infrastructure. We now present an implementation, In today’s web surfing experience, the task of solving a experiments and an ecosystem based on this infrastructure Captcha (Completely Automated Public Turing test to tailored to image labeling. tell Computers and Humans Apart) has become ubiquitous, with application ranging from preventing comment spam to The idea of a Captcha has first been formally introduced stopping automatic account creation. With the advent of in [1], along with the proposal to use hard AI problems ReCaptcha [8], the idea of using these tests to create la- for Captchas. Captchas based on recognizing objects – beled data sets was introduced. Our labeling tool on the specifically animals – in images are proposed in [2]. Initial Soylent Grid platform advances this idea in two ways, (1) observations on the usability of Captchas can be found in by supporting more than one task and (2) by creating an [9]. In [8], the authors present a way to display a Captcha open platform for all researchers to get their data sets la- problem that simultaneously provides security and gathers beled. Furthermore, we address the security and quality- user input for improving OCR systems, which is referred to of-service problems resulting from the openness. Given the as ReCaptcha. unrelenting demand for labeled image data sets by the AI community, we assume for sake of argument the availability The work of [10] also describes a general framework for hu- of a bottomless source of image data sets at our disposal. man computation involving researchers, website administra- tors, and users. In that work, they broadly outline different Due to the variety of tasks and data sets, the system becomes models for how the three entities could potentially interact – harder to attack if the different tasks and data sets are dis- including models where money or services are exchanged be- played at random and security measures for retiring data tween the person needing human computation and internet user. In contrast, our framework observes that each party Visiting Student from University of Potsdam, Germany ∗ provides a different value to the system, and we position †Contact author. them to interact in a cooperative setting. Users provide hu- man computation cycles and desire content from websites. Website administrators want users to access their content but desire security. Researchers provide data to be used for security but desire human computation. This work de- scribes how we channel the goals of each group so that their strengths are leveraged and needs are satisfied. 3. PROPOSED APPROACH The SoylentGrid platform keeps track of data from researchers that require human annotation. These data sets contain el- ements that are hand labeled by the researchers themselves – the control data – and elements without labels that need to be labeled by users – the experiment data. When a data set is initially uploaded to SoylentGrid, we first subject it to a period of evaluation to gauge the difficultly of the labeling task; this period of time is referred to as a data set’s pilot study. Details about how we conduct the pilot study are in Section 6.3. After this evaluation is complete, the data set is added to the SoylentGrid and is assigned a priority for how often the data from a particular researcher’s Figure 1: User Interface ‘Both’-Task. Displayed are set is presented to users. A higher priority for a data set two product images with bounding boxes around the means the data will be labeled sooner. The details of how words to be entered already drawn. The words are we determine priority are discussed in Section 6. also entered already. For the annotation task, the bounding boxes would be supplied by the task, for From a user’s point of view, we present each of the tasks the detection task, the label would be supplied by twice side-by-side. Following the approach of [8], one of the task. them is the experiment task, the other one the control task. For the control task, the task solution is already known, whereas for the experiment task, the solution is the label 3.2 User tasks sought by the researcher. The data for the tasks is drawn at Users want to access a resources on the internet, and an random respectively from the experiment and control data attempt to access one protected by SoylentGrid results in a sets. It is not revealed which task is experiment and which prompt to complete one of a number of tasks that acts as is control. a Captcha. We describe two tasks we’ve implemented for our experiments: image annotation and object detection. If the solution for the control task is correct, the user is granted access to the secured resource and the experiment In the annotation task, an image is displayed together with task data is saved. If the solution is not correct, the user an overlay rectangle highlighting a part of that image. The is presented with a new task. This approach provides a user is supposed to type in the word displayed in the high- Captcha, as long as the given task can only be solved with lighted area. Therefore, we can support labeling multiple low probability by a computer program. areas per image. The user can click on the image to see a larger version of the image as an overlay to the complete Since the user interface of our Captcha is going to be em- website. bedded into another website, the user interface has to be constrained in size, our implementation is 400 pixels wide For the detection task, a user is given a word and is supposed and 200 pixels high. Each of the displayed images is resized to draw a bounding box around the given word by either to 150x150 pixels. clicking on one corner and dragging and releasing the mouse or by clicking on two corners of the box. In each method, a tentative rectangle follows the cursor. 3.1 Definitions Three stakeholders participate in the platform: The ‘both’-task is a combination of the two tasks: The user chooses a word from the presented image, draws a rectangle Researchers: Provide image data sets and need labels around it and types it. This task has the highest requirement for control data items, because all possible words and their Website Administrators: Need portions of their website pro- bounding boxes in the image have to be previously known. tected by Captchas This task can be seen in figure 1. Users: Want access to a Captcha-secured resource 3.3 Researcher tasks Three different tasks are implemented: From an researcher’s point of view, SoylentGrid is used as an image labeling tool. He provides the system with his data Annotation: Type a text label given a bounding box set and configuration parameters. The data set contains labeled control data and and unlabeled experiment data. Detection: Draw a bounding box given a text label The configuration parameters consist of the level of tolerance to accept labels as being correct and the number of times a Both: A combination of both tasks particular unit of data needs to be consistently labeled by users to satisfy the researcher. In the following sections, we define each of the stakeholders and their associated tasks. 3.4 Website Admin tasks For a website administrator, the motivation to use a Captcha comes from the need to protect certain resources of a web- site, i.e. web forms, account creation sites, search boxes, control right is therefore the inverse of the number of words due to concerns for the site performance, spam or other in the dictionary (m). malicious behavior. A website administrator needs to in- 1 tegrate our Captcha solution into his web application to pann = (1) make sure that his resources are effectively protected. Key m to the adoption of Captchas is to provide an easy mecha- nism to use them with popular content management systems such as Drupal and Wordpress. Therefore, we’re planning Detection. The trivial attack on the detection task would to publish these plugins to facilitate widespread adoption of place a random bounding box on the given image surface.