Project Sidewalk: a Web-Based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data at Scale Manaswi SAHA
Total Page：16
File Type：pdf, Size：1020Kb
Singapore Management University Institutional Knowledge at Singapore Management University Research Collection School Of Information Systems School of Information Systems 5-2019 Project Sidewalk: A Web-based crowdsourcing tool for collecting sidewalk accessibility data at scale Manaswi SAHA Michael SAUGSTAD Hanuma MADDALI Aileen ZENG Ryan HOLLAND See next page for additional authors DOI: https://doi.org/10.1145/3290605.3300292 Follow this and additional works at: https://ink.library.smu.edu.sg/sis_research Part of the Databases and Information Systems Commons, and the Geographic Information Sciences Commons Citation SAHA, Manaswi; SAUGSTAD, Michael; MADDALI, Hanuma; ZENG, Aileen; HOLLAND, Ryan; BOWER, Steven; DASH, Aditya; CHEN, Sage; Li, Anthony; HARA, Kotaro; and FROEHLICH, Jon. Project Sidewalk: A Web-based crowdsourcing tool for collecting sidewalk accessibility data at scale. (2019). CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, May 4-9. 62:1-14. Research Collection School Of Information Systems. Available at: https://ink.library.smu.edu.sg/sis_research/4399 This Conference Proceeding Article is brought to you for free and open access by the School of Information Systems at Institutional Knowledge at Singapore Management University. It has been accepted for inclusion in Research Collection School Of Information Systems by an authorized administrator of Institutional Knowledge at Singapore Management University. For more information, please email [email protected]. Author Manaswi SAHA, Michael SAUGSTAD, Hanuma MADDALI, Aileen ZENG, Ryan HOLLAND, Steven BOWER, Aditya DASH, Sage CHEN, Anthony Li, Kotaro HARA, and Jon FROEHLICH This conference proceeding article is available at Institutional Knowledge at Singapore Management University: https://ink.library.smu.edu.sg/sis_research/4399 Project Sidewalk: A Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data at Scale 1Manaswi Saha, 1Michael Saugstad, 2Hanuma Teja Maddali, 1Aileen Zeng, 3Ryan Holland, 2Steven Bower, 2Aditya Dash, 4Sage Chen, 2Anthony Li, 5Kotaro Hara, 1Jon Froehlich 1University of Washington; 2University of Maryland, College Park; 3Montgomery Blair High School; 4University of Michigan; 5Singapore Management University {manaswi,saugstad,aileenz,jonf}@cs.washington.edu Figure 1: In an 18-month deployment study of Project Sidewalk, we collected 205,385 sidewalk accessibility labels, including curb ramps, missing curb ramps, sidewalk obstacles, and surface problems. Each dot above represents a geo-located label rendered at 50% translucency. Try out the tool at http://projectsidewalk.io. ABSTRACT demonstrate the potential of virtually auditing urban accessi- We introduce Project Sidewalk, a new web-based tool that bility and highlight tradeoffs between scalability and quality enables online crowdworkers to remotely label pedestrian- compared to traditional approaches. related accessibility problems by virtually walking through CCS CONCEPTS city streets in Google Street View. To train, engage, and sus- tain users, we apply basic game design principles such as • Human-centered computing → Accessibility systems interactive onboarding, mission-based tasks, and progress and tools; Interactive systems and tools; • Information dashboards. In an 18-month deployment study, 797 online systems → Crowdsourcing. users contributed 205,385 labels and audited 2,941 miles of KEYWORDS Washington DC streets. We compare behavioral and label- ing quality differences between paid crowdworkers and vol- Crowdsourcing, accessibility, mobility impairments, GIS unteers, investigate the effects of label type, label severity, ACM Reference Format: and majority vote on accuracy, and analyze common label- Manaswi Saha, Michael Saugstad, Hanuma Teja Maddali, Aileen ing errors. To complement these findings, we report on an Zeng, Ryan Holland, Steven Bower, Aditya Dash, Sage Chen, An- interview study with three key stakeholder groups (N=14) thony Li, Kotaro Hara, Jon Froehlich. 2019. Project Sidewalk: A soliciting reactions to our tool and methods. Our findings Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibil- ity Data at Scale. In CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/ 3290605.3300292 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies 1 INTRODUCTION are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights Geographic Information Systems (GIS) such as Google Maps, for components of this work owned by others than the author(s) must Waze, and Yelp have transformed the way people travel be honored. Abstracting with credit is permitted. To copy otherwise, or and access information about the physical world. While republish, to post on servers or to redistribute to lists, requires prior specific these systems contain terabytes of data about road networks permission and/or a fee. Request permissions from [email protected]. and points of interest (POIs), their information about physi- CHI 2019, May 4–9, 2019, Glasgow, Scotland UK cal accessibility is commensurately poor. GIS websites like © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. Axsmap.com, Wheelmap.org, and AccessTogether.org aim to ACM ISBN 978-1-4503-5970-2/19/05...$15.00 address this problem by collecting location-based accessibil- https://doi.org/10.1145/3290605.3300292 ity information provided by volunteers (i.e., crowdsourcing). CHI 2019, May 4–9, 2019, Glasgow, Scotland UK M. Saha et al. While these efforts are important and commendable, their study, paid workers also significantly outperformed regis- value propositions are intrinsically tied to the amount and tered and anonymous volunteers in finding accessibility prob- quality of data they collect. In a recent review of accessibility- lems (recall=68% vs. 61% and 49%, respectively) but precision oriented GIS sites, Ding et al. [15] found that most suffered was roughly equivalent for all groups (~70%). Our findings from serious data sparseness issues. For example, only 1.6% also show that the number of found issues significantly in- of the Wheelmap POIs had data entered on accessibility. One creases with the number of labelers per street—with five key limiting factor is the reliance on local populations with labelers, recall rose from 68% to 92%. physical experience of a place for data collection. While local To complement these findings, our interview study asked users who report data are likely to be reliable, the dependence about perceptions of and experiences with urban accessibil- on in situ reporting dramatically limits scalability—both who ity and solicited reactions to Project Sidewalk and the idea of can supply data and how much data they can easily supply. crowdsourcing accessibility in general. All three stakeholder In contrast, we are exploring a different approach embod- groups were positive: while government officials emphasized ied in a new interactive tool called Project Sidewalk (Figure 2), cost-savings and civic engagement, the MI and caregiver which enables online crowdworkers to contribute physical- groups focused more on personal utility and enhanced inde- world accessibility information by virtually walking through pendence. Key concerns also arose, including data reliability, city streets in Google Street View (GSV)—similar to a first- maintenance, and, for the MI participants, whether labels person video game. Rather than pulling solely from local properly reflected their accessibility challenges (the latter populations, our potential pool of users scales to anyone echoes findings from [24]). with an Internet connection and a web browser. Project Side- In summary, the contributions of this paper include: (i) walk extends previous work in streetscape imagery auditing Project Sidewalk, a novel web-based virtual auditing tool tools like Canvas [4], Spotlight [7], BusStop CSI [23], and for collecting urban accessibility data at scale; (ii) results Tohme [26], all which demonstrate the feasibility of virtual from an 18-month deployment and complementary data val- auditing and, crucially, that virtual audit data has high con- idation study exploring key behavioral and labeling quality cordance with traditional physical audits. However, this past differences between volunteer and paid crowdworkers; (iii) work has focused on small spatial regions, relied on special- findings from semi-structured interviews with three stake- ized user populations such as public health researchers [4, 7] holder groups soliciting reactions to Project Sidewalk and and paid crowdworkers [23, 26], and has not been publicly identifying key concerns and design suggestions; (iv) and our deployed. large, open-source sidewalk accessibility dataset1, which we In this paper, we present an 18-month deployment study believe is the largest of its kind. By scaling up data collection of Project Sidewalk in Washington DC. In total, 797 users methods for sidewalk accessibility, our overarching aim is contributed 205,385 geo-located accessibility labels and vir- to enable the development of new accessibility-aware map- tually audited the entirety of Washington DC (1,075 miles of ping tools (e.g., [24, 32]), provide increased transparency and city streets; see Figure 2).