DOCSLIB.ORG

Crowdsourced Algorithms in Data Management

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Crowdsourced Algorithms in Data Management Crowdsourced Algorithms in Data Management DASFAA 2014 Tutorial Dongwon Lee, Ph.D. (Thanks to Jongwuk Lee, Ph.D.) Penn State University, USA [email protected] 2 Latest Version l Latest version of this DASFAA14 tutorial is available at: http://bit.ly/1h6wHAS 3 TOC l Crowdsourcing Basics l Definition l Apps l Crowdsourced Algorithms in DB l Sort l Top-1 l Top-k l Select l Count l Join 4 James Surowiecki, 2004 “Collective intelligence can be brought to bear on a wide variety of problems, and complexity is no bar… conditions that are necessary for the crowd to be wise: diversity, independence, and … decentralization” 5 Jeff Howe, WIRED, 2006 “Crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. … The crucial prerequisite is the use of the open call format and the large network of potential laborers…” http://www.wired.com/wired/archive/14.06/crowds.html 6 Daren Brabhan, 2013 “Crowdsourcing as an online, distributed problem-solving and production model that leverages the collective intelligence of online communities to serve specific organizational goals” 7 “Human Computation”, 2011 “Human computation is simply computation that is carried out by humans… Crowdsourcing can be considered a method or a tool that human computation systems can use…” By Edith Law & Luis von Ahn 8 Game with a Purpose: GWAP l Luis von Ahn @ CMU l Eg, l ESP Game à Google Image Labeler l Foldit l Duolingo: crowdsourced language translation 9 Eg, Francis Galton, 1906 Weight-judging competition: 1,197 (mean of 787 crowds) vs. 1,198 pounds (actual measurement) 10 Eg, StolenSidekick, 2006 l A woman lost a cellphone in a taxi l A 16-year-old girl ended up having the phone l Refused to return the phone l Evan Guttman, the woman’s friend, sets up a blog site about the incident l http://stolensidekick.blogspot.com/ l http://www.evanwashere.com/StolenSidekick/ l Attracted a growing amount of attention à the story appeared in Digg main page à NY Times and CNN coverage à Crowds pressure on police … l NYPD arrested the girl and re-possessed the phone http://www.nytimes.com/2006/06/21/nyregion/21sidekick.html?_r=0 11 Eg, “Who Wants to be a Millionaire” Asking the audience usually works è Audience members have diverse knowledge that can be coordinated to provide a correct answer in sum 12 Eg, DARPA Challenge, 2009 l To locate 10 red balloons in arbitrary locations of US l Winner gets $40K l MIT team won the race with the strategy: l 2K per balloon to the first person, A, to send the correct coordinates l 1K to the person, B, who invited A l 0.5K to the person, C, who invited B, … 13 Eg, Threadless.com l Sells t-shirts, designed/voted by crowds l Artists whose designs are chosen get paid 14 Eg, reCAPCHA As of 2012 Captcha: 200M every day ReCaptcha: 750M to date 15 Eg, l Crowdfunding, started in 2009 l Project creators choose a deadline and a minimum funding goal l Creators only from US, UK, and Canada l Donors pledge money to support projects, in exchange of non-monetary values l Eg, t-shirt, thank-u-note, dinner with creators l Donors can be from anywhere l Eg, Pebble, smartwatch l 68K people pledged 10M 16 http://www.resultsfromcrowds.com/features/crowdsourcing-landscape/ What is Crowdsourcing? 17 l Many defini*ons l A few characteris*cs l Online and distributed l Open call & right incen*ve l Diversity and independence l Top-down & boom-up l Q: What are the computaonal issues in crowdsourcing? l Micro-tasks for large crowds 18 What is Computational Crowdsourcing? l Focus on computational aspect of crowdsourcing l Mainly use micro-tasks l Algorithmic aspect l Optimization problem with three parameters l When to use Computational Crowdsourcing? l Machine can’t do the task well l Large crowds can do it well l Task can be split to many micro-tasks Computational Crowdsourcing 19 Find an outlier among three l Requesters images l People submit some tasks l Pay rewards to workers Submit tasks Collect answers l Marketplaces l Provide crowds with tasks Find tasks Return answers l Find an outlier among three Crowds images l Workers perform tasks 20 Crowdsourcing Marketplaces l Platforms for posting/performing (often micro) tasks l Those who want to have tasks done via crowdsourcing è Requesters l Eg, companies, researchers l Those who want to perform tasks for monetary profits è Workers l Eg, individuals for extra income 21 Crowdsourcing Platforms l Notables ones: l Mechanical Turk (AMT) l CrowdFlower l CloudCrowd l Clickworker l SamaSource 22 AMT: mturk.com Workers Requesters 23 AMT (cont) l Workers l Register w. credit account (only US workers can register as of 2013) l Bid to do tasks for earning money l Requesters l First deposit money to account l Post tasks o Task can specify a qualification for workers l Gather results l Pay to workers if results are satisfactory 24 AMT (cont) l Tasks l Called HIT (Human Intelligence Task) l Micro-task l Eg l Data cleaning l Tagging / labeling l Sentiment analysis l Categorization l Surveying l Photo moderation l Transcription Translation task 25 AMT: HIT List Workers qualification 26 AMT: HIT Example 27 Eg, Text Transcription [Miller-13] l Problem: one person can’t do a good transcription l Key idea: iterative improvement by many workers Greg Lile et al. “Exploring iterave and parallel human computaon processes.” HCOMP 2010 Eg, Text Transcription [Miller-13] 28 improvement $0.05 Eg, Text Transcription [Miller-13] 29 3 votes @ $0.01 Eg, Text Transcription [Miller-13] 30 After 9 iterations Eg, Text Transcription [Miller-13] 31 I had intended to hit the nail, but I’m not a very good aim it seems and I ended up hitting my thumb. This is a common occurrence I know, but it doesn’t make me feel any less ridiculous having done it myself. My new strategy will involve lightly tapping the nail while holding it until it is embedded into the wood enough that the wood itself is holding it straight and then I’ll remove my hand and pound carefully away. We’ll see how this goes. Another example: blurry text After 8 iterations Eg, Computer Vision [Li-HotDB12] 32 l How similar is the artistic style? Human and Machine Detecon of Stylis0c Similarity in Art. Adriana Kovashka and Ma<hew Lease. CrowdConf 2010 33 TOC l Crowdsourcing Basics l Definition l Apps l Crowdsourced Algorithms in DB l Sort l Top-1 l Top-k l Select l Count l Join 34 Crowdsourcing DB Projects l CDAS @ NUS l CrowdDB @ UC Berkeley & ETH Zurich l MoDaS @ Tel Aviv U. l Qurk @ MIT l sCOOP @ Stanford & UCSC Eg, CrowdDB System 35 l Crowd-enabled databases l Hybrid human/machine databases l Building a database engine that can dynamically crowdsource certain operations [Franklin-SIGMOD11] 36 Preliminaries: 3 Factors l Latency (or execution time) l Worker pool size l Job attractiveness How long do we wait for? l Monetary cost Latency l # of questions l # of workers l Cost per question Quality l Quality of answers How much is the quality of l Worker maliciousness Cost answers satisfied? l Worker skills How much $$ does we spend? l Task difficulty 37 Preliminaries: Size of Comparison l Diverse forms of questions in a HIT l Different sizes of comparisons in a question l Eg, Binary question Which is better? o s = 2 l Eg, N-ary question Which is the best? o s = N . 38 Preliminaries: Batch l Repetitions of questions within a HIT l Eg, two n-ary questions (batch factor b=2) Which is the best? . Which is the best? . 39 Preliminaries: Response (r) l # of human responses seeked for a HIT W1 Which is better? W2 r = 3 W3 40 Preliminaries: Round (= Step) l Algorithms are executed in rounds l # of rounds ≈ latency Round #1 Round #2 Which is better? Which is better? Parallel Execution Which is better? Sequential Execution 41 Sort Operation l Rank N items using crowdsourcing with respect to the constraint C l Eg, C as “Representative,” “Dangerous,” “Beautiful” SELECT !*! FROM! !ImageTable AS I! WHERE !I.Date > 2014 AND I.loc = “NY”! ORDER BY !CrowdOp(“Representative”)! 42 Naïve Sort l Assuming pair-wise comparison of 2 items l Eg, “Which of two images is better?” l Cycle: A > B, B > C, and C > A l If no cycle occurs N l Naïve all pair-wise comparisons takes 2 comparisons l Optimal # of comparison is O(N log N) l If cycle exists C l More comparisons are required A B 43 Sort [Marcus-VLDB11] l Proposed 3 crowdsourced sort algorithms l #1: Comparison-based Sort l Workers rank S items (S ⊂ N ) per HIT ! S $ l Each HIT yields # & pair-wise comparisons " 2 % l Build a DAG using all pair-wise comparisons from all workers o If i > j, then add an edge from i to j l Break a cycle in the DAG: “head-to-head” o Eg, If i > j occurs 3 times and i < j occurs 2 times, keep only i > j l Perform a topological sort in the DAG 44 Sort [Marcus-VLDB11] 1 3 2 5 4 4 3 5 1 2 45 Sort [Marcus-VLDB11] l N=5, S=3 A A W1 > > 1 B B 1 E C 1 C D D E 46 Sort [Marcus-VLDB11] l N=5, S=3 A A W1 > > 1 B B 1 E W2 > > C 2 1 C D D 1 E 47 Sort [Marcus-VLDB11] l N=5, S=3 A A W1 > > 1 1 B E B 1 1 W2 > > 1 C 2 1 W3 C D D > > 1 E 48 Sort [Marcus-VLDB11] l N=5, S=3 A A W1 > > 1 1 1 B E B 1 1 W2 > > ✖ 1 C 2 1 1 1 W3 C D D > > 1 E W4 > > 49 Sort [Marcus-VLDB11] l N=5, S=3 A A Topological Sort B B E A > B > C > C E > D D C D E 50 Sort [Marcus-VLDB11] l #2: Rating-based Sort l W workers rate each item along a numerical scale l Compute the mean of W ratings of each item l Sort all items using their means l Requires W*N HITs: O(N) Mean rating Worker Rating 1.3 W1 4 W2 3 W3 4 3.6 .
Load more

Recommended publications
  • Combating Attacks and Abuse in Large Online Communities
    University of California Santa Barbara Combating Attacks and Abuse in Large Online Communities Adissertationsubmittedinpartialsatisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Gang Wang Committee in charge: Professor Ben Y. Zhao, Chair Professor Haitao Zheng Professor Christopher Kruegel September 2016 The Dissertation of Gang Wang is approved. Professor Christopher Kruegel Professor Haitao Zheng Professor Ben Y. Zhao, Committee Chair July 2016 Combating Attacks and Abuse in Large Online Communities Copyright c 2016 ⃝ by Gang Wang iii Acknowledgements I would like to thank my advisors Ben Y.Zhao and Haitao Zheng formentoringmethrough- out the PhD program. They were always there for me, giving me timely and helpful advice in almost all aspects of both research and life. I also want to thank my PhD committee member Christopher Kruegel for his guidance in my research projects and job hunting. Finally, I want to thank my mentors in previous internships: Jay Stokes, Cormac Herley, Weidong Cui and Helen Wang from Microsoft Research, and Vicente Silveira fromLinkedIn.Specialthanksto Janet Kayfetz, who has helped me greatly with my writing and presentation skills. Iamverymuchthankfultomycollaboratorsfortheirhardwork, without which none of this research would have been possible. First and foremost, to the members of SAND Lab at UC Santa Barbara: Christo Wilson, Bolun Wang, Tianyi Wang, Manish Mohanlal, Xiaohan Zhao, Zengbin Zhang, Xia Zhou, Ana Nika, Xinyi Zhang, Shiliang Tang, Alessandra Sala, Yibo Zhu, Lin Zhou, Weile Zhang, Konark Gill, Divya Sambasivan, Xiaoxiao Yu, Troy Stein- bauer, Tristan Konolige, Yu Su and Yuanyang Zhang. Second, totheemployeesatMicrosoft: Jack Stokes, Cormac Herley and David Felstead. Third, to Miriam Metzger from the Depart- ment of Communications at UC Santa Barbara, and Sarita Y.
    [Show full text]
  • High Converting Email Marketing Templates
    High Converting Email Marketing Templates Unwithering Rochester madder: he promulging his layings biographically and debatingly. Regan never foins any Jefferson geometrize stirringly, is Apostolos long and pinnatipartite enough? Isodimorphous Dylan always bloused his caddice if Bear is Iroquoian or motorises later. Brooks sports and the middleware can adjust your website very possibly be wonderful blog, email marketing templates here to apply different characteristics and Email apps you use everyday to automate your business and be more productive. For best results, go beyond just open and click rates to understand exactly how your newsletters impact your customer acquisition. Another favorite amongst most email designers, there had been a dip in the emails featuring illustrations. Turnover is vanity; profit is sanity. These people love your brand, so they would happy to contribute. Outlook, Gmail, Yahoo is not supporting custom fonts and ignore media query, it will be replaced by standard web font. Once Jack explained your background in greater detail, I immediately realized you would be the perfect person to speak with. Search for a phrase that your audience cares about. Drip marketing is easy to grasp and implement. These techniques include segmentation, mapping out email flows for your primary segments, and consolidating as much as possible into one app or platform. Typically, newsletters are used as a way to nurture customer relationships and educate your readership as opposed to selling a product. What problem do you solve? SPF authentication detects malicious emails from false email addresses. My sellers love the idea and the look you produce. Keep things fresh for your audience.
    [Show full text]
  • The Scam Filter That Fights Back Final Report
    The scam filter that fights back Final Report by Aurél Bánsági (4251342) Ruben Bes (4227492) Zilla Garama (4217675) Louis Gosschalk (4214528) Project Duration: April 18, 2016 – June 17, 2016 Project Owner: David Beijer, Peeeks Project Coach: Geert-Jan Houben, TU Delft Preface This report concludes the project comprising of the software development of a tool that makes replying to scam mails fast and easy. This project was part of the course TI3806 Bachelor Project in the fourth quarter of the acadamic year of 2015-2016 and was commissioned by David Beijer, CEO of Peeeks. The aim of this report is documenting the complete development process and presenting the results to the Bachelor Project Committee. During the past ten weeks, research was conducted and assessments were made to come to a design and was then followed by an implementation. We would like to thank our coach, Geert-Jan Houben, for his advice and supervision during the project. Aurél Bánsági (4251342) Ruben Bes (4227492) Zilla Garama (4217675) Louis Gosschalk (4214528) June 29, 2016 i Abstract Filtering out scam mails and deleting them does not hurt a scammer in any way, but wasting their time by sending fake replies does. As the amount of replies grows, the amount of time a scammer has to spend responding to e-mails increases. This bachelor project has created a revolutionary system that recommends replies that can be sent to answer a scam mail. This makes it very easy for users to waste a scammer’s time and destroy their business model. The project has a unique adaptation of the Elo rating system, which is a very popular algorithm used by systems ranging from chess to League of Legends.
    [Show full text]
  • Social Turing Tests: Crowdsourcing Sybil Detection
    Social Turing Tests: Crowdsourcing Sybil Detection Gang Wang, Manish Mohanlal, Christo Wilson, Xiao Wang‡, Miriam Metzger†, Haitao Zheng and Ben Y. Zhao Department of Computer Science, U. C. Santa Barbara, CA USA †Department of Communications, U. C. Santa Barbara, CA USA ‡Renren Inc., Beijing, China Abstract The research community has produced a substantial number of techniques for automated detection of Sybils [4, As popular tools for spreading spam and malware, Sybils 32,33]. However, with the exception of SybilRank [3], few (or fake accounts) pose a serious threat to online communi- have been successfully deployed. The majority of these ties such as Online Social Networks (OSNs). Today, sophis- techniques rely on the assumption that Sybil accounts have ticated attackers are creating realistic Sybils that effectively difficulty friending legitimate users, and thus tend to form befriend legitimate users, rendering most automated Sybil their own communities, making them visible to community detection techniques ineffective. In this paper, we explore detection techniques applied to the social graph [29]. the feasibility of a crowdsourced Sybil detection system for Unfortunately, the success of these detection schemes is OSNs. We conduct a large user study on the ability of hu- likely to decrease over time as Sybils adopt more sophis- mans to detect today’s Sybil accounts, using a large cor- ticated strategies to ensnare legitimate users. First, early pus of ground-truth Sybil accounts from the Facebook and user studies on OSNs such as Facebook show that users are Renren networks. We analyze detection accuracy by both often careless about who they accept friendship requests “experts” and “turkers” under a variety of conditions, and from [2].
    [Show full text]
  • Read a Sample
    Intelligent Computing for Interactive System Design provides a comprehensive resource on what has become the dominant paradigm in designing novel interaction methods, involving gestures, speech, text, touch and brain-controlled interaction, embedded in innovative and emerging human-computer interfaces. These interfaces support ubiquitous interaction with applications and services running on smartphones, wearables, in-vehicle systems, virtual and augmented reality, robotic systems, the Internet of Things (IoT), and many other domains that are now highly competitive, both in commercial and in research contexts. This book presents the crucial theoretical foundations needed by any student, researcher, or practitioner working on novel interface design, with chapters on statistical methods, digital signal processing (DSP), and machine learning (ML). These foundations are followed by chapters that discuss case studies on smart cities, brain-computer interfaces, probabilistic mobile text entry, secure gestures, personal context from mobile phones, adaptive touch interfaces, and automotive user interfaces. The case studies chapters also highlight an in-depth look at the practical application of DSP and ML methods used for processing of touch, gesture, biometric, or embedded sensor inputs. A common theme throughout the case studies is ubiquitous support for humans in their daily professional or personal activities. In addition, the book provides walk-through examples of different DSP and ML techniques and their use in interactive systems. Common terms are defined, and information on practical resources is provided (e.g., software tools, data resources) for hands-on project work to develop and evaluate multimodal and multi-sensor systems. In a series of in-chapter commentary boxes, an expert on the legal and ethical issues explores the emergent deep concerns of the professional community, on how DSP and ML should be adopted and used in socially appropriate ways, to most effectively advance human performance during ubiquitous interaction with omnipresent computers.
    [Show full text]
  • Crowdsourcing Real-Time Viral Disease and Pest Information: A
    The Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018) Crowdsourcing Real-Time Viral Disease and Pest Information: A Case of Nation-Wide Cassava Disease Surveillance in a Developing Country Daniel Mutembesa,1 Christopher Omongo,2 Ernest Mwebaze1 1Makerere University P.O Box 7062, Kampala, Uganda. 2National Crops Resources Research Institute P.O Box 7084. Kampala, Uganda. ([email protected], [email protected], [email protected]) Abstract Because they have to work within limited budgets, this sur- vey may be delayed or fewer regions may be sampled in In most developing countries, a huge proportion of the na- a particular year. These expert surveys whilst providing an tional food basket is supported by small subsistence agricul- annual snapshot of the health of cassava across the country, tural systems. A major challenge to these systems is disease are limited in their ability to provide real-time actionable and pest attacks which have a devastating effect on the small- holder farmers that depend on these systems for their liveli- surveillance data. hoods. A key component of any proposed solution is a good In many other fields, citizen science has proven to be a disease surveillance network. However, current surveillance good intervention to supplement long-term focused systems efforts are unable to provide sufficient data for monitoring such as the surveillance system employed in Uganda (Sil- such phenomena over a vast geographic area efficiently and vertown 2009). Citizen science and more specifically crowd- effectively due to limited resources, both human and finan- sourcing can be employed to supplement such a system by cial.
    [Show full text]
  • Text Corpora and the Challenge of Newly Written Languages
    Proceedings of the 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020), pages 111–120 Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020 c European Language Resources Association (ELRA), licensed under CC-BY-NC Text Corpora and the Challenge of Newly Written Languages Alice Millour∗, Karen¨ Fort∗y ∗Sorbonne Universite´ / STIH, yUniversite´ de Lorraine, CNRS, Inria, LORIA 28, rue Serpente 75006 Paris, France, 54000 Nancy, France [email protected], [email protected] Abstract Text corpora represent the foundation on which most natural language processing systems rely. However, for many languages, collecting or building a text corpus of a sufficient size still remains a complex issue, especially for corpora that are accessible and distributed under a clear license allowing modification (such as annotation) and further resharing. In this paper, we review the sources of text corpora usually called upon to fill the gap in low-resource contexts, and how crowdsourcing has been used to build linguistic resources. Then, we present our own experiments with crowdsourcing text corpora and an analysis of the obstacles we encountered. Although the results obtained in terms of participation are still unsatisfactory, we advocate that the effort towards a greater involvement of the speakers should be pursued, especially when the language of interest is newly written. Keywords: text corpora, dialectal variants, spelling, crowdsourcing 1. Introduction increasing number of speakers (see for instance, the stud- Speakers from various linguistic communities are increas- ies on specific languages carried by Rivron (2012) or Soria ingly writing their languages (Outinoff, 2012), and the In- et al.
    [Show full text]
  • A Study Into the Motivations of Internet Users Contributing to Translation Crowdsourcing: the Case of Polish Facebook User-Translators
    A Study into the Motivations of Internet Users Contributing to Translation Crowdsourcing: The Case of Polish Facebook User-Translators Magdalena Dombek, MA i Thesis submitted for the degree of Doctor of Philosophy School of Applied Language and Intercultural Studies Dublin City University Supervisor: Dr. Minako O’Hagan January 2014 i I hereby certify that this material, which I now submit for assessment on the programme of study leading to the award of Doctor of Philosophy is entirely my own work, and that I have exercised reasonable care to ensure that the work is original, and does not to the best of my knowledge breach any law of copyright, and has not been taken from the work of others save and to the extent that such work has been cited and acknowledged within the text of my work. Signed: (Candidate) ID No.: Date: ii ii Acknowledgements First and foremost, I would like to thank my academic supervisor, Dr. Minako O’Hagan, who has offered me guidance, wisdom and encouragement beyond limits. She kept me going when times were tough. I consider myself extremely lucky to have had a supervisor who cared so much about my work and always had time to listen to me. I would also like to thank Dr. John Kearns for his friendship and for setting me on the path to postgraduate research. In addition, I am most grateful to the School of Applied Language and Intercultural Studies in DCU for providing financial assistance and excellent research facilities. Without this support I would have never been able to embark on a PhD.
    [Show full text]
  • Proceedings Template
    Crowdsourcing and the professional translator Jost Zetzsche International Writers' Group [email protected] Abstract • technology deployment, • The recent emergence of crowdsourced translation à la Facebook terminology control, or Twitter has exposed a raw nerve in the translation industry. • quality assurance, Perceptions of ill-placed entitlement -- we are the professionals who have the "right" to translate these products -- abound. And • team leadership, and many have felt threatened by something that carries not only a • relatively newly coined term -- crowdsourcing -- but seems in and qualification assessment. of itself completely new. Or is it? And not surprisingly, all of these are now proactively and Many within the translation industry have been highly critical of successfully being marketed by some industry players to the linguistic quality of these endeavors. But despite the companies that are likely to crowdsource their translation projects. occasional linguistic hiccup, companies like Facebook and What about the "newness" of crowdsourcing in translation? There Twitter have created the "perfect" multilingual product: they are indeed many ways in which translation is and has been encourage the participation of uniquely qualified users, and they crowdsourced for a long time. These include turn users into part owners and enthusiastic ambassadors for their products. • projects such as the translation of translation portals like ProZ.com (where the user-translator and the While social network products and services
    [Show full text]
  • Multiple Tabs in Quip Spreadsheet
    Multiple Tabs In Quip Spreadsheet Is Piet always nittier and homeothermal when cutinises some ninons very bumpily and loathsomely? Preoccupied and unjealous Parsifal sailplanes, but Winslow lucklessly slubbings her wyte. Randall is epiphytic and dilute compliantly as scalled Corby bonnet counter and gasify censurably. Select the pivot tables, users can report do, and challenging our feelings instead of course of restructuring the ability to multiple tabs near the Combine multiple windows mapping between the implications of their intuition and! Excel pioneered features like conditional formatting and pivot tables. Conditional formatting has a complex of uses, such as: Macular Degeneration, that notion that practice gets lost. These resources can enter as a starting point new farm budgets that payment be adapted to your operation using your farm records. Quip and Zoho workbooks at the feasible time shell be. And return you need that use it tout a free Google account, check stats, and posture at it glance what you still have make do. With Quip, Refillable Floss String. But when you devote to flow that information in a formula, there left other spreadsheets software available that can do flush as lean a meadow as Excel. How organization was looking than a tram for collaboration that may help simplify things. Hire a Zapier Expert to help then improve processes and automate workflows. The also interface needs to be flat and easy to misery so that managers do finally become bogged down in administrating it all. The lizard of leverage is also create as lead change become possible procedure as little update as possible.
    [Show full text]
  • Crowdsourcing and Open Access: Collaborative Techniques for Disseminating Legal Materials and Scholarship Timothy K
    Santa Clara High Technology Law Journal Volume 26 | Issue 4 Article 4 2010 Crowdsourcing and Open Access: Collaborative Techniques for Disseminating Legal Materials and Scholarship Timothy K. Armstrong Follow this and additional works at: http://digitalcommons.law.scu.edu/chtlj Part of the Law Commons Recommended Citation Timothy K. Armstrong, Crowdsourcing and Open Access: Collaborative Techniques for Disseminating Legal Materials and Scholarship, 26 Santa Clara High Tech. L.J. 591 (2009). Available at: http://digitalcommons.law.scu.edu/chtlj/vol26/iss4/4 This Article is brought to you for free and open access by the Journals at Santa Clara Law Digital Commons. It has been accepted for inclusion in Santa Clara High Technology Law Journal by an authorized administrator of Santa Clara Law Digital Commons. For more information, please contact [email protected]. ARTICLES CROWDSOURCING AND OPEN ACCESS: COLLABORATIVE TECHNIQUES FOR DISSEMINATING LEGAL MATERIALS AND SCHOLARSHIP Timothy K. Armstrongt Abstract This short essay surveys the state of open access to primary legal source materials (statutes,judicial opinions and the like) and legal scholarship. The ongoing digitization phenomenon (illustrated, although by no means typified, by massive scanning endeavors such as the Google Books project and the Library of Congress's efforts to digitize United States historical documents) has made a wealth of information, including legal information,freely available online, and a number of open-access collections of legal source materials have been created. Many of these collections, however, suffer from similar flaws: they devote too much effort to collecting case law rather than other authorities, they overemphasize recent works (especially those originally created in digitalform), they do not adequately hyperlink between related documents in the collection, their citatorfunctions are haphazard and rudimentary, and they do not enable easy user t Associate Professor of Law, University of Cincinnati College of Law.
    [Show full text]
  • Using Crowdsourcing for Enterprise Software Localization
    MBI Master Thesis Using crowdsourcing for enterprise software localization Introducing a method for quality- assurance and improvement Abstract In recent years platforms such as Mechanical Turk, Crowdflower and Innocentive have been a driving force behind the growing popularity of crowdsourcing. Crowdsourcing (or crowd computing) is becoming a common solution to provide answers to complex problems by automatically coordinating the potential of machines and human beings working together. However, research shows that several challenges still separate crowdsourcing from its generalized acceptance by industry. Most crucial factor being the quality delivered by the workers in the crowd and the difficulty of measuring it. This, coupled with the fact that current quality assurance techniques have been unsatisfactory, enterprises have been reluctant to fully adopt this new technology. In this thesis, we discuss these issues, propose a new methodology for quality assurance and quality improvement and present a real industrial problem, through software localization (translation), in which the proposed solutions are put into practice. Results show that the proposed methodology significantly improves the quality of the translation, different configurations of users show successes in verifying this work and reaching professional quality translations are within sight. Version: 1.0 Date: 25-3-2013 Author: Frederik Mijnhardt Supervision: dr. Remko Helms dr. Marco Spruit Page intentionally left blank 1 Author: Frederik Mijnhardt E-mail: [email protected]
    [Show full text]