DOCSLIB.ORG

Use of Ai in Online Content Moderation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Use of Ai in Online Content Moderation USE OF AI IN ONLINE CONTENT MODERATION 2019 REPORT PRODUCED ON BEHALF OF Foreword from Ofcom As the UK’s converged communications regulator, Ofcom oversees broadband and mobile telecoms, TV, radio, video-on-demand services, post and the airwaves used by wireless devices. We have a statutory duty to promote media literacy, under which we carry out research into people’s use of – and attitudes towards – various communications services, including online services such as social media and video sharing platforms. This remit reflects the increasing convergence of the companies and markets that we regulate. In recent years a wide-ranging, global debate has emerged around the risks faced by internet users, with a specific focus on protecting users from harmful content. A key element of this debate has centred on the role and capabilities of automated approaches (driven by Artificial Intelligence and Machine Learning techniques) to enhance the effectiveness of online content moderation and offer users greater protection from potentially harmful material. These approaches may have implications for people’s future use of – and attitudes towards – online communications services, and may be applicable more broadly to developing new techniques for moderating and cataloguing content in the broadcast and audiovisual media industries, as well as back-office support functions in the telecoms, media and postal sectors. Ofcom has commissioned Cambridge Consultants to produce this report as a contribution to the evidence base on people’s use of and attitudes towards online services, which helps enable wider debate on the risks faced by internet users. Cambridge Consultants has retained editorial responsibility for this report throughout the process, so the opinions stated and highlighted policy implications should not necessarily be taken to represent the views of Ofcom. REPORT USE OF AI IN ONLINE CONTENT MODERATION The advances of AI in recent years will continue, driven by EXECUTIVE SUMMARY commercial applications and enabled by continued progress in algorithm development, the increasing availability of low- cost computational power and the widespread collection In the last two decades, online platforms that permit users to of data. There are, however, some inhibitors to making the interact and upload content for others to view have become most of the potential of AI, such as the lack of transparency integral to the lives of many people and have provided a benefit of some AI algorithms which are unable to fully explain the to society. However, there is growing awareness amongst the reasoning for their decisions. Society has not yet built up the public, businesses and policy makers of the potential damage same level of trust in AI systems as in humans when making caused by harmful online material. The user generated complex decisions. There are risks that bias is introduced into content (UGC) posted by users contributes to the richness AI systems by incorporating data which is unrepresentative or and variety of content on the internet but is not subject to by programming in the unconscious bias of human developers. the editorial controls associated with traditional media. This In addition, there is a shortage of staff suitably qualified in enables some users to post content which could harm others, developing and implementing AI systems. However, many particularly children or vulnerable people. Examples of this of these inhibitors are being addressed: the supply of AI- include content which is cruel and insensitive to others, which skilled engineers will increase and it is likely that society will promotes terrorism or depicts child abuse. gradually develop greater confidence in AI as it is increasingly seen to perform complex tasks well and it is adopted in more As the amount of UGC that platform users upload continues to aspects of our lives. Other issues will, however, continue to be accelerate,1 it has become impossible to identify and remove a problem for at least the short term and these are discussed harmful content using traditional human-led moderation in more detail in this report. approaches at the speed and scale necessary. Current approaches and challenges to online content moderation This paper examines the capabilities of artificial intelligence (AI) technologies in meeting the challenges of moderating Effective moderation of harmful online content is a challenging online content and how improvements are likely to enhance problem for many reasons. While many of these challenges those capabilities over approximately the next five years. affect both human and automated moderation systems, some are especially challenging for AI-based automation systems to Recent advances in AI and its potential future impact overcome. The term ‘AI’ is used in this report to refer to the capability There is a broad range of content which is potentially harmful, of a machine to exhibit human-like performance at a defined including but not limited to: child abuse material, violent and task, rather than refer to specific technical approaches, such extreme content, hate speech, graphic content, sexual content, as ‘machine learning’. Although AI has been through several cruel and insensitive material and spam content. Some harmful cycles of hype followed by disillusionment since its inception in content can be identified by analysing the content alone, but the 1950s, the current surge in investment and technological other content requires an understanding of the context around progress is likely to be sustained. The recent advances in AI have it to determine whether or not it is harmful. Interpreting been enabled by progress in new algorithms and the availability this context consistently is challenging for both human and of computational power and data, and AI capabilities are now automated systems because it requires a broader understanding delivering real commercial value across a range of sectors. of societal, cultural, historical and political factors. Some of these contextual considerations vary around the world The latest advances in AI have mainly been driven by machine due to differences in national laws and what societies deem learning, which enables a computer system to make decisions acceptable. Content moderation processes must therefore be and predict outcomes without being explicitly programmed to contextually aware and culturally-specific to be effective. perform these tasks. This approach requires a set of data to train the system or a training environment in which the system Online content may appear in numerous different formats can experiment. The most significant breakthrough of machine which are more difficult to analyse and moderate, such as video learning in recent times is the development of ‘deep neural content (which requires image analysis over multiple frames to networks’ which enable ‘deep learning’. These neural networks be combined with audio analysis) and memes (which require enable systems to recognise features in complex data inputs a combination of text and image analysis with contextual and such as human speech, images and text. For many applications, cultural understanding). Deepfakes, which are created using the performance of these systems in delivering the specific task machine learning to generate fake but convincing images, for which they have been trained now compares favourably with video, audio or text, have the potential to be extremely harmful humans, but AI still does make errors. and are difficult to detect by human or AI methods. 04 USE OF AI IN ONLINE CONTENT MODERATION REPORT In addition, content may be posted as a live video stream or live whether content is harmful or not. Clarity in platforms’ text chat which must be analysed and moderated in real time. community standards is therefore essential to enabling the This is more challenging because the level of harmfulness can development and refinement of AI systems to enforce the escalate quickly and only the previous and current elements of standards consistently. the content are available for consideration. Overall, it is not possible to fully automate effective content Over time, the language and format of online content evolve moderation. For the foreseeable future the input of human rapidly and some users will attempt to subvert moderation moderators will continue to be required to review highly systems such as by adjusting the words and phrases they use. contextual, nuanced content. Human input is expensive and Moderation systems must therefore adapt to keep pace with difficult to scale as the volume of content uploaded increases. these changes. It also requires individuals to view harmful content in order to moderate it, which can cause them significant psychological Online platforms may moderate the third-party content they distress. However, AI-based content moderation systems can host to reduce the risk of exposing their users to harmful reduce the need for human moderation and reduce the impact material and to mitigate the reputational risk to the organisation. on them of viewing harmful content. However, removing content which is not universally agreed to be harmful can also result in reputational damage and Many organisations follow an online content moderation undermine users’ freedom of expression. Facebook, for workflow which uses moderation at one or both of two points example, has been criticised for removing an image of a statue in the workflow: of Neptune in Bologna, Italy for being sexually explicit and the iconic photograph
Load more

Recommended publications
  • ``At the End of the Day Facebook Does What It Wants'': How Users
    “At the End of the Day Facebook Does What It Wants”: How Users Experience Contesting Algorithmic Content Moderation KRISTEN VACCARO, University of Illinois Urbana-Champaign CHRISTIAN SANDVIG, University of Michigan KARRIE KARAHALIOS, University of Illinois Urbana-Champaign Interest has grown in designing algorithmic decision making systems for contestability. In this work, we study how users experience contesting unfavorable social media content moderation decisions. A large-scale online experiment tests whether different forms of appeals can improve users’ experiences of automated decision making. We study the impact on users’ perceptions of the Fairness, Accountability, and Trustworthiness of algorithmic decisions, as well as their feelings of Control (FACT). Surprisingly, we find that none of the appeal designs improve FACT perceptions compared to a no appeal baseline. We qualitatively analyze how users write appeals, and find that they contest the decision itself, but also more fundamental issues like thegoalof moderating content, the idea of automation, and the inconsistency of the system as a whole. We conclude with suggestions for – as well as a discussion of the challenges of – designing for contestability. CCS Concepts: • Human-centered computing → Human computer interaction (HCI); Social media. Additional Key Words and Phrases: content moderation; algorithmic experience ACM Reference Format: Kristen Vaccaro, Christian Sandvig, and Karrie Karahalios. 2020. “At the End of the Day Facebook Does What It Wants”: How Users Experience Contesting Algorithmic Content Moderation. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 167 (October 2020), 22 pages. https://doi.org/10.1145/3415238 1 INTRODUCTION As algorithmic decision making systems become both more prevalent and more visible, interest has grown in how to design them to be trustworthy, understandable and fair.
    [Show full text]
  • Download Paper
    Lawless: the secret rules that govern our digital lives (and why we need new digital constitutions that protect our rights) Submitted version. Forthcoming 2019 Cambridge University Press. Nicolas P. Suzor Table of Contents Part I: a lawless internet Chapter 1. The hidden rules of the internet ............................................................................................. 6 Process matters ....................................................................................................................................................... 12 Chapter 2. Who makes the rules?.......................................................................................................... 17 Whose values apply? ............................................................................................................................................... 22 The moderation process .......................................................................................................................................... 25 Bias and accountability ........................................................................................................................................... 28 Chapter 3. The internet’s abuse problem ............................................................................................... 41 Abuse reflects and reinforces systemic inequalities ................................................................................................ 50 Dealing with abuse needs the involvement of platforms .......................................................................................
    [Show full text]
  • Content Moderation
    Robert Lewington (Twitch) & The Fair Play Alliance Executive Steering Committee April 2021 Being ‘Targeted’ about Content Moderation: Strategies for consistent, scalable and effective response to Disruption & Harm 1 Content Moderation: Best Practices for Targeted Reporting & reactive UGC Management At Scale March 2021 Abstract This document provides replicable best practice information on how to moderate User-Generated Content (UGC) in social applications or services (including digital media and video games). Its focus is on reactive moderation, a central component of the growing content moderation toolkit where a service provider responds to reports submitted by users of its service regarding UGC that may violate its Terms of Service. Specifically, the document explores and advocates for a ‘targeted’ approach to the creation of reporting mechanisms. This allows users to closely identify the specific infraction, utilise evidence of the infraction—access to which is facilitated as part of the design of the reporting process—enabling consistent content moderation at scale. Note, however, that we will also make passing-reference to pre, post and proactive (see Appendix A) moderation approaches. Specifics of how best to tailor these best practices to a particular application or service will differ based on various parameters, including: type of service (social media, video game etc.); type of media (text, image, audio, video etc.); sharing mechanism (feed/gallery, avatar, communication etc.); persistence (ephemeral vs. static/umutable) and others, and therefore this document should be considered a set of high-level instructive principles rather than prescriptive guidelines. Contents i. Background ii. The Content Moderation Flywheel ○ Community Guidelines/Code of Conduct ○ Targeted reporting ■ Context ○ Scalable Content Moderation ○ Education ○ Technology iii.
    [Show full text]
  • Getting Started with PX4 for Contributors Mark West PX4 Community Volunteer Format
    Getting Started with PX4 For Contributors Mark West PX4 Community Volunteer Format A word about format Limited Time + Big Subject = Compression Some Subjects Skipped = Look in Appendix Demo Videos Clipped = Look in Appendix Who is this for? Anybody Levels: L1: Operate a PX4 vehicle L2: Build a PX4 vehicle L3: Build the source L4: Modify the source L5: Contribute L1: Operate You want to operate a drivable unit See Appendix : Operate Vehicle L2: Build a Vehicle You want to build a drivable unit See Appendix : Build Vehicle L3: Build the Source You want to build the image from source Big step up from L2 Choose Toolchain Container L3: Build the Source : Toolchain All the tools you need to build an image / exe Compilers / Linkers / Tools / Etc Installation: Manual : If needed Convenience Scripts : Preferred L3: Build the Source : Container Container? Like a better VM, with toolchain installed Easy to install, easy to update, no interaction Image = container snapshot L3: Build the Source : Container Container images built in layers px4io/px4-dev-ros-melodic px4io/px4-dev-simulation-bionic px4io/px4-dev-base-bionic ubuntu L3: Build the Source : Container Containers are isolated, all interaction is explicit File System -v ~/src_d/Firmware:/src/firmware/:rw File System Directory Directory Network Port -p 14556:14556/udp Network Port Host Docker run params Container L3: Build the Source : Toolchain or Container? Container if possible Toolchain if no suitable container * But you can make your own containers L3: Build the Source : Toolchain Demo Build image for SITL Simulation 1) Install Toolchain 2) Git PX4 Source 3) Build PX4 4) Takeoff L3: Build the Source : Container Demo Build image for SITL Simulation 1) Install Docker 2) Git PX4 Source 3) Locate Container 4) Run Container 5) Build PX4 6) Build available on host computer L4: Modify the Source Next step up in complexity Why? Fix bugs, add features Loop: Edit, Build, Debug IDE: Visual Studio, Eclipse, QT Creator, etc.
    [Show full text]
  • Deeper Attention to Abusive User Content Moderation
    Deeper Attention to Abusive User Content Moderation John Pavlopoulos Prodromos Malakasiotis Ion Androutsopoulos Straintek, Athens, Greece Straintek, Athens, Greece Athens University of Economics [email protected] [email protected] and Business, Greece [email protected] Abstract suffer from abusive user comments, which dam- age their reputations and make them liable to fines, Experimenting with a new dataset of 1.6M e.g., when hosting comments encouraging illegal user comments from a news portal and an actions. They often employ moderators, who are existing dataset of 115K Wikipedia talk frequently overwhelmed, however, by the volume page comments, we show that an RNN op- and abusiveness of comments.3 Readers are dis- erating on word embeddings outpeforms appointed when non-abusive comments do not ap- the previous state of the art in moderation, pear quickly online because of moderation delays. which used logistic regression or an MLP Smaller news portals may be unable to employ classifier with character or word n-grams. moderators, and some are forced to shut down We also compare against a CNN operat- their comments sections entirely. ing on word embeddings, and a word-list We examine how deep learning (Goodfellow baseline. A novel, deep, classification- et al., 2016; Goldberg, 2016, 2017) can be em- specific attention mechanism improves the ployed to moderate user comments. We experi- performance of the RNN further, and can ment with a new dataset of approx. 1.6M manually also highlight suspicious words for free, moderated (accepted or rejected) user comments without including highlighted words in the from a Greek sports news portal (called Gazzetta), training data.
    [Show full text]
  • Reinforcement Learning and Trustworthy Autonomy
    Reinforcement Learning and Trustworthy Autonomy Jieliang Luo, Sam Green, Peter Feghali, George Legrady, and Çetin Kaya Koç Abstract Cyber-Physical Systems (CPS) possess physical and software inter- dependence and are typically designed by teams of mechanical, electrical, and software engineers. The interdisciplinary nature of CPS makes them difficult to design with safety guarantees. When autonomy is incorporated, design complexity and, especially, the difficulty of providing safety assurances are increased. Vision- based reinforcement learning is an increasingly popular family of machine learning algorithms that may be used to provide autonomy for CPS. Understanding how visual stimuli trigger various actions is critical for trustworthy autonomy. In this chapter we introduce reinforcement learning in the context of Microsoft’s AirSim drone simulator. Specifically, we guide the reader through the necessary steps for creating a drone simulation environment suitable for experimenting with vision- based reinforcement learning. We also explore how existing vision-oriented deep learning analysis methods may be applied toward safety verification in vision-based reinforcement learning applications. 1 Introduction Cyber-Physical Systems (CPS) are becoming increasingly powerful and complex. For example, standard passenger vehicles have millions of lines of code and tens of processors [1], and they are the product of years of planning and engineering J. Luo () · S. Green · P. Feghali · G. Legrady University of California Santa Barbara, Santa Barbara, CA, USA e-mail: [email protected]; [email protected]; [email protected]; [email protected] Ç. K. Koç Istinye˙ University, Istanbul,˙ Turkey Nanjing University of Aeronautics and Astronautics, Nanjing, China University of California Santa Barbara, Santa Barbara, CA, USA e-mail: [email protected] © Springer Nature Switzerland AG 2018 191 Ç.
    [Show full text]
  • Recommendations for the Facebook Content Review Board
    STANFORD Policy Practicum: Creating a Social Media Oversight Board Recommendations for the Facebook Content Review Board 2018-19 PRACTICUM RESEARCH TEAM: Shaimaa BAKR, Ph.D. Electrical Engineering ‘20 Madeline MAGNUSON, J.D. ’20 Fernando BERDION-DEL VALLE, J.D. ’20 Shawn MUSGRAVE, J.D. ’21 Isabella GARCIA-CAMARGO, B.S.’20 Ashwin RAMASWAMI, B.S. ‘21 Julia GREENBERG, J.D. ’19 Nora TAN, B.S. ’19 Tara IYER, B.S.’19 Marlena WISNIAK, LL.M. ’19 Alejandra LYNBERG, J.D. ’19 Monica ZWOLINSKI, J.D. ’19 INSTRUCTORS: Paul BREST, Faculty Director, Law and Policy Lab Daniel HO, William Benjamin Scott and Luna M. Scott Professor of Law Nathaniel PERSILY, James B. McClatchy Professor of Law Rob REICH, Faculty Director, Center for Ethics in Society TEACHING ASSISTANT: Liza STARR, J.D./M.B.A. ’21 SPRING POLICY CLIENT: Project on Democracy and the Internet, Stanford Center on Philanthropy and Civil Society 2019 559 Nathan Abbot Way Stanford, CA https://law.stanford.edu/education/only-at-sls/law-policy-lab// Contents Executive Summary ....................................................................................3 Introduction ................................................................................................8 I. The Need for Governing Principles ..........................................................9 II. Structure and Membership ...................................................................10 A. Selection Process for Board Members ..............................................................10 B. Membership Criteria .............................................................................................12
    [Show full text]
  • A Simple Platform for Reinforcement Learning of Simulated Flight Behaviors ⋆
    LM2020, 011, v2 (final): ’A Simple Platform for Reinforcement Learning of Simulated . 1 A Simple Platform for Reinforcement Learning of Simulated Flight Behaviors ⋆ Simon D. Levy Computer Science Department, Washington and Lee University, Lexington VA 24450, USA [email protected] Abstract. We present work-in-progress on a novel, open-source soft- ware platform supporting Deep Reinforcement Learning (DRL) of flight behaviors for Miniature Aerial Vehicles (MAVs). By using a physically re- alistic model of flight dynamics and a simple simulator for high-frequency visual events, our platform avoids some of the shortcomings associated with traditional MAV simulators. Implemented as an OpenAI Gym en- vironment, our simulator makes it easy to investigate the use of DRL for acquiring common behaviors like hovering and predation. We present preliminary experimental results on two such tasks, and discuss our cur- rent research directions. Our code, available as a public github repository, enables replication of our results on ordinary computer hardware. Keywords: Deep reinforcement learning Flight simulation Dynamic · · vision sensing. 1 Motivation Miniature Aerial Vehicles (MAVs, a.k.a. drones) are increasingly popular as a model for the behavior of insects and other flying animals [3]. The cost and risks associated with building and flying MAVs can however make such models inaccessible to many researchers. Even when such platforms are available, the number of experiments that must be run to collect sufficient data for paradigms like Deep Reinforcement Learning (DRL) makes simulation an attractive option. Popular MAV simulators like Microsoft AirSim [10], as well as our own sim- ulator [7], are built on top of video game engines like Unity or UnrealEngine4.
    [Show full text]
  • Chapter 3 Airsim Simulator
    MASTER THESIS Structured Flight Plan Interpreter for Drones in AirSim Francesco Rose SUPERVISED BY Cristina Barrado Muxi Universitat Politècnica de Catalunya Master in Aerospace Science & Technology January 2020 This Page Intentionally Left Blank Structured Flight Plan Interpreter for Drones in AirSim BY Francesco Rose DIPLOMA THESIS FOR DEGREE Master in Aerospace Science and Technology AT Universitat Politècnica de Catalunya SUPERVISED BY: Cristina Barrado Muxi Computer Architecture Department “E come i gru van cantando lor lai, faccendo in aere di sé lunga riga, così vid’io venir, traendo guai, ombre portate da la detta briga” Dante, Inferno, V Canto This Page Intentionally Left Blank ABSTRACT Nowadays, several Flight Plans for drones are planned and managed taking advantages of Extensible Markup Language (XML). In the mean time, to test drones performances as well as their behavior, simulators usefulness has been increasingly growing. Hence, what it takes to make a simulator capable of receiving commands from an XML file is a dynamic interface. The main objectives of this master thesis are basically three. First of all, the handwriting of an XML flight plan (FP) compatible with the simulator environment chosen. Then, the creation of a dynamic interface that can read whatever XML FP and that will transmit commands to the drone. Finally, using the simulator, it will be possible to test both interface and flight plan. Moreover, a dynamic interface aimed at managing two or more drones in parallel has been built and implemented as extra objective of this master thesis. In addition, assuming that two drones will be used to test this interface, it is required the handwriting of two more FPs.
    [Show full text]
  • An Aerial Mixed-Reality Environment for First-Person-View Drone Flying
    applied sciences Article An Aerial Mixed-Reality Environment for First-Person-View Drone Flying Dong-Hyun Kim , Yong-Guk Go and Soo-Mi Choi * Department of Computer Science and Engineering, Sejong University, Seoul 05006, Korea; [email protected] (D.-H.K.); lattechiff[email protected] (Y.-G.G.) * Correspondence: [email protected] Received: 27 June 2020; Accepted: 3 August 2020; Published: 6 August 2020 Abstract: A drone be able to fly without colliding to preserve the surroundings and its own safety. In addition, it must also incorporate numerous features of interest for drone users. In this paper, an aerial mixed-reality environment for first-person-view drone flying is proposed to provide an immersive experience and a safe environment for drone users by creating additional virtual obstacles when flying a drone in an open area. The proposed system is effective in perceiving the depth of obstacles, and enables bidirectional interaction between real and virtual worlds using a drone equipped with a stereo camera based on human binocular vision. In addition, it synchronizes the parameters of the real and virtual cameras to effectively and naturally create virtual objects in a real space. Based on user studies that included both general and expert users, we confirm that the proposed system successfully creates a mixed-reality environment using a flying drone by quickly recognizing real objects and stably combining them with virtual objects. Keywords: aerial mixed-reality; drones; stereo cameras; first-person-view; head-mounted display 1. Introduction As cameras mounted on drones can easily capture photographs of places that are inaccessible to people, drones have been recently applied in various fields, such as aerial photography [1], search and rescue [2], disaster management [3], and entertainment [4].
    [Show full text]
  • Online Communities As Semicommons
    The Virtues of Moderation: Online Communities as Semicommons James Grimmelmann* IP Scholars workshop draft: not intended for broad distribution or citation I. INTRODUCTION................................................................................................................................................2 II. BACKGROUND ..................................................................................................................................................4 A. PUBLIC AND PRIVATE GOODS ...........................................................................................................................4 B. THE TRAGIC STORY ..........................................................................................................................................6 C. THE COMEDIC STORY .......................................................................................................................................9 D. LAYERING.......................................................................................................................................................12 III. ONLINE SEMICOMMONS .............................................................................................................................14 A. A FORMAL MODEL .........................................................................................................................................14 B. COMMONS VIRTUES........................................................................................................................................17
    [Show full text]
  • Peter Van Der Perk
    Eindhoven University of Technology MASTER A distributed safety mechanism for autonomous vehicle software using hypervisors van der Perk, P.J. Award date: 2019 Link to publication Disclaimer This document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Student theses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the document as presented in the repository. The required complexity or quality of research of student theses may vary by program, and the required minimum study period may vary in duration. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain Department of Electrical Engineering Electronic Systems Research Group A Distributed Safety Mechanism for Autonomous Vehicle Software Using Hypervisors graduation project Peter van der Perk Supervisors: prof.dr. Kees Goossens dr. Andrei Terechko version 1.0 Eindhoven, June 2019 Abstract Autonomous vehicles rely on cyber-physical systems to provide comfort and safety to the passen- gers. The objective of safety designs is to avoid unacceptable risk of physical injury to people. Reaching this objective, however, is very challenging because of the growing complexity of both the Electronic Control Units (ECUs) and software architectures required for autonomous operation.
    [Show full text]