Written Statement for the Record July 16, 2019 Genius Media Group, Inc
Total Page:16
File Type:pdf, Size:1020Kb
Written Statement for the Record July 16, 2019 Genius Media Group, Inc. Subcommittee on Antitrust, Commercial and Administrative Law Committee on the Judiciary U.S. House of Representatives Attachment C: Google’s Search Quality Rater Guidelines (May 2019) Accessed at: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsear chworks/assets/searchqualityevaluatorguidelines.pdf General Guidelines May 16, 2019 General Guidelines Overview 5 0.0 Introduction to Search Quality Rating 6 0.1 The Purpose of Search Quality Rating 6 0.2 Raters Must Represent the User 6 0.3 Browser Requirements 6 0.4 Ad Blocking Extensions 6 0.5 Internet Safety Information 6 Part 1: Page Quality Rating Guideline 7 1.0 Introduction to Page Quality Rating 7 2.0 Understanding Webpages and Websites 7 2.1 Important Definitions 7 2.2 What is the Purpose of a Webpage? 8 2.3 Your Money or Your Life (YMYL) Pages 9 2.4 Understanding Webpage Content 9 2.4.1 Identifying the Main Content (MC) 9 2.4.2 Identifying the Supplementary Content (SC) 10 2.4.3 Identifying Advertisements/Monetization (Ads) 10 2.4.4 Summary of the Parts of the Page 11 2.5 Understanding the Website 11 2.5.1 Finding the Homepage 11 2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page 13 2.5.3 Finding About Us, Contact Information, and Customer Service Information 13 2.6 Reputation of the Website or Creator of the Main Content 14 2.6.1 Research on the Reputation of the Website or Creator of the Main Content 15 2.6.2 Sources of Reputation Information 15 2.6.3 Customer Reviews of Stores/Businesses 15 2.6.4 How to Search for Reputation Information 15 2.6.5 What to Do When You Find No Reputation Information 17 3.0 Overall Page Quality Rating 18 3.1 Page Quality Rating: Most Important Factors 18 3.2 Expertise, Authoritativeness, and Trustworthiness (E-A-T) 18 4.0 High Quality Pages 19 4.1 Characteristics of High Quality Pages 19 4.2 A Satisfying Amount of High Quality Main Content 20 4.3 Clear and Satisfying Website Information: Who is Responsible and Customer Service 20 4.4 Positive Reputation 20 4.5 A High Level of Expertise/Authoritativeness/Trustworthiness (E-A-T) 21 4.6 Examples of High Quality Pages 21 5.0 Highest Quality Pages 25 5.1 Very High Quality MC 25 5.2 Very Positive Reputation 25 5.3 Very High Level of E-A-T 25 Copyright 2019 1 5.4 Examples of Highest Quality Pages 26 6.0 Low Quality Pages 31 6.1 Lacking Expertise, Authoritativeness, or Trustworthiness (E-A-T) 31 6.2 Low Quality Main Content 31 6.3 Unsatisfying Amount of Main Content 32 6.4 Distracting Ads/SC 32 6.5 Mixed or Mildly Negative Reputation of the Website or Creator of the Main Content 32 6.6 Unsatisfying Amount of Information about the Website or Creator of the Main Content 33 6.7 Examples of Low Quality Pages 33 7.0 Lowest Quality Pages 36 7.1 Lack of Purpose Pages 37 7.2 Pages that Fail to Achieve Their Purpose 37 7.2.1 Lowest E-A-T 37 7.2.2 No/Little Main Content 37 7.2.3 Lowest Quality Main Content 38 7.2.4 Copied Main Content 38 7.2.5 How to Determine if Content is Copied 39 7.2.6 Auto-Generated Main Content 40 7.2.7 Obstructed or Inaccessible Main Content 40 7.2.8 Inadequate Information about the Website or Creator of the Main Content 40 7.2.9 Unmaintained Websites, and Hacked, Defaced, or Spammed Pages 40 7.3 Pages that Potentially Spread Hate 41 7.4 Potentially Harmful Pages 41 7.4.1 Encourage Harm 41 7.4.2 Malicious Pages 41 7.4.3 Negative or Malicious Reputation 42 7.5 Pages that Potentially Misinform Users 42 7.6 Pages that Potentially Deceive Users 43 7.6.1 Deceptive Page Purpose 43 7.6.2 Deceptive Page Design 43 7.7 Examples of Lowest Quality Pages 45 8.0 Medium Quality Pages 52 8.1 Examples of Medium Quality Pages 53 9.0 Page Quality Rating Tasks 55 9.1 Instructions for Rating Page Quality Tasks 56 9.1.1 Rating on Your Phone 56 9.2 Reputation and E-A-T: Website or the Creators of the Main Content? 56 10.0 Page Quality Criteria for Specific Types of Pages 57 10.1 Ratings for Encyclopedia Pages 57 10.2 Ratings for Pages with Error Messages or No MC 57 10.3 Ratings for Forums and Q&A pages 58 11.0 Page Quality Rating FAQs 62 Part 2: Understanding Mobile User Needs 64 12.0 Understanding Mobile Users, Mobile Queries, and Mobile Results 64 12.1 Important Rating Definitions and Ideas 65 Copyright 2019 2 12.2 Understanding the Query 66 12.3 Locale and User Location 66 12.4 Queries with an Explicit Location 67 12.5 Queries with Multiple Meanings 67 12.6 Query Meanings Can Change Over Time 68 12.7 Understanding User Intent 69 12.7.1 Know and Know Simple Queries 69 12.7.2 Do and Device Action Queries 70 12.7.3 Website Queries 71 12.7.4 Visit-in-Person Queries and User Location 72 12.7.5 Queries with Multiple User Intents 75 12.8 Understanding Result Blocks 75 12.8.1 Web Search Result Block Examples 75 12.8.2 Special Content Result Block Examples 76 12.8.3 Device Action Result Block Examples 78 12.8.4 How Device Action Results are Displayed in Rating Tasks 80 12.9 Rating on Your Phone Issues 83 Part 3: Needs Met Rating Guideline 84 13.0 Rating Using the Needs Met Scale 84 13.1 Rating Result Blocks: Block Content and Landing Pages 84 13.2 Fully Meets (FullyM) 87 13.2.1 Examples of Fully Meets (FullyM) Result Blocks 87 13.2.2 Examples of Queries that Cannot Have Fully Meets Results 96 13.3 Highly Meets (HM) 97 13.3.1 Examples of Highly Meets (HM) Result Blocks 97 13.4 Moderately Meets (MM) 107 13.4.1 Examples of Moderately Meets (MM) Result Blocks 107 13.5 Slightly Meets (SM) 109 13.5.1 Examples of Slightly Meets (SM) Result Blocks 109 13.6 Fails to Meet (FailsM) 112 13.6.1 Examples of Fails to Meet (FailsM) Result Blocks 112 14.0 Rating Porn, Foreign Language, Did Not Load, and Upsetting-Offensive Results 123 14.1 Porn Flag 123 14.2 Needs Met Rating for Porn Results 123 14.2.1 Needs Met Rating for Clear Non-Porn Intent Queries 123 14.2.2 Needs Met Rating for Possible Porn Intent Queries 124 14.2.3 Needs Met Rating for Clear Porn Intent Queries 124 14.3 Reporting Illegal Images 125 14.4 Foreign Language Flag 125 14.4.1 Using the Foreign Language Flag 125 14.4.2 Needs Met Rating for Foreign Language Results 126 14.5 Did Not Load Flag 128 14.5.1 Using the Did Not Load Flag 128 14.5.2 Needs Met Rating for Did Not Load Results 129 14.6 Upsetting-Offensive Flag 129 Copyright 2019 3 14.6.1 Using the Upsetting-Offensive Flag 130 14.6.2 Needs Met Rating for Upsetting-Offensive Tolerant Queries 132 15.0 The Relationship between Page Quality and Needs Met 134 16.0 Rating Queries with Multiple Interpretations and Intents 136 16.1 Rating Queries with Both Website and Visit-in-Person Intent 136 17.0 Specificity of Queries and Landing Pages 137 18.0 Needs Met Rating and Freshness 145 19.0 Misspelled and Mistyped Queries and Results 147 19.1 Misspelled and Mistyped Queries 147 19.2 Name Queries 148 20.0 Non-Fully Meets Results for URL Queries 148 21.0 Product Queries: Importance of Browsing and Researching 150 22.0 Rating Visit-in-Person Intent Queries 151 22.1 Examples Where User Location Does (and Does Not) Matter 152 23.0 Rating English Language Results in Non-English Locales 154 23.1 Examples of English (and Non-English) Results in Non-English Locales 155 Appendix: Using the Evaluation Platform 160 24.0 Overview 160 25.0 Acquiring Tasks 160 26.0 Rating Tasks Using the Rating Interface 160 27.0 Releasing Tasks 161 28.0 Understanding the User Location on the Task Page 163 29.0 Reporting Duplicate Results in Tasks 163 29.1 Pre-Identified Duplicates 163 29.2 Rater-Identified Duplicates 164 29.3 Reporting Duplicate Results 165 30.0 Simplified Needs Met Tasks 166 Copyright 2019 4 General Guidelines Overview Welcome to the Search Quality Rating Program! As a Search Quality Rater, you will work on many different types of rating projects. The General Guidelines primarily cover Page Quality (PQ) rating and Needs Met (NM) rating; however, the concepts are also important for many other types of rating tasks. For brevity, we refer to “Search Quality Raters” as “raters” in these guidelines. Copyright 2019 5 0.0 Introduction to Search Quality Rating 0.1 The Purpose of Search Quality Rating Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that are helpful for users in their specific language and locale. It is important that you are familiar with and comfortable using a search engine. We encourage you to be an expert in Google search! For example, experiment with using operators (e.g., quotes or a dash) in your searches or try using Google’s a dvanced search option.