Building Web Reputation Systems
Total Page:16
File Type:pdf, Size:1020Kb
Building Web Reputation Systems F. Randall Farmer and Bryce Glass Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Building Web Reputation Systems by F. Randall Farmer and Bryce Glass Copyright © 2010 F. Randall Farmer and Bryce Glass. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Mary E. Treseler Indexer: Ellen Troutman Zaig Production Editor: Loranah Dimant Cover Designer: Karen Montgomery Copyeditor: Genevieve d’Entremont Interior Designer: David Futato Proofreader: Loranah Dimant Illustrator: Robert Romano Printing History: March 2010: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Building Web Reputation Systems, the image of a Pionus parrot, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. TM This book uses RepKover™, a durable and flexible lay-flat binding. ISBN: 978-0-596-15979-5 [M] 1267644619 Table of Contents Preface . .................................................................... ix Part I. Reputation Defined and Illustrated 1. Reputation Systems Are Everywhere . ...................................... 3 An Opinionated Conversation 3 People Have Reputations, but So Do Things 4 Reputation Takes Place Within a Context 4 We Use Reputation to Make Better Decisions 5 The Reputation Statement 6 Explicit: Talk the Talk 6 Implicit: Walk the Walk 6 The Minimum Reputation Statement 6 Reputation Systems Bring Structure to Chaos 7 Reputation Systems Deeply Affect Our Lives 8 Local Reputation: It Takes a Village 8 Global Reputation: Collective Intelligence 9 FICO: A Study in Global Reputation and Its Challenges 10 Web FICO? 12 Reputation on the Web 12 Attention Doesn’t Scale 13 There’s a Whole Lotta Crap Out There 13 People Are Good. Basically. 15 The Reputation Virtuous Circle 17 Who’s Using Reputation Systems? 18 Challenges in Building Reputation Systems 19 Related Subjects 20 Conceptualizing Reputation Systems 20 2. A (Graphical) Grammar for Reputation . .................................... 21 The Reputation Statement and Its Components 22 iii Reputation Sources: Who or What Is Making a Claim? 23 Reputation Claims: What Is the Target’s Value to the Source? On What Scale? 24 Reputation Targets: What (or Who) Is the Focus of a Claim? 25 Molecules: Constructing Reputation Models Using Messages and Processes 26 Messages and Processes 26 Reputation Model Explained: Vote to Promote 28 Building on the Simplest Model 29 Complex Behavior: Containers and Reputation Statements As Targets 30 Solutions: Mixing Models to Make Systems 33 From Reputation Grammar to… 35 Part II. Extended Elements and Applied Examples 3. Building Blocks and Reputation Tips . ..................................... 39 Extending the Grammar: Building Blocks 39 The Data: Claim Types 39 Processes: Computing Reputation 46 Routers: Messages, Decisions, and Termination 54 Practitioner’s Tips: Reputation Is Tricky 57 The Power and Costs of Normalization 57 Liquidity: You Won’t Get Enough Input 58 Bias, Freshness, and Decay 60 Implementer’s Notes 64 Making Buildings from Blocks 65 4. Common Reputation Models . ............................................ 67 Simple Models 67 Favorites and Flags 68 This-or-That Voting 69 Ratings 70 Reviews 70 Points 71 Karma 72 Combining the Simple Models 74 User Reviews with Karma 75 eBay Seller Feedback Karma 78 Flickr Interestingness Scores for Content Quality 82 When and Why Simple Models Fail 89 Party Crashers 89 Keep Your Barn Door Closed (but Expect Peeking) 91 Reputation from Theory to Practice 94 iv | Table of Contents Part III. Building Web Reputation Systems 5. Planning Your System’s Design . .......................................... 97 Asking the Right Questions 97 What Are Your Goals? 98 Content Control Patterns 102 Incentives for User Participation, Quality, and Moderation 111 Consider Your Community 121 Better Questions 123 6. Objects, Inputs, Scope, and Mechanism . 125 The Objects in Your System 125 Architect, Understand Thyself 126 What Makes for a Good Reputable Entity? 129 Determining Inputs 131 User Actions Make Good Inputs 131 But Other Types of Inputs Are Important, Too 134 Good Inputs 135 Common Explicit Inputs 136 Common Implicit Inputs 143 Constraining Scope 146 Context Is King 146 Limit Scope: The Rule of Email 148 Applying Scope to Yahoo! EuroSport Message Board Reputation 149 Generating Reputation: Selecting the Right Mechanisms 150 The Heart of the Machine: Reputation Does Not Stand Alone 151 Common Reputation Generation Mechanisms and Patterns 151 Practitioner’s Tips: Negative Public Karma 161 Draw Your Diagram 163 7. Displaying Reputation . ................................................ 165 How to Use a Reputation: Three Questions 165 Who Will See a Reputation? 166 To Show or Not to Show? 166 Personal Reputations: For the Owner’s Eyes Only 169 Personal and Public Reputations Combined 171 Public Reputations: Widely Visible 171 Corporate Reputations Are Internal Use Only: Keep Them Hush-hush 172 How Will You Use Reputation to Modify Your Site’s Output? 172 Reputation Filtering 173 Reputation Ranking and Sorting 173 Reputation Decisions 174 Table of Contents | v Content Reputation Is Very Different from Karma 175 Content Reputation 176 Karma 176 Reputation Display Formats 178 Reputation Display Patterns 180 Normalized Score to Percentage 180 Points and Accumulators 182 Statistical Evidence 183 Levels 185 Ranked Lists 189 Practitioner’s Tips 194 Leaderboards Considered Harmful 194 Going Beyond Displaying Reputation 196 8. Using Reputation: The Good, The Bad, and the Ugly . 197 Up with the Good 198 Rank-Order Items in Lists and Search Results 199 Content Showcases 200 Down with the Bad 204 Configurable Quality Thresholds 205 Expressing Dissatisfaction 206 Out with the Ugly 207 Reporting Abuse 207 Teach Your Users How to Fish 209 Inferred Reputation for Content Submissions 210 A Private Conversation 212 Course-Correcting Feedback 213 Reputation Is Identity 214 On the User Profile 216 At the Point of Attribution 219 To Differentiate Within Listings 220 Putting It All Together 221 9. Application Integration, Testing, and Tuning . 223 Integrating with Your Application 223 Implementing Your Reputation Model 223 Rigging Inputs 225 Applied Outputs 225 Beware Feedback Loops! 226 Plan for Change 226 Testing Your System 227 Bench Testing Reputation Models 228 Environmental (Alpha) Testing Reputation Models 229 vi | Table of Contents Predeployment (Beta) Testing Reputation Models 230 Tuning Your System 232 Tuning for ROI: Metrics 232 Tuning for Behavior 236 Tuning for the Future 241 Learning by Example 241 10. Case Study: Yahoo! Answers Community Content Moderation . 243 What Is Yahoo! Answers? 243 A Marketplace for Questions and Yahoo! Answers 244 Attack of the Trolls 245 Built with Reputation 247 Avengers Assemble! 248 Initial Project Planning 249 Setting Goals 249 Who Controls the Content? 250 Incentives 250 The High-Level Project Model 251 Objects, Inputs, Scope, and Mechanism 252 The Objects 252 Limiting Scope 254 An Evolving Model 255 Displaying Reputation 268 Who Will See the Reputation? 268 How Will the Reputation Be Used to Modify Your Site’s Output? 268 Is This Reputation for a Content Item or a Person? 268 Using Reputation: The…Ugly 269 Application Integration, Testing, and Tuning 270 Application Integration 270 Testing Is Harder Than You Think 270 Lessons in Tuning: Users Protecting Their Power 271 Deployment and Results 273 Operational and Community Adjustments 274 Adieu 276 A. The Reputation Framework . ............................................ 279 B. Related Resources . ................................................... 303 Index . 307 Table of Contents | vii Preface What Is This Book About? Today’s Web is the product of over a billion hands and minds. Around the clock and around the globe, people are pumping out contributions small and large: full-length features on Vimeo, video shorts on YouTube, comments on Blogger, discussions on Yahoo! Groups, and tagged-and-titled Del.icio.us bookmarks. User-generated content and robust crowd participation have become the hallmarks of Web 2.0. But the booming popularity of social media has brought with it a whole new set of challenges for those who create websites and online communities, as well as for the users who consume those sites: • Problems of scale (how to manage—and present—an overwhelming inflow of user contributions) • Problems of quality (how to tell the good stuff from the bad) • Problems of engagement (how to reward contributors in a way that keeps them coming back) • Problems of moderation (how to “stamp out” the worst stuff quickly