DOCSLIB.ORG

By Nilesh Bansal a Thesis Submitted in Conformity with the Requirements

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
By Nilesh Bansal a Thesis Submitted in Conformity with the Requirements ONLINE ANALYSIS OF HIGH-VOLUME SOCIAL TEXT STEAMS by Nilesh Bansal A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Computer Science University of Toronto ⃝c Copyright 2013 by Nilesh Bansal Abstract Online Analysis of High-Volume Social Text Steams Nilesh Bansal Doctor of Philosophy Graduate Department of Computer Science University of Toronto 2013 Social media is one of the most disruptive developments of the past decade. The impact of this information revolution has been fundamental on our society. Information dissemination has never been cheaper and users are increasingly connected with each other. The line between content producers and consumers is blurred, leaving us with abundance of data produced in real-time by users around the world on multitude of topics. In this thesis we study techniques to aid an analyst in uncovering insights from this new media form which is modeled as a high volume social text stream. The aim is to develop practical algorithms with focus on the ability to scale, amenability to reliable operation, usability, and ease of implementation. Our work lies at the intersection of building large scale real world systems and developing theoretical foundation to support the same. We identify three key predicates to enable online methods for analysis of social data, namely : • Persistent Chatter Discovery to explore topics discussed over a period of time, • Cross-referencing Media Sources to initiate analysis using a document as the query, and • Contributor Understanding to create aggregate expertise and topic summaries of authors contributing online. The thesis defines each of the predicates in detail and covers proposed techniques, their practical applicability, and detailed experimental results to establish accuracy and scalability for each of the three predicates. We present BlogScope, the core data aggregation and management platform, developed as part of the thesis to enable implementation of the key predicates in real world setting. The system provides a web based user interface for searching social media conversations and analyzing the results in multitude of ways. BlogScope, and its modified versions, index tens to hundreds of billions of text documents while providing interactive query times. Specifically, BlogScope has been crawling 50 million active blogs with 3.25 billion blog posts. Same techniques have also been successfully tested on a Twitter stream of data, adding thousands of new Tweets every second and archiving over 30 billion documents. The social graph part of our database consists of 26 million Twitter user nodes with 17 billion follower edges. The BlogScope system has been used by over 10,000 unique visitors a day, and the commercial version of the system is used by thousands of enterprise clients globally. As social media continues to evolve at an exponential pace, there is a lot that still needs to be studied. The thesis concludes by outlining some of possible future research directions. ii Contents 1 Introduction 1 1.1 Overview . 3 2 BlogScope: System Description 5 2.1 Related Work . 6 2.2 Analyzing The Blogosphere . 7 2.2.1 Popularity Curve . 7 2.2.2 Keyword Bursts . 9 2.2.3 Keyword Correlations . 10 2.2.4 Hot Keywords . 11 2.2.5 Spatio-Temporal Search . 12 2.2.6 Authoritative Blog Ranking . 12 2.2.7 Interface . 13 2.3 System Architecture . 13 2.3.1 The BlogScope Crawler . 14 2.3.2 Spam Removal . 15 2.3.3 Searching and Indexing . 16 2.3.4 Spatial Component . 16 2.3.5 Popularity and Bursts . 16 2.3.6 Keyword Correlations . 17 2.3.7 Authoritative Ranking . 18 2.3.8 Hot Keywords . 19 3 Persistent Chatter Discovery by Identifying Stable Clusters 20 3.1 Related Work . 22 3.2 Cluster Generation . 23 3.3 Stable Clusters . 26 3.3.1 The Cluster Graph . 28 3.3.2 Breadth First Search . 28 3.3.3 Depth First Search . 30 3.3.4 Adapting the Threshold Algorithm . 33 3.3.5 Normalized Stable Clusters . 35 3.3.6 Online Version . 36 3.4 Experiments . 36 iii 3.4.1 Cluster Generation . 37 3.4.2 Stable Clusters . 37 3.4.3 Qualitative Results . 41 4 Persistent Chatter Discovery in Presence of Topic Hierarchies 44 4.1 Related Work . 45 4.2 Problem Definition . 46 4.3 Aggregation in Presence of Hierarchies . 47 4.4 Aggregation with Probabilistic Guarantees . 48 4.4.1 Probabilistic TA Assuming Geometric Distribution . 48 4.4.2 Probabilistic TA Using Histograms . 50 4.5 Deterministic Aggregation with Precision Guarantees . 52 4.6 Preprocessing for Aggregation . 54 4.6.1 Offline Preprocessing of Hierarchies . 56 4.6.2 Online Processing of Hierarchies . 56 4.7 Experiments . 56 4.7.1 Evaluating pH − RA ..................................... 56 4.7.2 Sparse Interval Set . 60 4.7.3 Real Data . 61 5 Cross-referencing Media Sources by Querying by Document 63 5.1 Related Work . 64 5.2 Problem Definition . 66 5.3 Phrase Extraction: QBD . 68 5.3.1 Extracting Candidate Phrases . 68 5.3.2 Scoring Candidate Phrases . 69 5.4 Using Wikipedia: QBD-W . 71 5.5 Experiments . 76 5.5.1 Data Sets and Alternative Techniques . 76 5.5.2 Quality of Phrase Extraction . 77 5.5.3 Quality of Document Retrieval . 78 6 Contributor Understanding with Social Author Profiles 82 6.1 Peckalytics . 84 6.1.1 Technology . 85 6.1.2 Usage Experience . 86 6.2 Topic Signatures . 87 6.2.1 Constructing Topic Signatures . 90 6.3 Related Work . 91 6.4 Problem Formulation . 91 6.5 Aggregate Signatures . 93 6.6 Temporal Evolution of Aggregate Signatures . 95 6.6.1 Minimizing RMSE . 96 6.6.2 Minimizing Hamming Distance . 96 iv 6.6.3 Algorithm . 96 6.7 Experiments . 97 6.7.1 Presentation . 98 6.7.2 Quantitative Evaluation . 98 6.7.3 Qualitative Evaluation . 99 7 Conclusion 109 7.1 Challenges . 109 7.2 Contributions . ..
Load more

Recommended publications
  • Uncovering Social Network Sybils in the Wild
    Uncovering Social Network Sybils in the Wild ZHI YANG, Peking University 2 CHRISTO WILSON, University of California, Santa Barbara XIAO WANG, Peking University TINGTING GAO,RenrenInc. BEN Y. ZHAO, University of California, Santa Barbara YAFEI DAI, Peking University Sybil accounts are fake identities created to unfairly increase the power or resources of a single malicious user. Researchers have long known about the existence of Sybil accounts in online communities such as file-sharing systems, but they have not been able to perform large-scale measurements to detect them or measure their activities. In this article, we describe our efforts to detect, characterize, and understand Sybil account activity in the Renren Online Social Network (OSN). We use ground truth provided by Renren Inc. to build measurement-based Sybil detectors and deploy them on Renren to detect more than 100,000 Sybil accounts. Using our full dataset of 650,000 Sybils, we examine several aspects of Sybil behavior. First, we study their link creation behavior and find that contrary to prior conjecture, Sybils in OSNs do not form tight-knit communities. Next, we examine the fine-grained behaviors of Sybils on Renren using clickstream data. Third, we investigate behind-the-scenes collusion between large groups of Sybils. Our results reveal that Sybils with no explicit social ties still act in concert to launch attacks. Finally, we investigate enhanced techniques to identify stealthy Sybils. In summary, our study advances the understanding of Sybil behavior on OSNs and shows that Sybils can effectively avoid existing community-based Sybil detectors. We hope that our results will foster new research on Sybil detection that is based on novel types of Sybil features.
    [Show full text]
  • Svms for the Blogosphere: Blog Identification and Splog Detection
    SVMs for the Blogosphere: Blog Identification and Splog Detection Pranam Kolari∗, Tim Finin and Anupam Joshi University of Maryland, Baltimore County Baltimore MD {kolari1, finin, joshi}@umbc.edu Abstract Most blog search engines identify blogs and index con- tent based on update pings received from ping servers1 or Weblogs, or blogs have become an important new way directly from blogs, or through crawling blog directories to publish information, engage in discussions and form communities. The increasing popularity of blogs has and blog hosting services. To increase their coverage, blog given rise to search and analysis engines focusing on the search engines continue to crawl the Web to discover, iden- “blogosphere”. A key requirement of such systems is to tify and index blogs. This enables staying ahead of compe- identify blogs as they crawl the Web. While this ensures tition in a domain where “size does matter”. Even if a web that only blogs are indexed, blog search engines are also crawl is inessential for blog search engines, it is still pos- often overwhelmed by spam blogs (splogs). Splogs not sible that processed update pings are from non-blogs. This only incur computational overheads but also reduce user requires that the source of the pings need to be verified as a satisfaction. In this paper we first describe experimental blog prior to indexing content.2 results of blog identification using Support Vector Ma- In the first part of this paper we address blog identifica- chines (SVM). We compare results of using different feature sets and introduce new features for blog iden- tion by experimenting with different feature sets.
    [Show full text]
  • Blogosphere: Research Issues, Tools, and Applications
    Blogosphere: Research Issues, Tools, and Applications Nitin Agarwal Huan Liu Computer Science and Engineering Department Arizona State University Tempe, AZ 85287 fNitin.Agarwal.2, [email protected] ABSTRACT ging. Acknowledging this fact, Times has named \You" as the person of the year 2006. This has created a consider- Weblogs, or Blogs, have facilitated people to express their able shift in the way information is assimilated by the indi- thoughts, voice their opinions, and share their experiences viduals. This paradigm shift can be attributed to the low and ideas. Individuals experience a sense of community, a barrier to publication and open standards of content genera- feeling of belonging, a bonding that members matter to one tion services like blogs, wikis, collaborative annotation, etc. another and their niche needs will be met through online These services have allowed the mass to contribute and edit interactions. Its open standards and low barrier to publi- articles publicly. Giving access to the mass to contribute cation have transformed information consumers to produc- or edit has also increased collaboration among the people ers. This has created a plethora of open-source intelligence, unlike previously where there was no collaboration as the or \collective wisdom" that acts as the storehouse of over- access to the content was limited to a chosen few. Increased whelming amounts of knowledge about the members, their collaboration has developed collective wisdom on the Inter- environment and the symbiosis between them. Nonetheless, net. \We the media" [21], is a phenomenon named by Dan vast amounts of this knowledge still remain to be discovered Gillmor: a world in which \the former audience", not a few and exploited in its suitable way.
    [Show full text]
  • Web Spam Taxonomy
    Web Spam Taxonomy Zolt´an Gy¨ongyi Hector Garcia-Molina Computer Science Department Computer Science Department Stanford University Stanford University [email protected] [email protected] Abstract techniques, but as far as we know, they still lack a fully effective set of tools for combating it. We believe Web spamming refers to actions intended to mislead that the first step in combating spam is understanding search engines into ranking some pages higher than it, that is, analyzing the techniques the spammers use they deserve. Recently, the amount of web spam has in- to mislead search engines. A proper understanding of creased dramatically, leading to a degradation of search spamming can then guide the development of appro- results. This paper presents a comprehensive taxon- priate countermeasures. omy of current spamming techniques, which we believe To that end, in this paper we organize web spam- can help in developing appropriate countermeasures. ming techniques into a taxonomy that can provide a framework for combating spam. We also provide an overview of published statistics about web spam to un- 1 Introduction derline the magnitude of the problem. There have been brief discussions of spam in the sci- As more and more people rely on the wealth of informa- entific literature [3, 6, 12]. One can also find details for tion available online, increased exposure on the World several specific techniques on the Web itself (e.g., [11]). Wide Web may yield significant financial gains for in- Nevertheless, we believe that this paper offers the first dividuals or organizations. Most frequently, search en- comprehensive taxonomy of all important spamming gines are the entryways to the Web; that is why some techniques known to date.
    [Show full text]
  • Robust Detection of Comment Spam Using Entropy Rate
    Robust Detection of Comment Spam Using Entropy Rate Alex Kantchelian Justin Ma Ling Huang UC Berkeley UC Berkeley Intel Labs [email protected] [email protected] [email protected] Sadia Afroz Anthony D. Joseph J. D. Tygar Drexel University, Philadelphia UC Berkeley UC Berkeley [email protected] [email protected] [email protected] ABSTRACT Keywords In this work, we design a method for blog comment spam detec- Spam filtering, Comment spam, Content complexity, Noisy label, tion using the assumption that spam is any kind of uninformative Logistic regression content. To measure the “informativeness” of a set of blog com- ments, we construct a language and tokenization independent met- 1. INTRODUCTION ric which we call content complexity, providing a normalized an- swer to the informal question “how much information does this Online social media have become indispensable, and a large part text contain?” We leverage this metric to create a small set of fea- of their success is that they are platforms for hosting user-generated tures well-adjusted to comment spam detection by computing the content. An important example of how users contribute value to a content complexity over groupings of messages sharing the same social media site is the inclusion of comment threads in online ar- author, the same sender IP, the same included links, etc. ticles of various kinds (news, personal blogs, etc). Through com- We evaluate our method against an exact set of tens of millions ments, users have an opportunity to share their insights with one of comments collected over a four months period and containing another.
    [Show full text]
  • Spam in Blogs and Social Media
    ȱȱȱȱ ȱ Pranam Kolari, Tim Finin Akshay Java, Anupam Joshi March 25, 2007 ȱ • Spam on the Internet –Variants – Social Media Spam • Reason behind Spam in Blogs • Detecting Spam Blogs • Trends and Issues • How can you help? • Conclusions Pranam Kolari is a UMBC PhD Tim Finin is a UMBC Professor student. His dissertation is on with over 30 years of experience spam blog detection, with tools in the applying AI to information developed in use both by academia and systems, intelligent interfaces and industry. He has active research interest robotics. Current interests include social in internal corporate blogs, the Semantic media, the Semantic Web and multi- Web and blog analytics. agent systems. Akshay Java is a UMBC PhD student. Anupam Joshi is a UMBC Pro- His dissertation is on identify- fessor with research interests in ing influence and opinions in the broad area of networked social media. His research interests computing and intelligent systems. He include blog analytics, information currently serves on the editorial board of retrieval, natural language processing the International Journal of the Semantic and the Semantic Web. Web and Information. Ƿ Ȭȱ • Early form seen around 1992 with MAKE MONEY FAST • 80-85% of all e-mail traffic is spam • In numbers 2005 - (June) 30 billion per day 2006 - (June) 55 billion per day 2006 - (December) 85 billion per day 2007 - (February) 90 billion per day Sources: IronPort, Wikipedia http://www.ironport.com/company/ironport_pr_2006-06-28.html ȱȱǵ • “Unsolicited usually commercial e-mail sent to a large
    [Show full text]
  • Advances in Online Learning-Based Spam Filtering
    ADVANCES IN ONLINE LEARNING-BASED SPAM FILTERING A dissertation submitted by D. Sculley, M.Ed., M.S. In partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science TUFTS UNIVERSITY August 2008 ADVISER: Carla E. Brodley Acknowledgments I would like to take this opportunity to thank my advisor Carla Brodley for her patient guidance, my parents David and Paula Sculley for their support and en- couragement, and my bride Jessica Evans for making everything worth doing. I gratefully acknowledge Rediff.com for funding the writing of this disserta- tion. D. Sculley TUFTS UNIVERSITY August 2008 ii Abstract The low cost of digital communication has given rise to the problem of email spam, which is unwanted, harmful, or abusive electronic content. In this thesis, we present several advances in the application of online machine learning methods for auto- matically filtering spam. We detail a sliding-window variant of Support Vector Machines that yields state of the art results for the standard online filtering task. We explore a variety of feature representations for spam data. We reduce human labeling cost through the use of efficient online active learning variants. We give practical solutions to the one-sided feedback scenario, in which users only give la- beling feedback on messages predicted to be non-spam. We investigate the impact of class label noise on machine learning-based spam filters, showing that previous benchmark evaluations rewarded filters prone to overfitting in real-world settings and proposing several modifications for combating these negative effects. Finally, we investigate the performance of these filtering methods on the more challenging task of abuse filtering in blog comments.
    [Show full text]
  • 2 Uncovering Social Network Sybils in the Wild
    Uncovering Social Network Sybils in the Wild ZHI YANG, Peking University 2 CHRISTO WILSON,UniversityofCalifornia,SantaBarbara XIAO WANG, Peking University TINGTING GAO,RenrenInc. BEN Y. ZHAO,UniversityofCalifornia,SantaBarbara YAFEI DAI , Peking University Sybil accounts are fake identities created to unfairly increase the power or resources of a single malicious user. Researchers have long known about the existence of Sybil accounts in online communities such as file-sharing systems, but they have not been able to perform large-scale measurements to detect them or measure their activities. In this article, we describe our efforts to detect, characterize, and understand Sybil account activity in the Renren Online Social Network (OSN). We use ground truth provided by Renren Inc. to build measurement-based Sybil detectors and deploy them on Renren to detect more than 100,000 Sybil accounts. Using our full dataset of 650,000 Sybils, we examine several aspects of Sybil behavior. First, we study their link creation behavior and find that contrary to prior conjecture, Sybils in OSNs do not form tight-knit communities. Next, we examine the fine-grained behaviors of Sybils on Renren using clickstream data. Third, we investigate behind-the-scenes collusion between large groups of Sybils. Our results reveal that Sybils with no explicit social ties still act in concert to launch attacks. Finally, we investigate enhanced techniques to identify stealthy Sybils. In summary, our study advances the understanding of Sybil behavior on OSNs and shows that Sybils can effectively avoid existing community-based Sybil detectors. We hope that our results will foster new research on Sybil detection that is based on novel types of Sybil features.
    [Show full text]
  • The SAGE Handbook of Web History by Niels Brügger, Ian
    38 Spam Finn Brunton THE COUNTERHISTORY OF THE WEB what the rules are, and who’s in charge. And the first conversation, over and over again: This chapter builds on a larger argument what exactly is ‘spam?’. Briefly looking at about the history of the Internet, and makes how this question got answered will bring us the case that this argument has something to the Web and what made it different. useful to say about the Web; and, likewise, Before the Web, before the formalization that the Web has something useful to say of the Internet, before Minitel and Prestel about the argument, expressing an aspect of and America Online, there were graduate stu- what is distinctive about the Web as a tech- dents in basements, typing on terminals that nology. The larger argument is this: spam connected to remote machines somewhere provides another history of the Internet, a out in the night (the night because comput- shadow history. In fact, following the history ers, of course, were for big, expensive, labor- of ‘spam’, in all its different meanings and intensive projects during the day – if you, a across different networks and platforms student, could get an account for access at all (ARPANET and Usenet, the Internet, email, it was probably for the 3 a.m. slot). Students the Web, user-generated content, comments, wrote programs, created games, traded mes- search engines, texting, and so on), lets us sages, and played pranks and tricks on each tell the history of the Internet itself entirely other. Being nerds of the sort that would stay through what its architects and inhabitants up overnight to get a few hours of computer sought to exclude.
    [Show full text]
  • Social Software: Fun and Games, Or Business Tools?
    Social software: fun and games, or business tools? Wendy A. Warr Wendy Warr & Associates Abstract. This is the era of social networking, collective intelligence, participation, collaborative creation, and bor- derless distribution. Every day we are bombarded with more publicity about collaborative environments, news feeds, blogs, wikis, podcasting, webcasting, folksonomies, social bookmarking, social citations, collab- orative filtering, recommender systems, media sharing, massive multiplayer online games, virtual worlds, and mash-ups. This sort of anarchic environment appeals to the digital natives, but which of these so-called ‘Web 2.0’ technologies are going to have a real business impact? This paper addresses the impact that issues such as quality control, security, privacy and bandwidth may have on the implementation of social net- working in hide-bound, large organizations. Keywords: blogs; digital natives; folksonomies; internet; podcasts; second life; social bookmarking; social networking; social software; virtual worlds; Web 2.0; wikis 1. Introduction1 Fifty years ago information was stored on punch cards. SDI services appeared about 10 years later and databases were available online from about 1978. In 1988 PCs were in common use and by 1998 the web was being used as a business tool. The web of the 1990s might be thought of as ‘Web 1.0’, for now in 2008 there is much hype about Web 2.0, but what does that mean? Web 2.0 is an umbrella term for a number of new internet services that are not necessarily closely related. Indeed, some people feel that Web 2.0 is not a valid overall title for these technologies. A reductionist view is that of a read–write web and lots of people using it.
    [Show full text]
  • D2.4 Weblog Spider Prototype and Associated Methodology
    SEVENTH FRAMEWORK PROGRAMME FP7-ICT-2009-6 BlogForever Grant agreement no.: 269963 BlogForever: D2.4 Weblog spider prototype and associated methodology Editor: M. Rynning Revision: First version Dissemination Level: Public Author(s): M. Rynning, V. Banos, K. Stepanyan, M. Joy, M. Gulliksen Due date of deliverable: 30th November 2011 Actual submission date: 30th November 2011 Start date of project: 01 March 2011 Duration: 30 months Lead Beneficiary name: CyberWatcher Abstract: This document presents potential solutions and technologies for monitoring, capturing and extracting data from weblogs. Additionally, the selected weblog spider prototype and associated methodologies are analysed. D2.2 Report: Weblog Spider Prototype and Associated Methodology 30 November 2011 Project co-funded by the European Commission within the Seventh Framework Programme (2007-2013) The BlogForever Consortium consists of: Aristotle University of Thessaloniki (AUTH) Greece European Organization for Nuclear Research (CERN) Switzerland University of Glasgow (UG) UK The University of Warwick (UW) UK University of London (UL) UK Technische Universitat Berlin (TUB) Germany Cyberwatcher Norway SRDC Yazilim Arastrirma ve Gelistrirme ve Danismanlik Ticaret Limited Sirketi (SRDC) Turkey Tero Ltd (Tero) Greece Mokono GMBH Germany Phaistos SA (Phaistos) Greece Altec Software Development S.A. (Altec) Greece BlogForever Consortium Page 2 of 71 D2.2 Report: Weblog Spider Prototype and Associated Methodology 30 November 2011 History Version Date Modification reason Modified
    [Show full text]
  • SPAM in the Blogosphere
    SPAM in the blogosphere Key Findings According to Umbria Inc., • Spam blogs currently comprise between ten to 20 percent of the blogs in the blogosphere growing from two percent in March 2005. For the week of October 24, 2005, Spam blogs comprised 13 percent of the blogs in the blogosphere (2.7 million out of 20.3 million blogs). • Blog search engines, which utilize keyword search exclusively, are rife with Spam blogs. In a limited test of three blog search engines, an average of 44 of the first 100 blog search listings were Spam blogs. • Keyword search approaches don’t appear to effectively filter Spam blogs due to their real- time nature. • Because Umbria uses a data mining approach for blog analysis, Umbria is able to filter out up to 95% of Spam blogs before performing analysis, yielding cleaner data and more accurate results than other blog analysis services. Abstract According to Technorati, in October 2005 there were more than 20 million blogs (or web logs) with over 80,000 new web logs created each day (or one new blog created every second of every day). While Technorati estimates the blogosphere is doubling in size every 5.5 months, estimates of the overall percentage of Spam blogs (or Splogs) contribution to the growth rate vary. Spam blogs – which are fake blogs designed to fool or “spoof” search engines and drive traffic to sites peddling everything from consumer electronics to online gambling to pornography -- are proliferating in the blogosphere at an astronomical rate. Umbria’s analysis of the blogosphere indicates that between 10%-20% of the blogosphere is now comprised of these Spam blogs.
    [Show full text]