DOCSLIB.ORG

Harvesting and Visualizing Twitter Data

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Harvesting and Visualizing Twitter Data Harvesting and Visualizing Twitter Data GIS Workshop Series Spring 2016 Data Services, Mason Libraries Goals “Do you want to explore data generated from social media activities by a topic?” • Introduce Various Tools to collect Twitter Data • Collect geo-tagged Twitter data for a certain topic • CartoDB: Mining tweets for geographic data and Visualizing (a Web Mapping Platform tool—Cloud Based GIS). Social Media Content • Real Time (Live) Streaming • Subject Matter of Interest • Actionable information • Historical Archives • Geospatial data available through SM. Why Twitter Data? • Open Source and Easily Accessible (1 % of Tweet Data to Public with Twitter APIs) • Characteristics of Tweets: 140 characters (text) in length, #hashtags (a specific topic), @, RT, geotagged information • Emerging as Data for Academic Research (http://geosocial.gmu.edu)--spatial & temporal Ways to Collect Geotagged Twitter Data • Collect Tweets without Programming (Nvivo Ncapture/CartoDB) • Collect tweets with Streaming APIs (https://dev.twitter.com/streaming/public ) + some scripting languages (Ruby, Python, JavaScript, PHP) • Scraping with Python and Twitter APIs (In-house library tool) -Capture Live Streaming Data -Extract by a Keyword, a geographic boundary (a specific location), geotagged information & more. -Result in CSV CartoDB Twitter Maps—NEW service • Search Tweets directly and get all geotagged tweets on a specific topic WT using the Twitter APIs or programming • Download tweets from the last 30 days • Perform Geospatial Analysis • Visualize the data on a map/share or publish “One Stop shop via CartoDB” Collecting Data/Visualization • Exercise: CartoDB Twitter Maps (login with your Google or with your own CartoDB account ). See Example) http://www.portland- communications.com/2014/08/indyref-scottish- independence-debate-on-twitter/ • CartoDB Editor --SQL statements for Query (“PostGIS”) • Visualization Style (Simple, Density, Cluster, etc.) CartoDB: Basic Exercises 1) Connect to Twitter 2) Harvest your search 3) Download (various formats including shp format) 4) Edit/Visualize --Create new columns (latitude and longitude) --Create rows/Remove rows --Remove duplicates --Create a subset of the data (based on SQLs) --Add another layer (polygon) and spatial join (points falling in polygons) Simple SQL Statements SELECT some_column(s) FROM some_data-source(s) WHERE some_condition(s); • (the name of the columns you want to select) http://workshops.boundlessgeo.com/postgis- intro/simple_sql.html (see more) Exercise Data • Download Twitter Data for Exercise --Countries.csv --Twitter_messi --Twitter_ronaldo --countries.shp Exercise SQL Statements Example: Update the values in “Ronaldo” column with the number of tweets mentioning his name per country update countries set ronaldo = ( SELECT count(m.cartodb_id) FROM twitter_ronaldo as m, countries as c where st_intersects(m.the_geom,c.the_geom) and countries.the_geom = c.the_geom ); Another SQL Example Example: Updating the “mostplayer” column with either the word “Messi” or “Ronaldo” based on comparing the numbers in “Messi column” and “Ronaldo column”. update countries set mostplayer = case when messi > ronaldo then 'Messi' when ronaldo > messi then 'Ronaldo' else 'N/A' end; Another SQL Example Example: Selecting the number of “Messi” tweets in each country ordered from highest to lowest. (Spatial Query) SELECT c.name, count(m.cartodb_id) FROM twitter_messi as m, countries as c WHERE st_intersects(m.the_geom,c.the_geom) group by c.name order by count desc; See More • Backtweets.com • LibraryTutorial: “Steps to Collect Twitter Data” http://infoguides.gmu.edu/gis/tutorials • PostGIS SQL Statements: http://workshops.boundlessgeo.com/postgis- intro/ • http://www.qsrinternational.com/support/faqs/w hat-is-ncapture • https://www.youtube.com/channel/UCOjxVdT7wK bHKA5PWvFsW3g (Tips for GIS by Ahmad A.) Qs and Contact Joy Suh Geospatial Resources Librarian ([email protected]) Ahmad Aburizaiza GRA for GIS ([email protected]) .
Load more

Recommended publications
  • COMPUTER PROGRAMMER ANALYST Exam No
    BILL DE BLASIO DEPARTMENT OF CITYWIDE ADMINISTRATIVE SERVICES Mayor BUREAU OF EXAMINATIONS LISETTE CAMILO Commissioner NOTICE OF EXAMINATION PROMOTION TO COMPUTER PROGRAMMER ANALYST Exam No. 1537 WHEN TO APPLY: From: April 7, 2021 APPLICATION FEE: $68.00 To: April 27, 2021 If you choose to pay the application fee with a credit/debit/gift card, you will be charged a service fee of 2.00% of the payment amount. This service fee is nonrefundable. YOU ARE RESPONSIBLE FOR READING THIS ENTIRE NOTICE BEFORE YOU SUBMIT YOUR APPLICATION. WHAT THE JOB INVOLVES: Computer Programmer Analysts, under general supervision, with some latitude for independent or unreviewed action or decision, perform work in the design, development, implementation, maintenance, and enhancement of database management systems, operating systems, data communication systems, and/or computer applications, and may supervise work in the development of computer programs. All Computer Programmer Analysts perform related work. (This is a brief description of what you might do in this position and does not include all the duties of this position.) THE SALARY: The current minimum salary is $51,233 per annum. This rate is subject to change. There are two assignment levels within this class of positions. Promotions will generally be made to Assignment Level I. After promotion, employees may be assigned to the higher assignment level at the discretion of the agency. ELIGIBILITY TO TAKE EXAMINATION: This examination is open to each employee of an agency under the jurisdiction of the Commissioner of the Department of Citywide Administrative Services who on the last day of the application period: 1.
    [Show full text]
  • Data Processing and Visualisation Tools
    DATA PROCESSING AND VISUALISATION TOOLS European Public Sector Information Platform Topic Report No. 2013 / 07 DATA PROCESSING AND VISUALISATION TOOLS Author: datos.gob.es Published: August 2013 ePSIplatform Topic Report No. 2013/07, August 2013 1 DATA PROCESSING AND VISUALISATION TOOLS Table of Contents Keywords: ...................................................................................................................................... 4 Abstract/ Executive Summary: ...................................................................................................... 4 1 Introduction ............................................................................................................................ 5 2 Tool features ........................................................................................................................... 6 2.1 Processing tools ............................................................................................................... 6 A. Refinement tools ............................................................................................................... 6 2.1.1 DataWrangler ............................................................................................................ 6 2.1.2 Google Refine ............................................................................................................ 7 B. Conversion tools ................................................................................................................ 8 2.1.3 Mr. Data Converter...................................................................................................
    [Show full text]
  • I Work for Natural Resources Canada in the Canada Centre for Mapping
    I work for Natural Resources Canada in the Canada Centre for Mapping and Earth Observation, where as a technologist and developer, I have been supporting the development of geo-standards, spatial data infrastructure, or “SDI”, and open spatial data for about 10 years. 1 Today, I’m going to talk about the community, concepts and technology of the Maps for HTML Community Group. The objective of the Maps for HTML initiative is straightforward: to extend HTML to include Web map semantics and behaviour, such as users have come to expect of Web maps. 2 Before getting in to the technology discussions, I think it’s really important to back up and take stock of the situation facing mapping professionals today. 3 Paul Ramsey is a leader in the open source geospatial software development community who currently works for the CartoDB consumer web mapping platform. In a recent presentation to a meeting of Canadian government mapping executives, Paul told us that government mapping programs were no longer relevant. In fairness, Paul did say sorry for having to say that. You know, sometimes it is hard to hear the truth, and I would have to say that Paul wasn’t completely wrong, so what I really want to say in response to Paul is ‘thank you’. 4 Thank you for the opportunity to talk about a subject that has been in the back of my mind not just since I began promoting standards for geospatial information and Spatial Data Infrastructure, and open spatial data, but since the first day I did ‘View Source’ on an HTML page containing a Web map and did not see anything that could possibly produce a map.
    [Show full text]
  • Review of Web Mapping: Eras, Trends and Directions
    International Journal of Geo-Information Review Review of Web Mapping: Eras, Trends and Directions Bert Veenendaal 1,*, Maria Antonia Brovelli 2 ID and Songnian Li 3 ID 1 Department of Spatial Sciences, Curtin University, GPO Box U1987, Perth 6845, Australia 2 Department of Civil and Environmental Engineering (DICA), Politecnico di Milano, P.zza Leonardo da Vinci 32, 20133 Milan, Italy; [email protected] 3 Department of Civil Engineering, Ryerson University, 350 Victoria Street, Toronto, ON M5B 2K3, Canada; [email protected] * Correspondence: [email protected]; Tel.: +618-9266-7701 Received: 28 July 2017; Accepted: 16 October 2017; Published: 21 October 2017 Abstract: Web mapping and the use of geospatial information online have evolved rapidly over the past few decades. Almost everyone in the world uses mapping information, whether or not one realizes it. Almost every mobile phone now has location services and every event and object on the earth has a location. The use of this geospatial location data has expanded rapidly, thanks to the development of the Internet. Huge volumes of geospatial data are available and daily being captured online, and are used in web applications and maps for viewing, analysis, modeling and simulation. This paper reviews the developments of web mapping from the first static online map images to the current highly interactive, multi-sourced web mapping services that have been increasingly moved to cloud computing platforms. The whole environment of web mapping captures the integration and interaction between three components found online, namely, geospatial information, people and functionality. In this paper, the trends and interactions among these components are identified and reviewed in relation to the technology developments.
    [Show full text]
  • Aplicación Web Para El Inventario De Presiones En Ríos Con Cartodb”
    UNIVERSITAT OBERTA DE CATALUNYA Ingeniería Técnica en Informática de Gestión (ITIG) APLICACIÓN WEB PARA EL INVENTARIO DE PRESIONES EN RÍOS CON CARTODB Alumno/a: Lourdes Sánchez Redondo Dirigido por: Víctor Velarde Gutiérrez Co-dirigido por: Antoni Pérez Navarro CURSO 2012-13 (Septiembre /Enero) TFC de Sistemas de Información Geográfica Lourdes Sánchez Redondo 2012/2013 ÍNDICE DE CONTENIDOS ÍNDICE DE TABLAS ..................................................................................................... 3 ÍNDICE DE ILUSTRACIONES .................................................................................... 3 RESUMEN....................................................................................................................... 5 1. INTRODUCCIÓN ................................................................................................. 6 2. DEFINICIÓN Y OBJETIVO DEL PROYECTO PROPUESTO....................... 6 2.1. DESCRIPCIÓN DEL PROBLEMA ............................................................................. 7 2.2. DESCRIPCIÓN DEL PROYECTO .............................................................................. 8 2.3. RESUMEN DE LOS OBJETIVOS ............................................................................... 8 2.4. ENTREGABLES ....................................................................................................... 8 3. PLANIFICACIÓN DEL PROYECTO ................................................................. 9 3.1. MARCO TEMPORAL .............................................................................................
    [Show full text]
  • 2.1.3 Twitter and GIS Data
    A Thesis Submitted for the Degree of PhD at the University of Warwick Permanent WRAP URL: http://wrap.warwick.ac.uk/130202 Copyright and reuse: This thesis is made available online and is protected by original copyright. Please scroll down to view the document itself. Please refer to the repository record for this item for information to help you to cite it. Our policy information is available from the repository home page. For more information, please contact the WRAP Team at: [email protected] warwick.ac.uk/lib-publications Exploring Happiness Indicators In Cities and Industrial Sectors Using Twitter and Urban GIS Data by Neha Gupta Thesis Submitted to the University of Warwick in partial fulfilment of the requirements for admission to the degree of Doctor of Philosophy Supervised by: Prof. Stephen Jarvis and Dr. Weisi Guo Department of Computer Science September 2018 Contents List of Tables iv List of Figures v Acknowledgments viii Declarations x Abstract xii Acronyms xiv Chapter 1 Introduction 1 1.1 Motivation - Social Media Data to Study Sentiment . .1 1.2 Summary of Research Questions . .3 1.3 Thesis Contribution . .4 1.4 Conclusion and Thesis Outline . .6 Chapter 2 Literature Review 7 2.1 Overview of Social Media Analytics . .7 2.1.1 Application In Businesses . .8 2.1.2 Application In Government . .9 2.1.3 Twitter and GIS data . 10 2.2 Twitter Sentiment and Urban Spatial Analysis . 14 2.2.1 Sentiment Classification - Lexicon Based . 16 2.2.2 Sentiment Classification - Machine Learning Methods . 17 2.2.3 Analysing Spatial Patterns of Tweets .
    [Show full text]
  • Web Developer
    Web Developer Position Open Now San Francisco, CA Seattle, WA Have a passion for developing great web applications, and love the idea of doing that for groups working to protect the environment, achieve social justice and improve public health? We’re looking for you! GreenInfo Network is an innovative nonprofit technology consultant specializing in mapping and related information, with a strong national reputation earned over 18 years of service to hundreds of nonprofits and public agencies. We offer an exciting, creative, and friendly environment where you can use your talent to make real difference in the world, while also advancing your career. Our staff of 10 works on 20-30 projects at a time, using a range of tools that include GIS, modeling, web applications, and communication products. Our wide range of projects (mostly in the U.S.) includes: conservation and environmental mapping, demographic assessments, social justice analysis, online map applications, database development, and more. GreenInfo web mapping products are known for elegant design and cartography and intuitive user interfaces, all developed within nonprofit budgets. Visit greeninfo.org to learn more about us, and browse examples of our interactive projects. THE POSITION We are looking for a Web Developer with a passion for programming and using open source tools to create interactive maps and complex data visualization. Your key responsibility will be to implement designs of websites and web mapping applications. We want you to have the ability to create modern web applications with visual clarity and usability, while distilling and integrating complex data sets. You’ll work alongside our existing staff members who provide robust development of the website content, structure, and functionality.
    [Show full text]
  • DC GIS Strategic Plan Update 2016-2021 Page I
    Geospatial Strategic Plan for the District of Columbia 2016 - 2021 Final March 31, 2016 Submitted by: www.AppGeo.com 24 School Street, Suite 500, Boston, MA 02108 Tel: 617-447-2400 Fax: 617-259-1688 TABLE OF CONTENTS 1 Executive Summary 1-1 2 Background 2-1 2.1 Introduction 2-1 2.2 Relevance of DC GIS 2-2 2.3 DC GIS Program Background 2-2 3 Current Status & Context 3-7 3.1 A National Perspective 3-7 3.2 District Perspective 3-8 3.3 The Impact of the Cloud 3-8 3.4 Data and Analytics 3-9 3.5 The Open Data Policy 3-9 3.6 Progress 3-10 3.7 Existing Infrastructure 3-12 3.8 Strengths, Weaknesses, Opportunities, Threats 3-23 4 Mission & Goals 4-30 4.1 Mission Statement 4-30 4.2 Long-term Programmatic Goals 4-30 4.3 Short-term Success Factors for Each Goal 4-30 5 Requirements 5-33 5.1 Data Requirements 5-33 5.2 Technological Infrastructure Requirements 5-35 DC GIS Strategic Plan Update 2016-2021 Page i 5.3 Organizational/Governance Requirements 5-36 5.4 Data Governance Requirements 5-40 6 High Level Implementation Strategy 6-41 6.1 Phasing 6-41 6.2 Program Budget Plan 6-41 6.3 Marketing the Program 6-42 6.4 Measuring Success 6-43 7 Appendices 7-47 7.1 Strategic Planning Methodology 7-47 7.2 Glossary of Acronyms 7-47 7.3 GIS Steering Committee Bylaws 7-49 7.4 Mayor’s Order 2002-27 7-52 7.5 Survey Summary 7-55 7.6 Document History 7-75 7.7 Acknowledgements 7-75 DC GIS Strategic Plan Update 2016-2021 Page ii 1 EXECUTIVE SUMMARY This Strategic Plan provides DC GIS, under the coordination of the Office of the Chief Technology Officer (CTO), with strategic guidance and program direction for the next five years.
    [Show full text]
  • Spatial Opendata Technical Information Note 03/2018 May 2018
    Spatial Opendata Technical Information Note 03/2018 May 2018 Contents Executive summary ................................................................................................................................. 2 Open data ............................................................................................................................................... 2 Geo-spatial .............................................................................................................................................. 2 Uses of open-data ................................................................................................................................... 2 Data types ............................................................................................................................................... 3 Projections .............................................................................................................................................. 5 Licences ................................................................................................................................................... 5 Software .................................................................................................................................................. 6 Data sources ............................................................................................................................................ 7 References & further reading 8 Page 1| Spatial Opendata | LI Technical Information Note
    [Show full text]
  • Cartodb Platform Documentation Documentation Release 1.0.0
    CartoDB Platform Documentation Documentation Release 1.0.0 CartoDB, Inc. Nov 20, 2020 Contents 1 CARTO introduction 3 1.1 What can I do with CARTO?......................................4 2 Components 7 2.1 PostgreSQL................................................7 2.2 PostGIS..................................................7 2.3 Redis...................................................8 2.4 CARTO PostgreSQL extension.....................................8 2.5 CARTO Builder.............................................8 2.6 Maps API.................................................9 2.7 SQL API.................................................9 3 Installation 11 3.1 System requirements........................................... 11 3.2 PostgreSQL................................................ 12 3.3 GIS dependencies............................................ 13 3.4 PostGIS.................................................. 13 3.5 Redis................................................... 13 3.6 Node.js.................................................. 14 3.7 SQL API................................................. 14 3.8 MAPS API................................................ 15 3.9 Ruby................................................... 15 3.10 Builder.................................................. 15 4 Running CARTO 19 4.1 First run, setting up an user....................................... 19 4.2 Running all the processes........................................ 19 5 Configuration 21 5.1 Basemaps................................................
    [Show full text]
  • CARTODB: VISUALIZE SPATIAL DATA on the WEB Thematic Mapping of Enschede Socio-Economic Data
    CARTODB: VISUALIZE SPATIAL DATA ON THE WEB Thematic Mapping of Enschede socio-economic data Barend Köbben Version 2.1 [Enschede] November 7, 2014 Contents 1 The CartoDB web application 1 2Mappingsocio-economicdatafortheEnschedeneigh- bourhoods 1 3 Using the CartoDB web interface and wizards 2 3.1 Creating a data table .................. 2 3.2 Creating a map ..................... 2 3.3 Changing the map visualization ............ 3 3.4 Changing the info window contents .......... 4 3.5 Adding layers ...................... 4 3.6 Publishing your visualizations ............. 4 ©ITC—University of Twente, Faculty of Geo–Information Science and Earth Observation. This document may be freely reproduced for educational use. It may not be edited or translated without the consent of the copyright holder. Key points This is an exercise description for a hands-on workshop to experiment with web-based technologies to visualise spatial data. You will use CartoDB, an interactive web site, to visualize your spatial data on the web. 1TheCartoDBwebapplication CartoDB is an interactive web site that you can use to visualize your spatial data on the web. On their website (http://cartodb.com) they describe it as follows: “CartoDB is a product of Vizzuality, a data visualization consulting company, and was first created to meet the companyńs own internal needs to map large and complex geospatial data sets. While using the platform for various internal projects we realized its full usefulness and potential and decided to make it open source”. On the site there is also a small intro movie: http://vimeo.com/ vizzuality/introducing-cartodb CartoDB is free to use for testing and small projects (using up to 5 data tables and 5Mb of data storage), but you will need to sign up for it: Task 1 : Go to the website and click the “Sign up now” button.
    [Show full text]
  • Carl V. Lewis
    Carl V. Lewis CONTACT 266 Washington Ave [email protected] D9 http://carlvlewis.net, Brooklyn, NY 11205 http://linkedin.com/in/carlvlewis (912) 816-7007 OVERVIEW A post-platform journalist, web developer, design thinker, product manager, interaction designer, digital media educator and data visualization developer with more than 7 years of experience in data journalism, news graphics, newsroom interactive storytelling, curriculum design, digital publishing, homepage editing, product management, media entrepreneurship, digital strategy/social media management, mobile front-end design/development, interaction programming and Wordpress/Drupal content management development. EDUCATION M.S. in Digital Media, Honors. Aug. 2011 — May 2012 Columbia University Graduate School of Journalism -Specializations in data visualization, interactive design, audience engagement and consumer reporting. -Awarded multiple honors designations, including only honor's status in entire cohort for Data Journalism focus and Journalism Business Models focus. -Elected as online editor and officer for Society of Professional Journalists chapter, the school's official student governing body. -Built new SPJ site from scratch using Twitter Bootstrap as a framework built on top of the popular WordPress CMS. -Grew SPJ digital audience reach among class from 150 per week to 1,482 unique impressions per week. -Served as online editor for Society for Hispanic Journalists, building organization's website and leading initial marketing efforts. (Recommendations provided from Susan E. McGregor, Ava Seave and Daniel Medina; Arlene Morgan, Sree Srenevasian and Rebecca Castillo may also be contacted as references). B.A. in Journalism, B.A. in Southern Studies, Aug. 2007 — May 2011 Honors. Mercer University -Graduated summa cum laude with a 3.94 GPA, University Honors, Presidential Service Fellowship and the annual Jimmy Carter Award for Community Outreach and Public Service, given to a graduating senior who exemplifies the school's long-held tradition of servant leadership and community activism by Pres.
    [Show full text]