A Geospatial Infrastructure to Collect, Evaluate, and Distribute Volunteered Geographic Information for Disaster Management

University of Calgary PRISM: University of Calgary's Digital Repository

Graduate Studies The Vault: Electronic Theses and Dissertations

2016 A Geospatial Infrastructure to Collect, Evaluate, and Distribute Volunteered Geographic Information for Disaster Management

Poorazizi, Mohammad Ebrahim

Poorazizi, M. E. (2016). A Geospatial Infrastructure to Collect, Evaluate, and Distribute Volunteered Geographic Information for Disaster Management (Unpublished doctoral thesis). University of Calgary, Calgary, AB. doi:10.11575/PRISM/24747 http://hdl.handle.net/11023/3369 doctoral thesis

University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY

A Geospatial Infrastructure to Collect, Evaluate, and Distribute Volunteered Geographic

Information for Disaster Management

Mohammad Ebrahim Poorazizi

A THESIS

SUBMITTED TO THE FACULTY OF GRADUATE STUDIES

IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE

DEGREE OF DOCTOR OF PHILOSOPHY

GRADUATE PROGRAM IN GEOMATICS ENGINEERING

CALGARY, ALBERTA

SEPTEMBER, 2016

Recent disasters, such as the 2010 Haiti earthquake, have drawn attention to the potential role of citizens as active information producers. By using location-aware devices such as smartphones to collect geographic information in the form of geo-tagged text, photos, or videos, and sharing this information through online social media, such as Twitter, citizens create Volunteered

Geographic Information (VGI). This thesis presents a framework for the effective use of VGI in disaster management platforms. The proposed framework consists of four components: (i) a VGI brokering module, to provide a standard service interface to retrieve VGI from multiple social media streams, (ii) a VGI quality control component, to evaluate spatiotemporal relevance and credibility of VGI, (iii) a VGI publisher module, which uses a service-based delivery mechanism to disseminate VGI, and (iv) a VGI discovery component, which acts like a yellow-pages service to find, browse, and query available VGI datasets. A set of quality metrics specifically designed for VGI evaluation is introduced. This research also presents a prototype implementation including an evaluation with social media data collected during Typhoon Hagupit (i.e., Typhoon

Ruby), which hit the Philippines during December 2014. The evaluation results suggest that the proposed framework provides a promising solution towards an effective use of VGI in disaster management platforms. Utilization of the proposed quality metrics on the collected VGI database

– with multiple social media stream contributions – will allow disaster response teams to make informed decisions that could save lives, meet basic humanitarian needs earlier, and perhaps limit environmental and economic damage.

ii Acknowledgements

To Mahsa, my wife and partner in life. Thank you for your endless love and support; for standing by me throughout the worst of times; and for making this journey possible. Words cannot express my gratitude.

To Dr. Andrew Hunter, thank you for your invaluable guidance, constant support, and trust.

Your insight kept me focused and grounded.

To Dr. Stefan Steiniger, who was always willing to help. Thank you for your never-ending support and encouragement.

To Dr. Derek Lichti, thank you for all your help and support when I needed most. It has been greatly appreciated.

To Dr. Steve Liang, Alberta Innovates — Technology Futures and Alberta Innovation &

Advanced Education, the Department of Geomatics Engineering, and the University of Calgary, your financial support has allowed me to pursue this work. I am very grateful for your assistance.

To my committee members, Drs. Steve Liang, Xin Wang, Daniel Jacobson, and Mohsen

Kalantari, thank you for your invaluable feedback and support. Your contribution has improved this work in many ways.

To Ali Naghshbandieh, Sina Kiaei, Zakaria Ahamdi Bonakdar, Morteza Azimi, Javad

Behrouzfar, Hamid Ebrahimi, Zahra Lari, Ali Mousavi, Ehsan Mohammadi, Babak

Amjadiparvar, and Sina Taghvakish, thank you for all the great moments we have spent together.

To Mom and Dad, thank you for your unconditional love. Without your encouragement and continual support pursuing my dreams would be impossible. To my sisters, Farideh and Atefeh, thank you for always being available when I needed your support.

iii Table of Contents

Abstract ...... ii Acknowledgements ...... iii Table of Contents ...... iv List of Figures and Illustrations ...... ix List of Abbreviations ...... xi

CHAPTER ONE: INTRODUCTION ...... 1 1.1 Problem Statement ...... 1 1.2 Motivation ...... 5 1.2.1 Data Management ...... 12 1.2.2 Data Quality ...... 14 1.2.3 Information Sharing ...... 17 1.2.4 System Architecture ...... 18 1.3 Research Questions ...... 19 1.4 Research Objectives ...... 20 1.5 Research Methodology ...... 21 1.6 Assumptions, Scope, and Limitations ...... 24 1.7 Main Contributions ...... 26 1.8 Thesis Structure ...... 27

CHAPTER TWO: THEORETICAL BACKGROUND AND STATE OF THE ART .....28 2.1 Disaster Management Platforms ...... 28 2.1.1 Service-Oriented Architecture ...... 31 2.1.1.1 Granularity and Wrapping Resources ...... 32 2.1.1.2 Integration and Reusability ...... 34 2.1.1.3 Standard Interfaces and Interoperability ...... 37 2.1.2 OGC Standards ...... 39 2.1.2.1 Web Processing Service (WPS) ...... 39 2.1.2.2 Sensor Web Enablement (SWE) ...... 40 2.1.2.3 Catalog Service for the Web (CSW) ...... 41 2.1.3 Volunteered Geographic Information ...... 42 2.2 Three Generations of Disaster Management Platforms: An Introduction ...... 44 2.2.1 SDI-based Platforms ...... 46 2.2.2 VGI-based Platforms ...... 47 2.2.3 Hybrid Platforms ...... 48 2.3 Three Generations of Disaster Management Platforms: A Comparison ...... 50 2.3.1 Enabling infrastructures ...... 54 2.3.2 Platform Functions ...... 56 2.3.2.1 Data Discovery and Access ...... 56 2.3.2.2 Data Quality ...... 57 2.3.2.3 Data Distribution ...... 60 2.4 Chapter Summary ...... 61

CHAPTER THREE: METHODS ...... 63 3.1 Definition of Disaster Management Platform Attributes to Guide the Development63

iv 3.2 Conceptual Design ...... 66 3.2.1 VGI Broker ...... 67 3.2.2 VGI QC ...... 68 3.2.3 VGI Publisher ...... 69 3.2.4 VGI Discovery ...... 69 3.3 Quality Evaluation Metrics ...... 69 3.3.1 Positional Nearness ...... 72 3.3.2 Temporal Nearness ...... 75 3.3.3 Semantic Similarity ...... 75 3.3.4 Cross-referencing ...... 76 3.3.5 Credibility ...... 77 3.3.6 Quality Score ...... 78 3.4 Geoparsing ...... 80 3.4.1 Place Name Extraction ...... 81 3.4.2 Geocoding ...... 83 3.5 Content Analysis for Semantic Similarity Assessment ...... 84 3.5.1 Text Classification ...... 85 3.5.2 Similarity Assessment ...... 88 3.6 Technical Design ...... 91 3.6.1 Service Design Principles ...... 92 3.6.1.1 Creating Services ...... 92 3.6.1.2 Connecting Services ...... 93 3.6.1.3 Publishing Services ...... 94 3.6.2 Service Development Strategy ...... 97 3.6.2.1 VGI Broker Services ...... 99 3.6.2.2 VGI QC Services ...... 100 3.6.2.3 VGI Publisher Services ...... 103 3.6.2.4 VGI Discovery Services ...... 105 3.7 Implementation ...... 106 3.8 Chapter Summary ...... 108

CHAPTER FOUR: EXPERIMENTS AND RESULTS ...... 109 4.1 The VGI Framework in Action ...... 109 4.2 Place Name Extractor Selection ...... 112 4.3 Geocoder Selection ...... 116 4.4 Text Classifier Selection ...... 123 4.5 Evaluation of VGI Quality Score Calculation and Metrics ...... 127 4.5.1 How VGI QC Works ...... 128 4.5.2 Characteristics of Collected Contributions for Positional Nearness ...... 131 4.5.3 Characteristics of Collected Contributions for Temporal Nearness ...... 134 4.5.4 Characteristics of Collected Contributions for Semantic Similarity ...... 136 4.5.5 Characteristics of Collected Contributions for Cross-referencing ...... 138 4.5.6 Characteristics of Collected Contributions for Credibility ...... 140 4.5.7 Overall Scores per Social Media Stream ...... 141 4.6 Chapter Summary ...... 142

CHAPTER FIVE: EVALUATION AND DISCUSSION ...... 143

v 5.1 Evaluation of Quality Control Metrics ...... 143 5.1.1 Positional Nearness ...... 143 5.1.2 Temporal Nearness ...... 146 5.1.3 Semantic Similarity ...... 147 5.1.4 Cross-referencing ...... 149 5.1.5 Credibility ...... 149 5.1.6 Total QS ...... 151 5.2 Evaluation of the Technical Framework ...... 160 5.3 Chapter Summary ...... 166

CHAPTER SIX: CONCLUSIONS ...... 167 6.1 Revisiting the Research Questions ...... 167 6.1.1 Research Question 1 ...... 167 6.1.2 Research Question 2 ...... 168 6.1.3 Research Question 3 ...... 169 6.1.4 Research Question 4 ...... 170 6.1.5 Research Question 5 ...... 171 6.2 Summary of Contributions ...... 172 6.3 Future Work ...... 173

REFERENCES ...... 178

vi List of Tables

Table 2.1. Service types in terms of granularity...... 33

Table 2.2. Potential of VGI as a resource for SDIs (see Genovese & Roche [125] for a complete analysis)...... 49

Table 2.3. A comparison of web-based disaster management platforms...... 53

Table 2.4. Benefits of using SOA in disaster management platforms...... 55

Table 2.5. Research works that examined VGI quality assessment based on accuracy and credibility metrics...... 59

Table 3.1. VGI quality evaluation methods and the metrics used in this research (adapted from Goodchild & Li [142] and Haklay [158])...... 71

Table 3.2. Suggested credibility factors for well-known social media platforms...... 78

Table 3.3. A summary of quality evaluation metrics and their characteristics...... 80

Table 3.4. Categories used for classifying tweets...... 86

Table 3.5. A summary of challenges and proposed solutions for integrating OGC and W3C services...... 96

Table 3.6. Mapping the VGI data elements to O&M ...... 104

Table 3.7. The query parameters supported by VGI Publisher to search and filter VGI ...... 105

Table 4.1. Python APIs used to develop the VGI Broker, search parameters, and the number of collected contributions from each social media stream...... 110

Table 4.2. A brief description of place name extraction methods ...... 112

Table 4.3. Key features of the NER systems compared in this study (adapted from Derczynski et al. [168])...... 114

Table 4.4. A contingency table or confusion matrix used to evaluate the accuracy of classification tasks [211]...... 116

Table 4.5. The performance of NER engines (best scores are underlined)...... 116

Table 4.6. Characteristics of online geocoding services used this study (adapted from Chow et al. [173]) ...... 117

Table 4.7. Performance of the geocoders...... 118

Table 4.8. Performance of the geocoders when using the country code...... 119

vii Table 4.9. The geocoded place names that have a distance value of greater than 100 km...... 121

Table 4.10. A brief description of machine learning algorithms used in this experiment...... 123

Table 4.11. The performance of machine learning algorithms on the original and resampled datasets (high precision, recall, and F1 scores are underlined)...... 126

Table 4.12. The performance of ensemble methods (high scores are underlined)...... 127

Table 4.13. The extracted words from the tweet and maximum word similarity scores...... 129

Table 4.14. The credibility factors for the given tweet and their maximum values among all contributions...... 130

Table 4.15. The number of geo-referenced contributions from each social media stream...... 132

Table 4.16. Counts of Positional Nearness Score (PNS) values for each media stream...... 134

Table 4.17. Counts of Temporal Nearness Score (TNS) values for each dataset...... 136

Table 4.18. Counts of Semantic Similarity Score (SSS) values of each dataset...... 138

Table 4.19. Counts of Cross-Referencing Score (CRS) values of each dataset...... 140

Table 4.20. Counts of Credibility Score (CS) values of each dataset...... 141

Table 4.21. Counts of total Quality Score (QS) values of each dataset...... 142

Table 5.1. Correlations between the selected credibility factors and the credibility scores calculated for each social media stream...... 151

Table 5.2. Correlations between the quality scores...... 152

Table 5.3. Correlations between the quality scores when removing location-dependent metrics (i.e., PNS and CRS)...... 153

Table 5.4. A sample of Twitter contributions with the highest total quality score...... 154

Table 5.5. A sample of Google Plus contributions with the highest total quality score...... 155

Table 5.6. A sample of Flickr contributions with the highest total quality score...... 157

Table 5.7. A sample of Instagram contributions with the highest total quality score...... 158

Table 5.8. Quality attributes requirements and business goals addressed by the VGI framework...... 164

viii List of Figures and Illustrations

Figure 1.1. The first tweet posted by the City of Calgary regarding the flooding event...... 7

Figure 1.2. Although the City of Calgary posted updates on social media, people were encouraged to follow the latest news on its own website...... 8

Figure 1.3. A screenshot of the news blog operated by the City of Calgary to broadcast crisis information. The issues regarding the accessibility of Calgary.ca is highlighted...... 9

Figure 1.4. Situation updates published by the City of Calgary on Twitter, which is actually a link to an image map of LRT (Light Rail Transit) and BRT (Bus Rapid Transit) service detours...... 10

Figure 1.5. A screenshot of the crisis map mashup developed for Calgary’s flood 2013...... 11

Figure 1.6. An overview of the proposed framework...... 23

Figure 2.1: The generic framework for disaster management (adapted from Yang et al. [62]). .. 29

Figure 2.2. Service chaining design principles...... 35

Figure 2.3. Architecture and workflow in a centralized (left) and cascaded (right) control patterns...... 36

Figure 2.4. Three generations of disaster management platforms...... 45

Figure 3.1. VGI framework business goals...... 65

Figure 3.2. A conceptual workflow for discovery, assessment, and dissemination of VGI...... 67

Figure 3.3. An imaginary location of a tweet that falls into the (imaginary) spatial extents of Twitter, Flickr, and Google Plus...... 76

Figure 3.4. Geoparsing procedure workflow...... 81

Figure 3.5. Content filtering and semantic similarity evaluation workflow...... 85

Figure 3.6. Hierarchical semantic knowledge base (adopted from Li et al. [190])...... 90

Figure 3.7. A reference architecture incorporating proposed VGI framework components (highlighted elements are the focus of this work)...... 98

Figure 3.8. The VGI Broker as WPS instances...... 100

Figure 3.9. The VGI QC processing workflow. The numbers on the arrows represent the order of execution within the workflow...... 102

ix Figure 4.1. The geocoded place names by Google and Nominatim. Lines connect the points with the same place name...... 120

Figure 4.2. Longitude (right) and latitude (left) of geocoded points returned by Google and Nominatim...... 122

Figure 4.3. A tweet sent during Typhoon Ruby in the Philippines during December 2014...... 128

Figure 4.4. Spatial distribution of all social media contributions for Typhoon Ruby (each circle represents a certain number of social media contributions)...... 133

Figure 4.5. Spatial distribution of social media contributions with estimated location in the Philippines...... 133

Figure 4.6. Temporal dynamics of users’ contributions on social media...... 135

Figure 4.7. Temporal dynamics of all social media contributions and relevant contributions. .. 137

Figure 4.8. Spatial distribution of relevant contributions (see Figure 4.4 for spatial distribution of all contributions)...... 138

Figure 4.9. The spatial extent of contributions from Flickr, Google Plus, Instagram, and Twitter (the numbers represent the amount of social media contributions). The extents are not rectangular because of the projection system used to visualize data globally...... 139

Figure 5.1. Part of the results of standard compliance tests performed to ensure interoperability of system components...... 163

x List of Abbreviations

Abbreviation Definition

ANOVA Analysis of Variance

API Application Programming Interface

CPS Calgary Police Service

CRED Centre for Research on the Epidemiology of Disasters

CRF Conditional Random Field

CRS Cross-referencing Score

CS Credibility Score

CSW Catalog Service for the Web

DaaS Data as a Service

FN False Negative

FOSS Free and Open Source Software

FP False Positive

GIS Geographic Information System

GPS Global Positioning System

KVP Key-Value-Pair

LBS Location-based Service

LCA Lowest Common Ancestor

MBR Minimum Bounding Rectangle

MEP Municipal Emergency Plan

ML Machine Learning

NER Named-entity Recognition

xi NLP Natural Language Processing

O&M Observations and Measurements

OGC Open Geospatial Consortium

OSM OpenStreetMap

OSMCP OpenStreetMap Collaborative Project

PNS Positional Nearness Score

POI Point of Interest

POS Part-of-speech

QC Quality Control

QoS Quality of Service

REST REpresentational State Transfer

SA Selective Availability

SaaS Software as a Service

SAS Sensor Alert Service

SDI Spatial Data Infrastructure

SensorML Sensor Model Language

SOA Service-Oriented Architecture

SOAP Standard Object Access Protocol

SOS Sensor Observation Service

SPS Sensor Planning Service

SSS Semantic Similarity Score

SVM Support Vector Machine

SWE Sensor Web Enablement

xii TF-IDF Term Frequency - Inverse Document Frequency

TN True Negative

TNS Temporal Nearness Score

Total QS Total Quality Score

TP True Positive

UN United Nations

VGI Volunteered Geographic Information

W3C World Wide Web Consortium

WCS Web Coverage Service

WFS Web Feature Service

WMS Web Map Service

WNS Web Notification Service

WPS Web Processing Service

WSDL Web Service Description Language

XSD XML Schema Definition

xiii

CHAPTER ONE: INTRODUCTION

1.1 Problem Statement

Natural disasters pose a great threat to people’s lives and their properties and have a negative impact on economies [1]. According to the World Disaster Report 2015, between 2005 and 2014, more than 760,000 people were killed by natural disasters, and another 1.9 billion affected, while the cost of disasters was over 1,500 trillion USD [2]. Scientists consider climate change, population growth, urbanization and industrialization, and disorderly occupation as driving factors that promote the increase of natural disasters [3,4].

Disaster management aims to limit the impact of disasters through preventing loss of human life, delivering assistance to the victims, and achieving rapid and effective recovery [5]. The disaster management cycle consists of four successive stages [6]:

1. Prevention – the “normal” state of the social system before the impact, where long-term

activities are accomplished to eliminate or diminish disaster risks.

2. Preparedness – where necessary actions for developing emergency response plans are

undertaken to reduce effects of disasters when they occur.

3. Response – where disaster relief operations (e.g., medical aid, search and rescue, etc.) are

performed to help those who affected by the disaster.

4. Recovery – where rehabilitation and readjustment actions are conducted to restore the

social system to its normal condition.

In the literature, prevention and preparedness stages are classified as pre-disaster phases while response and recovery are categorized as post-disaster phases (see [7,8,3,9]). Jongman et al. [10]

argue that the effectiveness of the (disaster management) process, in both pre- and post-disaster phases, is dependent upon accurate and timely information regarding the geographical location and impacts of the ongoing event [10]. Such information helps disaster responders (e.g., humanitarian organizations, disaster relief agencies, or local actors) answer the following questions:

1. When did the disaster happen?

2. Where are the affected areas?

3. What are the impacts of the disaster?

The answers to these questions provide spatial (where), temporal (when), and thematic (what) information about an event. The insights gained from analysis of such information can be of great value to decision-makers in different phases of a disaster.

Traditionally, information reaches disaster responders through a “top-down” paradigm: public authorities inform the public on potential risks or events through common media outlets such as radio and television [11]. This strategy cannot efficiently function in a disaster event where situations are always changing1, demanding a change in response, which requires up-to-date information [4]. This can be partially explained due to the fact that in conventional (data management) systems public authorities are responsible for collecting, managing, updating, and publishing information, which are in line with a set of established rules and procedures to ensure reliability and credibility [12]. This may lead to a delay of many hours or even several days until

1 The rate of change can be different depending on the type of the disaster; it can be quick for “rapid onset disasters” (e.g., typhoons) and slow for “slow onset disasters” (e.g., droughts). 2

the disaster responders receive the information, and, consequently, impair the disaster management efforts [4,10].

For example, satellite imagery has been one of the most prominent (traditional) data sources used to support disaster response operations from mapping of burnt areas for forest fires to infrastructure damage assessment for earthquakes [9]. However, Jongman et al. [10] discuss that although satellite imagery can be used for any phase of disaster management, it is especially useful in pre-disaster phases, since they are less time-sensitive compared to post-disaster phases.

This is because it usually takes 24 hours or more to process the satellite images, extract the actionable information, and send them to the disaster responders on the ground [10].

In the past decade, advances in information and communication technologies have opened up new avenues for collaboration and participation. Today, many citizens use mobile devices, with

Internet access, to share text messages, images, and videos through social media services such as

Facebook, Twitter, and Instagram. Thus, the citizens’ role has transcended from (information) recipient to (information) producer, or as coined by Budhathoki et al. [13], “produser”. The information that is voluntary produced by citizens is also known as crowdsourced information or user-generated content. If user-generated content conveys spatial information2, it is called volunteered geographic information (VGI) [15]. This thesis refers to all types of user-generated content, with or without spatial reference, as VGI, since it is an established term in GIScience.

2 The spatial information in user-generated content can be explicit or implicit. It is explicit if it contains information about the characteristics of a place (e.g., attached geographical coordinates). If information is not about a place but still can be geocoded, it is implicit (e.g., place names) [14]. 3

OpenStreetMap3 (OSM), a free editable map of the world, is one of the most prominent and well- known examples of VGI on the Internet [16]. The 2010 Haiti earthquake was one of the first major events to use OSM in disaster response operations. Shortly after the event, over 600 volunteers from all over the world worked together to map the affected areas and contributed over one million edits for Haiti on OSM [17]. Social media has also played a key role in distributing information during disasters. People, both affected citizens and those outside the impact zone, and media outlets have used social media to collate and share crisis information during wildfires [18], earthquakes [19], floods [20], winter storms [21], heavy snowfall [22], and typhoons [23]. This has created a well-established pattern: “a disaster strikes, and the crisis data collection begins” [24]. Meier [17] reviews this pattern in action in different crises worldwide and investigates the impact of social media technologies on disaster management – supporting emergency response in real-time. Hence, public authorities and traditional emergency response organizations have recently begun to recognize the added value of information available via social media during disasters. For example, the Red Cross organizations in the United States,

Australia, and the Philippines have started to incorporate social media and VGI into their disaster management processes [10,25]. Summarizing the positive aspects of using VGI in disaster management mentioned in the literature [10,11,17,25–28] the following benefits can be identified:

• Timely information exchange,

3 http://www.openstreetmap.org/ 4

• Improved communication,

• Increased community engagement, and

• Acquisition of diverse local information in (near) real-time without limitations of other

technologies (e.g., satellite imagery).

However, when one considers VGI as an information source for disaster management, then there are also challenges to its use. In particular, social media streams contain large amounts of irrelevant messages such as rumors [29], advertisements [30], or even misinformation [31]. So, one major challenge to using VGI is how to process VGI streams and deliver credible and relevant information to disaster responders and citizens. A recent survey by the US

Congressional Research Service reveals that administrative cost to monitor multiple social media streams and verify the quality of incoming information in a timely manner is a significant barrier to adopting VGI during disasters [32]. Hence, there is a need to develop automatic computational methods for collecting, filtering, and validation of VGI [11,25,31,33,34]. Therefore, the research presented here concentrates on developing and testing a framework that aims to convert VGI collected from social media services into a reliable source of information and, thus, facilitates incorporation of VGI into disaster management platforms. The following sections outline the motivation of this research and subsequently define the research questions, objectives, and scope and limitations. The chapter ends with an outline of the thesis’ structure.

1.2 Motivation

In June 2013, Albertans witnessed a catastrophic flooding event, which ranked fifth among the top-ten global economic losses in 2013, and resulted in the loss of four lives, the displacement of

thousands of citizens from their homes, the disruption of hundreds of businesses, and significant damage to both private and public property, and public infrastructure [35].

As the disaster unfolded, people flooded social media with information, photos, and breaking news regarding the ongoing event. Besides citizens, Calgary’s official emergency responders, such as the Calgary Police Service (CPS) and the City of Calgary, also used social media (e.g.,

Twitter, Facebook, etc.) to broadcast safety-critical information and situation updates. The

Calgary Herald reports that the City of Calgary’s Twitter account4 published around 1,500 messages during the event that were reposted (i.e., retweeted) about 20,000 times [36]. The role of social media was described as “vital” in the crisis and the city’ performance on using social media’s potential to enhance crisis communication was recognized.

“It’s about two-way communications [and] being responsive and extremely

informative in real-time. Social media is the reason for these new rules. Not

just that but these are the expectations of one’s market in a crisis and if you

don’t meet those expectations you’ll find yourself in a multitude of different

crises within the crisis… I was so impressed with Calgary. Not just their use

of Twitter but their use of hashtags. Some cities use Twitter but they don’t use

hashtags and then nobody can track and monitor (information) and easily find

and funnel.” – Melissa Agnes, crisis communication expert [36].

4 @cityofcalgary 6

The City of Calgary posted the first tweet about the intense rainfall and extreme increase in the flow of the rivers in the area on June 20, 2013 at 8:15 am (see Figure 1.1). Accordingly, the

Municipal Emergency Plan (MEP) was activated and, shortly after that, mandatory evacuation orders were in effect in parts of the region.

Figure 1.1. The first tweet posted by the City of Calgary regarding the flooding event. Despite the use of social media for crisis communication, the City of Calgary website5 was the main of source of information regarding the post-disaster incidents (e.g., power outages, road and bridge closures, etc.) and the progress of disaster response operations (e.g., deployment of emergency crews, evacuation, etc.), which hit 1.1 million views throughout the event [36]. The social media users were even asked to monitor the City of Calgary website for latest news (see

Figure 1.2).

5 http://www.calgary.ca/ 7

Figure 1.2. Although the City of Calgary posted updates on social media, people were encouraged to follow the latest news on its own website. However, due to high traffic volumes, the City of Calgary website was shut down on the first day of the event and redirected to another domain (Figure 1.3). This indicates that, although the City of Calgary was well prepared to deal with the disaster on the ground6, its technical infrastructure suffered from the lack of scalability at that point. Another issue that we, the University of

Calgary GIScience research group, observed was the lack of a crisis map in the first days of the event, which could improve understanding of crisis information. Using a crisis map is beneficial as:

• People can quickly locate all the information, in a specific geographic area, at a glance

without having to read large volumes of information;

• Relief organizations can coordinate resource distribution and make better decisions; and

6 In September 2013, the City of Calgary awarded its first official recognition for its response to the flooding [36]. 8

• Emergency responders and citizens can find the physical constraints caused by the

disaster, such as non-accessible river bridges, and take an informed action [37].

In response to this need, on June 21, 2013 we started to develop an interactive flood mapping mashup to help people find and use critical emergency information, such as road and bridge closures, power outage areas, emergency aid locations, etc. [38].

Figure 1.3. A screenshot of the news blog operated by the City of Calgary to broadcast crisis information. The issues regarding the accessibility of Calgary.ca is highlighted. However, since critical crisis information were being published through emergency responders’ official websites and their social media accounts, the main challenge was how to collect, extract, and visualize useful information on a regular basis. A part of the issue was that public authorities

(e.g., the City of Calgary) did not provide crisis information in common spatial data formats 9

(e.g., Shapefile, KML, or GeoJSON). Instead, they published the information in plain text and

PDFs or image formats – see Figure 1.4 as an example. Hence, we had to manually collect data, extract the information, and convert it to GeoJSON (i.e., a web-friendly format) to be able to display situation updates on the map. Finally, the crisis map mashup went live on June 23, 2013

(Figure 1.5). The use of free and open source software (FOSS) tools and open data allowed us to rapidly deploy the website and provide citizens access to a wide range of crisis data via a single map.

Figure 1.4. Situation updates published by the City of Calgary on Twitter, which is actually a link to an image map of LRT (Light Rail Transit) and BRT (Bus Rapid Transit) service detours.

Figure 1.5. A screenshot of the crisis map mashup developed for Calgary’s flood 2013. On July 4, 2013, the City of Calgary lifted the local state of emergency [39], however, we kept updating the information until July 8, 20137. Since our sole focus was to build and deploy the website as soon as possible, we overlooked the use of an online tracking tool (e.g., Google

Analytics) to analyze the number of visits. However, we received a positive feedback from the

7 We updated the website 18 times within 19 days of operation, while most of the updates occurred during the first four days of the event. 11

community, through our blog8 and word of mouth, and our crisis map mashup was recognized in the GeoAlberta mApp Gallery Competition 20139.

The development of the crisis map mashup for Calgary’s flood 2013 was a real-world experience. It paved the way for me, as a researcher, to analyze the information needs during a disaster and develop my understanding of characteristics of a “bottom-up” disaster management platform10. Based on the lessons learned during our work on the Calgary flood map, the following sections highlight the current challenges in development of bottom-up disaster management platforms, which frame the motivation of this research.

1.2.1 Data Management

During the Calgary flooding event, crisis information was published through various platforms including the official emergency responders’ websites (e.g. the City of Calgary’s website) and social media (e.g., Twitter, Facebook, and YouTube). However, as mentioned before, the City of

Calgary’s website was the main source of latest updates, which was down for sometime during the first day of the event. So how could safety-critical and time-sensitive information be distributed? An answer may be social media, since social media services played a major role as popular information propagation platforms, which enable public authorities and citizens to share and discover (near) real-time situation updates on a scalable infrastructure.

8 http://planyourplace.ca/pypblog/?p=612 9 http://planyourplace.ca/pypblog/?p=634 10 It refers to a platform that enables incorporation of VGI as a (complimentary) source of information. 12

From a technical perspective, this means, compared to a website hosted on a private server, social media platforms are supported by enterprise (cloud-based) service providers and, therefore, the probability of service downtime is lower accordingly. From a public participation standpoint, The Huffington Post reports that people’s engagement on social media was remarkable during the 2013 flood in Calgary [40]. People acted as both collector and distributor of information: some people posted what they witnessed on the ground in the form of text, photo, and video while others broadcast the information that is already on social media to reach more people – the latter group is also called “information brokers” [41]. Studies show that affected communities create similar information flow patterns when a disaster occurs regardless of its geographical region, which has been proved to be effective in increasing situational awareness11 in different types of disasters worldwide (see [25,27,31] for a review).

VGI can be more effective if it is used as a complementary source of information to authoritative data in disaster management platforms [27,31,33]. To create such platforms, some data management challenges need to be addressed. Because of the dynamic nature of disasters, these systems should be able to regularly, if not continuously, collect and share up-to-date crisis- related VGI from different social media streams. Usually social media services provide access to

VGI through different application programming interfaces (APIs) and data encodings. This can make data access and retrieval difficult for client applications, such as data visualization and

11 Situational awareness “describes the idealized state of understanding what is happening in an event with many actors and other moving parts, especially with respect to the needs of command and control operations.” [20]. Simply, it is the knowledge of what is happening in the affected communities during a disaster. 13

response management tools within a disaster management platform, because they need to implement different APIs and understand specific data encodings to be able to access and retrieve data [42]. In addition, the information flows on social media at a high rate, especially when large disasters occur, which makes monitoring multiple social media a challenging task

[27]. For example, 16,000 tweets per minute is one of the largest peak of Twitter usages during a natural disaster [25]. Hence, there is a need for an effective data collection mechanism to enable discovery of and access to VGI from various social media platforms.

1.2.2 Data Quality

Nowadays, there is an explosion of social media activity: users upload more than 400 hours of video to YouTube every minute; people post more than 80 million photos daily on Instagram; and during a weekend of the Coachella festival12, around 3.8 million tweets were published on

Twitter about it [43]. With this huge volume of data, YouTube, Instagram, Twitter and all content-delivery companies work everyday to improve content-recommender algorithms to give their users the “best stuff”. This highlights the fact that a large portion of content on social media

(or generally on the Internet) is “bad”, or to be fair, is not of interest [43]. Studies show that a

(large) fraction of VGI broadcast on social media is not relevant to a particular disaster, which is

12 It is an annual music and arts festival held in California, US. 14

also referred to as “noise”13 [30,31,45,46]. Therefore, there is a need to analyze VGI streams and discard noise to be able to use VGI as an information source in disaster management platforms.

The Calgary’s flood 2013 was not an exception. An article reports that people published around

857,000 tweets containing the most popular hashtags such as #YYCFlood14 during the event

[40]. Not surprisingly, there were some irrelevant messages, or not informative messages to be precise, among them, for example:

“#downtown #Calgary at 8am on a #thursday it looks like a ghost town!

#yycflood #abflood”

Although the tweet is about the event, it does not have (actionable) crisis information and, therefore, does not contribute to situational awareness. The same interpretation applies to the following message, which had around 2.1 million impressions15 on Twitter [40]:

“Sending our thoughts to the @NHLFlames and everyone in Alberta affected

by the #yycflood”

13 It refers to a VGI that does not contain disaster-related information such as caution and advice, infrastructure damage, medical aid, volunteering services, donations, etc. This type of VGI is also considered as “not informative” [44]. 14 According to the report, #Calgary, #ABFlood, #YYC, #YYCHelps, #CalgaryFlood, and #YYCFlood were the most popular hashtags used to share crisis information during the event, while #YYCFlood was used in more than 38% of the tweets [40]. 15 In simple words, it is a metric that shows how many views a tweet has had since it has been published. 15

The message shows support and sympathy, but it is still not informative. Hence, during the flooding event, we had to “manually” monitor Twitter, verify the tweets, and extract relevant information to visualize them on the map.

Related to the usefulness of a media stream message is the process of VGI verification, which concerns the quality or credibility of VGI. The need for VGI verification has been described as one of the main obstacles in using VGI in disaster management, especially by official emergency responders [11,25,27,28,31,33,46–48]. However, the quality assessment of VGI is a challenging task because of three main reasons. First, considering the amount of information that flows on social media, it is seems ambitious for emergency responders to evaluate each individual message on social media in real-time to find useful information [44]. To tackle this issue requires development of automatic computational methods that implement quality control processes.

Second, since VGI is created and disseminated in a different way compared to authoritative spatial data, it demands a different approach to quality assessment [49]. In this respect, a different set of quality metrics than those for authoritative data has to be defined, to enable effective evaluation of VGI and thus improve its utility. Finally, social media messages are brief

(e.g., 140 characters for tweets) and informal and, therefore, applying the methods that are used to process structured, long texts such as news articles to deal with VGI may lead to poor and misleading results [50]. This poses a significant challenge to VGI processing and it is necessary to develop specialized content analysis techniques [25].

1.2.3 Information Sharing

As mentioned earlier, official emergency responders published crisis information in different formats, from plain text to maps in image formats, during the 2013 Calgary flooding event.

Hence, data had to be converted into a web-friendly format – in our case GeoJSON – to be able to include it on a web-map. Despite the issues regarding the manual conversion process (e.g., time-consuming), our approach limits the (re)use of data for other purposes than web-based visualization, for example for mobile notification services. This is because we did not use a standard data publication mechanism for information sharing. This issue is also a major challenge in current disaster management systems that often distribute data in heterogeneous and not-standardized formats, which hinders integration of data into other (existing) disaster management platforms and, in turn, forms “data islands” and “system islands” [51].

Following standard data transfer protocols facilitates communication, data exchange, and using received data, thus, ensures interoperability [52]. In GIScience, interoperability is achieved using open geospatial standards such as Open Geospatial Consortium´s16 (OGC) web service specifications and data encodings. Such standards are used to develop interoperable solutions in spatial data infrastructures (SDIs) to provide the capability of discovering, accessing, and sharing spatial data among a wide range of stakeholders [53]. Although information sharing mechanisms in SDIs were initially designed for authoritative spatial data, they can be adopted to overcome the issues of VGI heterogeneity in disaster management platforms [42,54]. Further investigations

16 http://www.opengeospatial.org/ 17

are required in this respect to develop standards-based, interoperable information sharing mechanisms for VGI that can facilitate integration of VGI and authoritative data in disaster management platforms.

1.2.4 System Architecture

Although public authorities tried to make full use of social media to improve communication with citizens during the Calgary flooding event in 2013, they referred people to check out the

City of Calgary’ website for latest updates – see Figure 1.2 as an example. Moreover, not all people are users of social media and, therefore, the City of Calgary’ website was likely the only information resource for those who were not following news on social media. Consequently, high demand overloaded the City of Calgary’s website and resulted in its unavailability for a period of time during the event. This highlights the scalability17 problem of the technical infrastructure that the City of Calgary used to develop for disaster response. From a data management viewpoint, Ding et al. [51] report that current disaster management systems used by public authorities worldwide are mainly closed, static, and centralized desktop solutions in which

(dynamic) information integration and resource sharing and reuse are still unsatisfied. As a result, such systems fail to incorporate information from other data sources such as authoritative data from other organizations (e.g., national mapping agencies) or VGI from social media and,

17 It refers to the capability of a system to handle a growing amount of workload, or its potential to be enlarged in order to accommodate that growth [55].

therefore, cannot efficiently respond to the frequent demand for real-time information in a disaster.

To address these issues, a technical architecture should be designed and employed to build the disaster management system components in such a way that they are reusable, interoperable, and scalable. As a result, such an architecture allows the VGI system to be easily customized, developed, and deployed. In this context, it is essential to investigate architectural design patterns and principles to draw guidelines for the development of effective disaster management platforms.

1.3 Research Questions

Given the challenges identified above, the research questions that guide this study are:

1. What specific information values does VGI present that can complement authoritative

spatial information in a disaster? What are the strengths and weakness of VGI in this

context?

2. What would be a typical (geo)processing workflow that allows collecting, validating, and

sharing VGI in an automated way? What specific design requirements should be

considered to develop the system components?

3. Considering a massive amount of VGI from multiple social media platforms, what

mechanism can facilitate discovery of and access to VGI from different social media

platforms?

4. What strategy can be used to tackle the verification, i.e., credibility, of VGI from social

media streams? What quality control metrics are suitable to evaluate VGI credibility?

What will the quality evaluation tell us about VGI?

5. How can quality-assessed VGI be shared in a way that increases availability of VGI and,

eventually, facilitates incorporation of VGI and authoritative data in disaster management

platforms?

1.4 Research Objectives

This research primarily aims to develop a framework for the effective use of VGI in disaster management platforms. Thereby it specifically focuses on the development of a robust workflow to facilitate collecting, validating, and sharing VGI, published on social media, that is relevant, credible, and actionable for emergency response. This, in turn, will support the use of VGI, as a complementary source of information to authoritative data, in disaster management platforms and perhaps spatial data infrastructures.

To achieve this general objective and address the research questions, the following research objectives are addressed in this study:

1. To propose a new approach to facilitate discovery of VGI from multiple social media

platforms.

2. To introduce a new quality control model to enable filtering and validation of VGI.

Specifically, to identify a set of quality metrics that are suitable for VGI quality

assessment and to design a (geo)processing workflow to automate the quality control

procedure.

3. To propose an effective mechanism to facilitate sharing of VGI. The main focus in

particular is to enable incorporation of VGI with authoritative data in disaster

management platforms.

4. To present a reference architecture that allows the components to be plugged into a

coherent system for disaster management that is easy-to-access, easy-to-use, and easy-to-

maintain. Particularly, the focus is to identify architectural patterns and design principles

in creating (geo)processing workflows and to present guidelines for development of

effective disaster management platforms.

5. To evaluate the proposed framework. In this context, the suitability of VGI quality

metrics and the usefulness of quality assessment results are evaluated and strengths and

weaknesses of VGI for use in disaster response are discussed. Finally, an evaluation of

the effectiveness and efficiency of the technical architecture in terms of qualitative (e.g.,

interoperability) and quantitative (e.g., performance) metrics is to be performed.

1.5 Research Methodology

This research is developed under design-science research paradigm, which involves creation and evaluation of artifacts to solve organizational problems within information systems research [56].

Accordingly, the goal of this study is to design and develop a geospatial infrastructure that provides collection, evaluation, and distribution of VGI for disaster management purposes.

Hevner et al. [56] outlines seven principles that should be addressed for effective design-science research to be accomplished. Adapting those principles, the following guidelines represent the research methodology undertaken in this study:

1. Design-science research must produce a viable artifact. In this research, the proposed

infrastructure is the artifact designed, with a focus on disaster management.

2. The objective of design-science research is to develop technology-based solutions to

important and relevant problems. This research proposes a novel solution to address the

problems identified in Section 1.2 and to answer the research questions outlined in

Section 1.3. Figure 1.6 gives an overview of how the proposed solution functions:

“components” are building blocks the framework that perform the “processes” and

generate the “outputs”. First, VGI Broker finds and retrieves (new) VGI from different

social media services. Afterwards, using a new (VGI-specific) quality control model,

VGI QC (Quality Control) analyzes the new data to evaluate the quality of its content.

Then, using VGI Publisher, the quality-assessed VGI is encoded as a standard-complaint

data model and published using interoperable data services – the output of this process is

called “standard VGI” in Figure 1.6. Finally, VGI Discovery generates metadata

descriptions for the “standard VGI” and publishes them in open catalogues to facilitate

discovery of VGI.

3. The utility, quality, and efficacy of a design artifact must be rigorously demonstrated via

well-executed evaluation methods. This research employs qualitative and quantitative

evaluation methods to measure the applicability and quality of the platform and discuss

its weakness and strengths with respect to the case study.

4. Effective design-science research must provide clear and verifiable contributions in the

areas of the design artifact. This thesis contributes to the research by producing design

foundations and an artifact that responds to the research problems, questions, and

objectives, and by testing to determine the usefulness of the solution.

Figure 1.6. An overview of the proposed framework. 5. Design-science research relies upon the application of rigorous methods in both the

construction and evaluation of the designed artifact. This research employs open

standards/specifications and follows a SOA (Service-Oriented Architecture)-based design

strategy to design the system components. Free and open source software (FOSS) tools

are used to implement a proof-of-concept and evaluate the utility of the proposed

framework using social media data collected during Typhoon Hagupit (i.e., Typhoon

Ruby), which hit the Philippines during December 2014.

6. Design-science is essentially a search process to discover an effective solution to a

problem. This research employs and extends best practices to develop an innovative 23

solution for the use of VGI in disaster response. The whole process is conducted as a

search process; it begins with inquiry into the problem in Chapter 1. The search continues

by further investigating the problem space with a review and analysis of the current state

of disaster management platforms and highlighting the research gaps (Chapter 2). This is

followed by the research methodology presented in Chapter 3 to tackle the research

objectives. The search resumes with the design and construction of a prototype, described

in Chapter 4. Since this research deals with the first iterations in the design-test cycle, the

solution (i.e., artifact) will likely evolve beyond its current state in future iterations.

7. Design-science research must be presented effectively to a range of appropriate

audiences. This goal is achieved through this thesis, journal publications, conference

presentations, blogs, and social media.

1.6 Assumptions, Scope, and Limitations

This research assumes that development of a VGI framework is a promising solution to support disaster management activities, specifically for natural disasters18. This assumption is in accordance with the background of the researcher and his disciplinary bias as a Geomatics

Engineer with specialization in GIScience. The researcher’s background will also influence different steps in the development cycle of the proposed framework, from design and implementation to deployment and evaluation, thereby introducing bias. To address concerns of bias, this study has considered existing knowledge and findings of previous works, and a mixed

18 However, the developed methods can be adopted for other types of disasters such as technological hazards (e.g., industrial pollution, nuclear radiation, chemical spills, etc.). 24

methods study design that integrates both quantitative and qualitative analysis to address the limitations of a particular method and the researcher, in relation to system evaluation [57].

This research does not strive to detect or predict disaster events based on social media activity.

Rather, the focus of this study is to respond to information needs during the post-disaster phases of the disaster cycle – i.e., disaster response, and in a lesser extend to recovery. In this context, this research combines and applies recent concepts and tools from different scientific fields, including GIScience, Machine Learning (ML), and Natural Language Processing (NLP). It aims to contribute to existing knowledge by developing a framework for the effective use of VGI in disaster response.

This study does not take into account the network structure and topology of social media platforms to evaluate communities of users and their relationships. Instead, it aims to incorporate users’ reactions on social media, as a representation of relationships between users, into the evaluation process – for example how many times a social media post has been viewed, liked, or reposted.

There are some limitations to using the proposed platform as a tool for disaster management.

First, people living in remote and less developed areas, especially in developing countries, may have limited access to the Internet and social networking platforms such as Twitter. The reasons can be

• The lack of efficient underlying technological infrastructure such as a reliable electricity

grid, cabling, or networking, etc., which are often destroyed or interrupted by disasters,

• A large number of people living in remote areas tend to have less access to information

technology, and

• Internet censorship, which may be imposed by governments, private organizations, or a

group of people to control what can be accessed, published, or viewed on the Internet

[57–59].

Apart from technological limitations, there is a need for political support to enable integration of bottom-up disaster management platforms with emergency communication strategies. In this context, it is necessary for government organizations to adopt legal and policy frameworks outlined in SDIs, so that they can be extended to provide user-driven capacity building, i.e., integration and management of VGI with authoritative data. Addressing the limitations mentioned above was not part of this thesis and the issues are hopefully addressed elsewhere.

1.7 Main Contributions

The most notable scientific contributions of this research are:

• A novel conceptual framework was proposed that provides the essential components for

collection, verification, and dissemination of VGI to enable its incorporation in disaster

management platforms.

• An effective data collection mechanism was introduced that facilitates VGI search and

retrieval from different social media platforms.

• A new data verification model was proposed that aims to evaluate the quality of VGI in

terms of spatial, temporal, semantic, and credibility factors.

• An effective data dissemination mechanism was introduced that promotes VGI sharing

and reuse, and enables discovery of and access to VGI based on standard methods.

• A novel (reference) technical architecture was presented that enables integration of the

proposed framework’s components into a standards-based service-oriented application.

1.8 Thesis Structure

The remainder of this thesis is outlined as follows. Chapter 2 provides an overview of background information and necessary concepts for the use of VGI in disaster response. This is followed by a review and analysis of the current state of disaster management platforms and concludes with an evaluation of existing challenges from a technical perspective.

Chapter 3 describes the proposed methodology for design and development of VGI framework.

It then introduces a set of quality evaluation metrics, outlines system design principles, and presents a service-based architecture including a description of key components and modules.

Chapter 4 presents a prototype implementation and reports the results of an extensive analysis on social media data collected during Typhoon Ruby.

Chapter 5 evaluates the proposed framework and discusses its weaknesses and strengths in terms of technical (qualitative and quantitative) attributes and the set of quality metrics with respect to the case study’s social media data.

Finally, Chapter 6 presents a summary of findings of this work. The chapter ends with conclusions and possible directions for future research.

CHAPTER TWO: THEORETICAL BACKGROUND AND STATE OF THE ART

This chapter provides background information and concepts that are essential to understanding the methods, results, and findings of this research, and reviews the state of the art in disaster management platforms. In this respect, it first introduces a generic framework for development of disaster management platforms and describes its main components. This is followed by a review and analysis of the current state of disaster management platforms in terms of their functionality and enabling technologies. In the end, the chapter evaluates existing research gaps and challenges from a technical perspective.

2.1 Disaster Management Platforms

The Centre for Research on the Epidemiology of Disasters (CRED) defines a disaster as “a situation or event, which overwhelms local capacity, necessitating a request to national or international level for external assistance; an unforeseen and often sudden event that causes great damage, destruction and human suffering” [60]. Disaster management refers to a process of responding to a disaster in order to lessen its impacts. It is a continuous process, consisting of four phases (i.e., prevention, preparedness, response, and recovery), during which emergency responders provide humanitarian aid to those who are affected by the disaster [61]. As discussed earlier, the effectiveness of disaster management operations is dependent on how well emergency responders can respond to “what”, “where”, and “when” questions (see Section 1.2).

To answer these practical questions requires the use of the most recent data to be able to produce sound decision supporting information [62]. This research considers disaster management platforms as a promising solution to address this information need. The building blocks for such

platforms are outlined in a framework, which is referred to as the “generic framework” hereafter, as shown in Figure 2.1.

Figure 2.1: The generic framework for disaster management (adapted from Yang et al. [62]). The framework consists of the following components:

(1) Functions – the functional components within the framework provide discovery, access, analysis, and visualization of data. These include:

• Discovery functions, to access and use available resources (i.e., data and services).

• Quality control functions, which evaluate quality of resources, especially VGI.

• Dynamic data management functions, which provide collecting, filtering, interpretation,

verification, and sharing methods for dynamic data (e.g., VGI).

• Processing functions, which provide methods for the analysis and simulation of data,

integration of existing models, and creation of new resources that contain attributes

useful for disaster management.

• User interface functions, for efficient and informative visualization of data and simulation

results.

(2) Enabling Infrastructure – This component provides technological support to create, develop, and maintain all necessary functions for disaster management platforms. It includes:

• Communication networks and protocols, which provide an environment for access and

dissemination of data such as HTTP, SMS, or social media.

• Data frameworks, which provide spatial, temporal, and thematic datasets for disaster

management including authoritative data, in-situ sensor data, VGI, and metadata.

• SDI (Spatial Data Infrastructure), which provides standards and policies that facilitate

human-machine and machine-machine interactions.

• Computing platforms, which provide computing resources (i.e., software, hardware, and

network) for data- and computing-intensive tasks such as cloud computing platforms.

• System architectures, which provide technical frameworks for creation and development

of disaster management platforms such as service-oriented architecture (SOA) [1,63,64],

mashup architectures [62,65], etc.

(3) Community – the community represents different stakeholders involved in a disaster. These include public authorities, decision-makers, NGOs, and citizens that interact with and utilize the platform.

It should be noted that the goal of this research is not to cover the needs of all target communities during a disaster. Rather, the aim is to develop an infrastructure that provides relevant and credible VGI as valuable input for disaster response operations utilized by some particular members of the community. For example, emergency responders (e.g., fire fighters) can extract road/bridge closures from a credible piece of VGI, provided by the proposed platform, to find accessible routes to affected areas. Therefore, this research focuses on functions and enabling infrastructures in general and does not discuss specific needs of different community members.

However, over the past years, community members have realized various components of the generic framework to build disaster management platforms according to their specific needs.

This has resulted in a wide variety of platforms with different characteristics, which are reviewed in Section 2.2. But before that the following sections explain some of the main terms, models, and components of the generic framework that are central to this research.

2.1.1 Service-Oriented Architecture

A service-oriented architecture (SOA) is an architectural style that supports design and development of applications as standard services [66]. Services are basic computing units, which can be invoked individually to accomplish a single task or be integrated into larger, complex services to create service-oriented applications. In a service-based development approach, it is important to maintain an appropriate balance between multiple criteria (e.g., flexibility,

reusability, and performance) [67]. To achieve this goal and have services tailored to specific application use cases, different architectural design patterns and principles should be considered.

The design principles, discussed next, are grouped into three main categories: service granularity, reusability, and interoperability. The first section (2.1.1.1) highlights the importance of service granularity for creating (individual) services. Section 2.1.1.2 then reviews service chaining architectures and workflow design patterns for integrating (individual) services and building service-oriented applications. Finally, Section 2.1.1.3 presents standard approaches for publishing (individual) services.

2.1.1.1 Granularity and Wrapping Resources

The goal in designing services is finding the right balance between multiple criteria such as flexibility, reusability, and performance, that represent different level of service granularity [67].

Service granularity refers to the scope of a web service in terms of the functionality that is exposed [68]. Fine-grained services provide elementary, non-decomposable operations exposed as single service instances. They are services that generally handle simple forms of data exchange. Therefore, they can be reused in other workflows, or integrated with other service instances to meet new application requirements. In coarse-grained services, a group of operations are wrapped into a single service instance (see Table 2.1). Although the number of service requests to perform a specific task is reduced, it might be more complicated to integrate them with other services, or modify them to meet new application requirements [68].

A coarse-grained service should provide better performance, which has a direct impact on end- users, and has become a best practice for designing business services [69,70]. Conversely, fine-

grained services offer flexibility, and increase service reusability, which are key success factors when designing services in SOA [71]. Therefore, a good practice is to consider the spectrum of granularity levels to find the right balance between service reusability and performance for various use cases [68].

A service design strategy can follow a top-down approach, a bottom-up approach, or utilize both strategies. Given a top-down perspective, services and models decompose into smaller subsystems that enable the identification of relevant processes. The decomposition process is terminated when all application requirements are met. Alternatively, a bottom-up approach focuses on wrapping different software modules or scientific algorithms within a service [72].

The composition process is continuous and iterative, and stops when application requirements are met. The two design approaches are generally used in parallel to develop models and scenarios that are reusable and efficient [73].

Table 2.1. Service types in terms of granularity.

Service type Description Example

Fine-grained Services that provide elementary, Logic: retrieve polygon data from a database non-decomposable operations and calculate its area exposed as single service instances. Service 1: retrieve data

Service 2: calculate area

Coarse-grained Service that provide a group of Logic: retrieve polygon data from a database operations, wrapped into a single and calculate its area service instance. Service 1: retrieve data and calculate area

2.1.1.2 Integration and Reusability

Alameh [74] discussed design patterns for executing a workflow of services19 and proposed three service chaining architectures:

• Transparent pattern (or user-defined service chaining), which requires full user

interaction to: (a) find all required services, (b) create the service chain, and (c) manage

service execution.

• Translucent pattern (or workflow-managed chaining), which requires that the client is

aware of the distinct processes and can manage workflow tasks through invocation of a

mediated module (e.g., a gateway).

• Opaque pattern, which describes an aggregate service where the client invokes a static

pre-configured composite service that manages the process chain. The service acts like a

standard web service in that the client is not aware of individual processing steps and

cannot monitor the execution of the services.

To realize these architectural patterns and create service chains for specific applications, several service chaining design principles should be considered [75], including (1) the workflow control pattern, (2) the data interaction pattern, and (3) the communication pattern (Figure 2.2).

19 It refers to a set of services that perform a task according to a set of procedural rules. 34

Figure 2.2. Service chaining design principles. (1) Workflow Control Pattern – The workflow control pattern is a design strategy related to the management and execution of a service chain. Cascading and centralized chaining methods are two patterns for describing and executing the workflow of service activities [75] – see Figure

2.3. The cascading chaining method describes a workflow in which only the service providing the result is invoked directly and the remaining service invocations (e.g., for data retrieval) are managed by the solicited service. This method has some drawbacks in terms of design and validation of the workflow. It can be difficult to manage data extraction, monitoring of workflow execution states, error handling, and dynamic handling of cases in which the workflow behaviour depends on intermediate data [75]. To address such challenges, centralized chaining can be adopted as an alternative. It describes a workflow of services in which all service invocations are controlled by a central component (e.g., a workflow management module). With this approach, a client application, or a workflow engine, initiates all workflow interaction, and controls management and execution of the workflow.

Figure 2.3. Architecture and workflow in a centralized (left) and cascaded (right) control patterns. From a software maintenance perspective, one of the main drawbacks of the client-controlled approach is that the client needs to be updated should service reference endpoints change, if there are updates to service interfaces, or if a workflow is reconfigured. Therefore, this approach is not suitable for spatial processing applications in which the workflow is generated by dynamic operation selection and service configurations [75]. To overcome such challenges, Friis-

Christensen et al. [75] proposed an engine-controlled approach to manage the execution of a workflow through (i) workflow languages (e.g., BPEL and XLANG) or (ii) management services. In a workflow language-based approach, a workflow execution engine is used to handle the sequence of requests to the services. In a service-based approach, the workflow engine is implemented as a service instance itself. Compared to a client-controlled approach, the engine- controlled approach is more sustainable as it allows the service chain instance to be updated without affecting client applications, and can also be reused in other workflows.

(2) Data Interaction Pattern – A data interaction pattern describes how data are transferred between services involved in a workflow. Two dominant mechanisms exist for data 36

transportation in a workflow: data passing by value, and data passing by reference [75]. The data passing by value approach transfers data as values between each service in a workflow. While data can be compressed, spatial processing workflows are usually data-intensive. Data references can minimize data transfer bottlenecks because the approach requires that a common local data store (to the service(s)) be utilized by all services involved in a workflow so that each data element can be uniquely identifiable by a reference.

(3) Communication Pattern – Two communication patterns, synchronous and asynchronous, are distinguished [64] to realize the message exchange mechanisms between services in the workflow, or between a client and a service chain. However, the synchronous approach is not suited for long-duration (geo)processing tasks and can dramatically increase response times, especially when data and processing services are distributed [64]. Asynchronous communication patterns are a more efficient solution for time-consuming processing operations [75], as the response does not return immediately to the consumer, but can be retrieved during a different communication session once the process has been completed. To create such communication patterns, the service chain should be capable of managing processing task queues, and accommodate a notification mechanism to inform the service consumer when a task has been completed (or has failed to be completed).

2.1.1.3 Standard Interfaces and Interoperability

A set of standard-compatible tools can facilitate discovery, access, processing, and visualization of information [76]. As such, the definition and use of open standards play a key role in the establishment of interoperable services.

Interoperability refers to “the capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units” [77]. In SOA, services are basic computing units, which can be integrated into larger, complex services to create service-oriented applications. A service is a standards-based, loosely-coupled unit composed of a service interface and a service implementation [77]. The clean separation between “what the service offers” (service interface) and “how the service works” (service implementation) ensures interoperability [78].

Web service platforms implemented using SOA principles (also known as W3C services) have resulted in efficient and reliable systems that allow service providers to advertise their capabilities to potential users [1]. SOA is an enabling infrastructure for realizing interoperability through (i) consistent descriptions of service interfaces (WSDL: Web Service Description

Language), (ii) an authoritative message exchange format (SOAP: Simple Object Access

Protocol), and (iii) a standard method for defining the exchange of data (XML and XSD: XML

Schema Definition) [79].

In GIScience, interoperability is achieved by using standard interfaces, standard metadata, and well-known specialized data encodings defined within an SDI framework [80]. Currently, most of the standard services deployed in SDIs use OGC standards [68]. By adopting SOA principles,

OGC standards promote interoperability through a set of specifications that enable spatial data and functionality to be published, accessed, and adapted to wide variety of applications. The next section introduces three OGC standards that provide standard access to data and processing capabilities, following DaaS (Data as a Service) and SaaS (Software as a Service) approaches.

2.1.2 OGC Standards

The Open Geospatial Consortium (OGC) is an international organization that develops open standards for spatial data and services. These OGC standards enable interoperability through a set of specifications that enable discovery of geospatial resources (e.g., CSW: Catalogue Service for the Web [81]), sharing geospatial data (e.g., WFS: Web Feature Service [82] and WCS: Web

Coverage Service [83]), (geo)processing capabilities (e.g., WPS: Web Processing Service [84]), and visualization of spatial information (e.g., WMS: Web Map Service [85]). Since this study employs WPS, the Sensor Web Enablement (SWE) framework [86], and CSW standards, the following sections explain them in more details.

2.1.2.1 Web Processing Service (WPS)

The OGC released version 1.0.0 of the WPS specification in June 2007 [84]. The specification, along with the OGC Web Processing Service Best Practice discussion paper [87], describe a web service interface that defines how a client and server should cooperate during the execution of a spatial analysis, and how results of the process should be presented. Clients can send requests via three core operations using three methods: Key-Value-Pair (KVP) encoding via HTTP GET,

XML via HTTP POST, or a SOAP/WSDL approach. The WPS specification defines three mandatory operations that enable spatial processing on the Internet [84]:

• The GetCapabilities operation, which allows a client to request and receive service

metadata documents that describe the capabilities of a specific server implementation.

• The DescribeProcess operation, which returns detailed information about a process’

requirements, such as input and output parameters, as well as allowable data formats.

• The Execute operation, which invokes a specific process implemented by the WPS, using

the input parameters provided, and returns the results of the service to a client.

2.1.2.2 Sensor Web Enablement (SWE)

Sensor Web Enablement is an OGC initiative to provide an information infrastructure containing standard interfaces and data encodings that enable all types of web-resident sensors to be discoverable, accessible, and controllable via the Web [86]. The SWE architecture covers many aspects of environmental data acquisition and management, with three data models and four service implementation specifications [88]. O&M (Observations and Measurements) and Sensor

Model Language (SensorML) are standard data models. XML schemas are used for encoding sensor observations and describing sensor systems as well as related processes and processing components associated with the measurement and post-measurement transformation of observations [89]. The four service interfaces include Sensor Observation Service (SOS), Sensor

Planning Service (SPS), Sensor Alert Service (SAS), and Web Notification Service (WNS).

They are used for retrieving sensor observations, controlling or tasking sensors, publishing alerts from sensors, registering users, and sending notification messages to them [86,90].

The OGC defines SOS as a web service interface for providing access to observations from sensors and sensor systems in an open and standard way that is consistent for all sensor systems including remote, in-situ, fixed, and mobile sensors [91]. From this viewpoint, SOS is considered an intermediary between a client and sensor observation repository. Clients can send requests through three core operations using KVP encoding via HTTP GET, or XML via HTTP POST, or a SOAP/WSDL approach [91]. GetCapabilities, DescribeSensor, and GetObservation are three

mandatory operations to help users discover and retrieve service and sensor metadata, and observations [91]:

• The GetCapabilities operation allows clients to discover specific service metadata.

• The DescribeSensor operation enables clients to request information about a sensor. A

SensorML document describing the sensor and its capabilities is returned in response to

the DescribeSensor request.

• The GetObservation operation enables clients to request observation data generated by a

sensor or sensor system. It supports a multitude of parameters and filters, which give

clients the ability to query a set of sensors based on time, location, phenomena, features,

and measurement values of the observations. The response from the GetObservation is

encoded using O&M.

The O&M describes the data format for sensor data, which is an abstract model and XML schema encoding for observations and measurements [92]. Within O&M, the main concept is

“observation”, which is defined as an event that occurs at a certain point in time (e.g., 2016-05-

03T14:30:00) and generates an estimated value of an observed phenomenon (e.g., surface water temperature) for a feature of interest (e.g., Johnson Lake). Within the O&M framework, sensor data is encoded using the SWE Common data model to achieve data interoperability [88].

2.1.2.3 Catalog Service for the Web (CSW)

The Catalog Service for the Web provides capabilities for publishing and finding spatial data and service over the Web based upon their metadata [81]. Metadata documents describe the capabilities of a specific server implementation that provides a service. Examples of advertised

information include: access protocols, supported operations, data type, data format, etc., through which clients are able to find, access, and use the service. The CSW publishes metadata documents using different metadata content specifications such as Dublin Core, ISO 19139, or

FGDC. Clients can communicate with the CSW using three methods: KVP via HTTP GET,

XML via HTTP POST, or a SOAP/WSDL approach [81]. The CSW specification defines four mandatory operations that enable discovery of spatial data and services on the Internet [81]:

• The GetCapabilities operation, which allows clients to retrieve service metadata from a

server.

• The DescribeRecord operation, which allows clients to discover elements of the data

model supported by the CSW.

• The GetRecords operation, which allows to look up records using a series of search

parameters.

• The GetRecordById operation, which retrieves a record based on its unique identifier.

2.1.3 Volunteered Geographic Information

In the 2000s, different technological advancement led to start a paradigm shift in production and use of spatial information. First, Global Positioning System (GPS) technology was enabled to the public without “selective availability”20 in 2000 and GPS handheld devices started to become available for reasonable prices and with reasonable positioning accuracy [94]. Second, broadband Internet connections became ubiquitous and the Web technologies evolved from a

20 “Selective Availability (SA) was an intentional degradation of public GPS signals implemented for national security reasons.” [93] 42

one-directional broadcasting media to a two-way information-sharing platform, also known as

Web 2.0 [95]. As a result, between 2004 and 2008, several mapping platforms, such as Google

Maps Maker21, Wikimapia22, and OpenStreetMap (OSM)23, emerged that provided an ecosystem of participation where value (i.e., spatial information) was created through user contributions

[95]. Third, in the late 2000s, the introduction of location-aware mobile devices (i.e., smartphones) made it even easier for ordinary citizens to create online geo-enabled (or geo- tagged) content during their daily lives [96]. Today, people are able to post location-enabled messages on Twitter, share geo-tagged photos on Flickr, and publish videos on Instagram with geographical coordinates attached.

The spatial information produced by citizens through Web 2.0 service such as social media and online mapping tools is called VGI [97]. This new type of spatial information can complement authoritative spatial data collected by governmental agencies or private organizations [98]. For example, researchers [99,100] report that OSM data in urban areas can have very detailed and up-to-date information since OSM contributors mapped details such as street surfaces or sidewalks width and incline – information that is rarely found in authoritative datasets. As outlined in the introduction, VGI has also changed disaster management, in that citizens and public authorities use Web 2.0 technologies for sharing crisis information [27]. Examples include humanitarian aid [101], emergency reporting [102], and crisis mapping [103]. However, as

21 https://www.google.ca/mapmaker 22 http://wikimapia.org/ 23 http://www.openstreetmap.org/ 43

mentioned in Section 1.2, the credibility of VGI has been stressed as a major issue for utilization in disaster management [104]. Several strategies have been proposed to overcome the alleged lack of credibility of VGI, which will be discussed in Section 2.3.2.2. The following section reviews prominent disaster management platforms from a technical perspective. This overview is not exhaustive; rather the intent is to give an overview of the technological progress in building such platforms until today.

2.2 Three Generations of Disaster Management Platforms: An Introduction

Information and communication infrastructure components are one of the building blocks of disaster management systems. Many systems have been developed over the past decade – moving from traditional telephone-, radio-, and television-based systems towards modern web- based platforms [65]. In the early 2000s, disasters, such as the terrorist attacks of 9/11 and

Hurricane Katrina in the US, demonstrated that traditional disaster management systems are limited in their ability to meet community-wide information sharing and communication needs of all stakeholders, e.g., jurisdictional authorities, emergency respondents, and citizens [105]. For example, during 9/11, incompatible technology was a significant problem in crisis communication where emergency responders from different organizations were unable to communicate due to incompatible radio systems [106]. In the response to Hurricane Katrina, a lack of systems capable of tracking and sharing information was the main issue, which significantly hampered the response operation due to a lack of information on the ground [107].

Thus, compared to the Internet, the amount of information published by telephone, radio, or television is limited, and the systems focus on one-direction information flow. This limits the

level of interaction possible [57]. Hence, there is a need for platforms that can provide many-to- many communication and offer effective information sharing mechanisms that facilitate rather than impede disaster management. As a response to this demand, different approaches to design and development of disaster management platforms have been investigated over the past decade.

We can group these works into three generations based on the data framework(s) they use (see

Figure 2.4):

• SDI-based platforms, which represent the platforms that are designed based on SDI

principles. These platforms use only authoritative (spatial) data to support disaster

response operations.

• VGI-based platforms, which represent the disaster management platforms that place VGI

at the center of the information system.

• Hybrid platforms, which represent the platforms that aim to integrate VGI with

authoritative (spatial) data and, thus, realize incorporation of VGI into SDI.

Figure 2.4. Three generations of disaster management platforms. 45

The following sections describe the characteristics of each generation in more details.

2.2.1 SDI-based Platforms

The first generation covers systems that were designed based on SDI principles [53]. A spatial data infrastructure [108] presents a promising framework to facilitate and coordinate the exchange and sharing of spatial data in disaster management systems, resulting in improved quality of decision-making and increased efficiency of disaster management activities [109–

112]. The effectiveness of SDI-based disaster management platforms has been evaluated in evacuation scenarios after a bomb threat [113], wildfire risk assessment [114], and a flood warning system [115], among others.

Although SDIs facilitate data sharing and management, SDI implementation follows a “top- down” approach that does not consider that non-institutional users might contribute data in a participatory fashion [116]. This leads to a provide-consume paradigm, where only official data providers such as national mapping or environmental agencies are permitted to collect, deploy, and maintain resources [54]. Moreover, official data providers have strict update and release cycles that may hinder access to timely information, especially during a disaster. For example, after the Haiti earthquake in 2010, the GIS (Geographic Information System) databases that were available lacked critical up-to-date post-earthquake information that complicated rescue and recovery efforts in the first days following the earthquake24 [117]. SDI-based platforms also tend

24 Although high-quality satellite images of post-earthquake Haiti were collected and made freely available within 24 hours of the disaster by commercial geospatial content providers such as DigitalGlobe [117], there was still a need to process the images to extract useful information (e.g., tracing roads and buildings) and perform required analyses (e.g., damage assessment analysis). 46

to use a complex deployment mechanism, which can impede citizens’ participation during data collection and resource deployment [54]. However, during the Haiti event, a volunteer community was able to quickly build an information infrastructure that permitted collaborative data collection and distribution by using free and open source tools and services such as OpenStreetMap and

Ushahidi25. The approach placed appropriate tools in the hands of a concerned public who were able to increase situational awareness that ultimately facilitated emergency response activities

[117].

2.2.2 VGI-based Platforms

The 2010 Haiti earthquake has highlighted the role that the Internet can have in a participatory environment [117], in which people not only consume content, but also produce new content [118].

The idea of “citizens as sensors” [15] underscores the potential that citizens have to be active information producers who can provide timely and cost-effective information (i.e., VGI) in support of disaster management activities [15,33]. In addition, local people often have a greater awareness of what is happening on the ground during a disaster than do traditional authoritative data collectors. This local knowledge should be used to complement authoritative scientific knowledge

[119]. Subsequently, the second generation of disaster management platforms places VGI at the center of the management system.

The use of VGI in disaster management has four main benefits:

• It significantly decreases the time required to collect crisis information [120].

25 https://www.ushahidi.com/ 47

• It often has comparable accuracy to authoritative sources [121].

• Its update and refresh rates are generally very rapid, especially for the affected area [122].

• As the data is open and freely accessible, different crisis management platforms from,

perhaps, different organizations can discover, process, and publish them without

restrictions [65].

VGI-centric platforms have been deployed and evaluated in several real-world disasters including the 2008 post-election violence in Kenya [123], the 2009 forest fire around Marseille,

France [124], and the 2013 Pakistan Earthquake [44], among others.

2.2.3 Hybrid Platforms

As outlined by De Longueville et al. [33], VGI is a rich and complementary source of information for SDIs, especially in the context of disaster management. Therefore, the third generation of disaster management systems finally aims to incorporate VGI into SDI (Figure

2.4). Here, the user’s role in an SDI changes from a passive recipient of data to an active

“produser” [13]. This implies a shift away from traditional practices and protocols in data production. It gradually fades the monopolistic position of authoritative organizations on crisis- related data acquisition and allows anybody, with access to the technology, to contribute in this process.

Genovese & Roche [125] discuss the strengths, weaknesses, opportunities, and threats of VGI for improving SDI in the global context of north vs. south (i.e., developed vs. developing countries)

– see Table 2.2 for an overview. Their investigation suggests that although substantial funding has been dedicated to the creation of SDIs in developed countries, there are still issues that

hinder VGI-SDI integration. For example, one weakness is the ability of users to understand VGI quality and credibility. Therefore, VGI inclusion in official SDIs may pose a threat to data integrity, and tools for quality evaluations are needed. Genovese & Roche [125] identify economics to be a limiting factor regarding SDI infrastructure development, particularly in developing countries. They also highlight that SDI map coverage seems to not be uniform: urban areas tend to have more complete coverage than rural areas. To address these drawbacks there may be opportunities to use VGI to fill existing administrative geospatial data holes in SDIs.

Works by Díaz et al. [54] and Schade et al. [28] discuss the different aspects of integration of

VGI with authoritative data under an SDI paradigm in further detail. The following section reviews several research works that develop SDI-based, VGI-based, and hybrid disaster management platforms and evaluates them according to the functions they provide and the enabling infrastructures they are built upon.

Table 2.2. Potential of VGI as a resource for SDIs (see Genovese & Roche [125] for a complete analysis).

North (developed countries) South (developing countries)

Strength Locally generated, limitless, constantly Sharing information at local level to fill the updated, cost-free information. information gaps.

Weakness Lack of control over the data (VGI credibility Lack of technological infrastructure (Internet, issue). mobile phones, or even power).

Opportunities VGI may become a new source of free data. VGI may empower citizens by making them part of collaborative governance.

Threats VGI could reduce the importance of Projects may fail because of the shortage of authoritative data. financial support.

2.3 Three Generations of Disaster Management Platforms: A Comparison

This section reviews some (prominent) implementations of SDI-based, VGI-based, and hybrid disaster management platforms and characterizes them based on the generic framework’s components (introduced in Section 2.1): functions and enabling infrastructures (see Figure 2.1).

It is not an exhaustive inventory, since the goal here is to give a short overview of recent works.

Several implementations of SDI frameworks for disaster management have been tested in different case studies. Weiser & Zipf [113] evaluate the suitability of web services developed based on SDI standards for disaster response operations. They develop a workflow of OGC web services, including WFS, WMS, and WPS, to build an emergency route-planning tool that supports disaster responders in a bomb evacuation scenario. In a case study of wildfire risk assessment, Mazzetti et al. [114] employ SDI standard services for facilitating spatial data access and processing in a grid platform for the European Civil Protection e-Infrastructure. They use

WPS to wrap a fire risk assessment model providing both a standard-compliant access interface and distributed processing. The input datasets are provided by a WCS server, which is deployed to provide interoperable access to raster data. Agosto et al. [115] present an early warning system for flood events. Their system collects data from various sources such as NASA Tropical

Rainfall Monitoring Mission website and the US Geological Survey’s Centre for Earth

Resources Observation and Sciences, processes them, and compares them to historical data series to detect potential flooding. It then publishes detected events on a map through WFS and WMS services.

Several research works have reported the successful use of VGI-centric platforms in crisis events. Okolloh and other Kenyan journalists and human right activists developed Ushahidi for mapping reports of violence after the 2007 Kenyan elections [123]. They set up a crisis map using Google Maps API, which displayed the human rights abuses that citizens reported through the Internet or SMS. Since then, the platform has been under active development and now supports integration of additional technologies such as Twitter to improve sharing crisis information. De Longueville et al. [124] demonstrate the role of VGI to support emergency planning and risk assessment activities by focusing on a forest fire event. They developed an application that used Twitter’s API26 to retrieve tweets, and data mining methods to extract relevant VGI for further processing. Imran et al. [44] present a platform called AIDR27 (Artificial

Intelligence for Disaster Response) for VGI acquisition and verification during disasters, which has successfully been evaluated in the 2013 Pakistan Earthquake. AIDR uses Twitter’s API to collect crisis-related messages, employs machine learning classification techniques to filter out irrelevant tweets, and deploys a web-based portal to publish the results.

Finally, as an attempt to harness the potential of both authoritative and voluntary spatial data worlds, some researchers discuss the different aspects of integration of VGI with authoritative data under an SDI paradigm. Díaz et al. [54] present a geospatial middleware toolset that enables users to integrate and share their resources into an SDI. They develop a framework that provides data access and processing capabilities using OGC web services including WFS, WCS, WMS,

26 https://dev.twitter.com/ 27 http://aidr.qcri.org/ 51

WPS, and CSW. The system allows users to build and execute a geoprocessing task (i.e., a workflow), and share the output (i.e., VGI) as a standard SDI service, which can either be a WFS or WCS, depending on the data type. They evaluate the effectiveness of their approach in a case study of forest fire. Schade et al. [28] propose an approach that uses the SWE framework to transform VGI into a standard data source for SDIs. They use Web mining techniques to collect

VGI from social media and employ the SOS standard to provide an interoperable access to VGI.

They employ semantic similarity techniques to validate VGI. For example, in a forest fire detection case study, they compare the description attached to Flickr’s photos with a set of disaster-related words to filter out irrelevant data. In the end, they use spatiotemporal clustering methods to detect potential events, and create and send alerts using Sensor Event Service.

Table 2.3 lists all the works reviewed in this section and provides technical descriptions of these systems. The following sections evaluate them in terms of their functionality and enabling infrastructures and highlight the challenges in existing disaster management platforms.

Table 2.3. A comparison of web-based disaster management platforms. SDI-based Platforms VGI-based Platforms Hybrid Platforms 1-i 1-ii 1-iii 2-i 2-ii 2-iii 3-i 3-ii

Weiser & Zipf Mazzetti et al. Agosto et al. Okolloh De Longueville et Imran et al. Díaz et al. Schade et al. (2007) [113] (2009) [114] (2011) [115] (2009) [123] al. (2009) [124] (2014) [44] (2011) [54] (2013) [28] Communication HTTP, Social HTTP, Social HTTP, Social Networks and HTTP HTTP HTTP HTTP, SMS HTTP media Media media Protocols System Client-server SOA Client-server Client-server Client-server Client-server SOA Client-server Enabling Architectures Infrastructure Computing Local/Cloud Grid Grid Cloud Local Local/Cloud Local/Cloud Local/Cloud Platforms Standards W3C, OGC W3C, OGC W3C, OGC W3C W3C W3C W3C, OGC W3C, OGC Data Authoritative Authoritative Authoritative Authoritative Authoritative VGI VGI VGI Frameworks data data data data, VGI data, VGI Service-based Ad hoc Service-based Service-based Discovery Service-based Service-based Ad hoc discovery Ad hoc discovery discovery and discovery, discovery, and Access access access discovery and access access access access Yes Yes Function Yes Quality Control No Yes (QoS) No No (automatic No (spatiotemporal (manually) classification) validation) Service-based Service-based Service-based Ad hoc Ad hoc Ad hoc Service-based Service-based Dissemination dissemination dissemination dissemination dissemination dissemination dissemination dissemination dissemination

2.3.1 Enabling infrastructures

In terms of enabling infrastructures, the World Wide Web is the underlying communication channel for all the platforms, although some of them employ additional communication mediums such as

SMS and social networking websites (e.g., VGI-centric and hybrid platforms). From a system architecture perspective, most of the online disaster management platforms were developed using a client-server architecture. However, the prototypes developed by Mazzetti et al. [114] and Díaz et al. [54] were implemented based on a SOA. “Client-server” refers to a traditional distributed architecture where a service requester (i.e., client) and a service provider (i.e., server) work together to accomplish a task in a tightly coupled, “ad hoc” manner.

The use of SOA for online disaster management applications is useful for three reasons. First, SOA supports a “Data as a Service” (DaaS) approach [126], which provides an interoperable solution to access data stored at different locations, as is usually the case with data needed for disaster management. Second, the adoption of SOA leads to an architecture that enables functions to be delivered based on a “Software as a Service” (SaaS) mechanism [127] utilizing common communication standards. This way, disaster data management and analysis functions can be provided for users in different locations with different access levels. It also enables distributed deployment of functionality so that distributed processing can be employed during high-demand times [128]. Third, an SOA-based development approach enables the production of systems that can be adapted to changing requirements and technologies. These are easier to maintain and allow a consistent treatment of data and functionality [129] – see Table 2.4 for a summary.

In terms of deployment, most of the platforms reviewed can be deployed on a local-, grid-, or cloud-computing environment, depending on the use case requirements. “Cloud” refers to a large-scale infrastructure that delivers on-demand, dynamically scalable resources to external

consumers over the Internet [130]. “Grid” platforms indicate a distributed system that consists of a collection of (pre-reserved) computer resources (e.g., computing power and storage) working together to reach a common goal.

Table 2.4. Benefits of using SOA in disaster management platforms.

SOA advantages Description

Data as a Service (DaaS) It provides interoperable access to data with different types and format, stored at different locations

Software as a Service (SaaS) It provides standard-compliant access to analytical functions for users in different locations

Service-based development It allows developers to build maintainable and sustainable products.

With respect to standard compliance and interoperability, all platforms were developed based on

SDI guidelines (i.e., open geospatial standards for geospatial web services, also known as OGC services) except for VGI-centric systems, which followed W3C-compatible approaches (i.e., open IT standards for web services, also known as W3C services) for platform development.

Most of the platforms use authoritative data as the main source of information, while some of them support real-time (or dynamic) data streams such as VGI. Authoritative data includes topography and land use/land cover data from national mapping agencies, and meteorological data from global data providers such as NASA28 and the USGS29. VGI examples include social media activities on Twitter and Flickr, and users’ contributions on crisis mapping platforms, and the outputs of spatial processing workflows that scientists designed and executed.

28 National Aeronautics and Space Administration: https://www.nasa.gov/ 29 United States Geological Survey: https://www.usgs.gov/

2.3.2 Platform Functions

2.3.2.1 Data Discovery and Access

The platforms of all three generations support discovery of and access to (spatial) data. To do so, most use standard-based approaches, but some adopt non-standard “ad hoc” methods30, for instance the VGI-based systems. Okolloh’s [123] Ushahidi platform permits users to search and contribute information using Ushahidi’s geoportal component, or by sending mobile phone text messages (SMS). De Longueville et al. [124] utilize Twitter’s API to search and retrieve crisis- related information (i.e., tweets), and web-crawling scripts to filter and classify the content.

Imran et al. [44] describe a platform based on free and open source tools and services that used in the 2013 Pakistan earthquake to support emergency response activities. The system uses

Twitter’s API to collect disaster-related messages (i.e., tweets), and deploys a web-based interface to disseminate the validated tweets.

There is still a lack of effective, flexible, and interoperable mechanisms for discovery of VGI. For example, recent approaches used spatial, temporal, and textual (e.g., type of disaster) criteria to search and retrieve VGI [28,42,131]. However, data quality is not considered in these search approaches, which is a shortcoming. This need emerges, for example, from the existence of spam and noisy content [132,133]. Therefore, there is a need to develop a discovery mechanism that considers quality-related parameters when searching for VGI. A demand for this is even more important when considering VGI-SDI integration, since quality is a major concern in this context.

30 The use of “service-based” access methods requires the definition of communication procedures and vocabulary, i.e., standards. Subsequently, the term “ad hoc” is used to refer to non-standards-based communication.

2.3.2.2 Data Quality

Except for the SDI-based systems, all platforms are capable of handling VGI generated by citizens. However, automated quality control functionality is implemented in only two of eight platforms in Table 2.3. Poser & Dransch [134] discuss two general approaches to VGI quality assessment: “quality-as-accuracy”, and “quality-as-credibility”. The first concept measures the level of similarity between the data produced and the real-world phenomena it describes. This approach is mainly used by data providers [134]. The second concept, often applied in the context of Web 2.0, refers to the credibility of data, especially data generated by (non-expert) users. While accuracy is an objective property, credibility is subjective, and tends to rely on users rating the credibility of other users and the information they contributed [104].

Poser & Dransch [134] propose different quality assessment approaches for different phases of the disaster management cycle, i.e., for prevention, preparedness, response, and recovery. They suggest using the quality-as-credibility approach in the prevention and preparedness phases, where there are continuous contributions from citizens. They then propose the use of the quality- as-accuracy approach during the response phase, where there is a need to collect factual information about the crisis to determine its impact. Examples of quality evaluation following either one of the two general approaches are presented by several authors (see Table 2.5) and are discussed below.

Based on the quality-as-accuracy approach, Haklay [99] examines the quality of OSM for

London, England through a comparison with Ordnance Survey datasets. To do so, he uses positional and attribute accuracy, completeness, and consistency as quality evaluation metrics.

To extend the work of Haklay to France, Girres & Touya [135] study the quality of French

OpenStreetMap data with a larger set of spatial data quality evaluation criteria including

geometric, attribute, semantic, and temporal accuracy, logical consistency, completeness, lineage, and usage. Jackson et al. [136] propose a method for quantifying the completeness and accuracy of a set of school locations in Colorado Metropolitan Area using an authoritative spatial dataset as the reference benchmark from the Department of Education and two VGI datasets from OSM and the USGS OSM Collaborative Project (OSMCP). Fan et al. [137] evaluate the quality of building footprint data in OpenStreetMap for Munich, Germany. In their work, they use completeness, semantic accuracy, position accuracy, and shape accuracy as quality evaluation criteria. De Brito Moreira et al. [138] compare VGI with authoritative data to estimate the accuracy of VGI (i.e., water level) collected by a number of volunteer students.

However, there are some limitations in using the methods reviewed above, including the availability and accuracy of reference data, contradictory-licensing restrictions, and high procurement costs of the authoritative data [139].

Based on the quality-as-credibility approach, Bishr & Mantelas [140] and Bishr & Kuhn [141] propose a trust and reputation model for quality assessment of VGI. They constructed a computational model that considers spatiotemporal context for urban planning [140] and water quality management applications [141]. Another quality assessment approach proposed by

Goodchild & Li [142] emphasize the use of procedures to control and enhance quality during the acquisition and compilation of spatial data. This is similar to quality assurance processes used by traditional mapping agencies. The method requires mechanisms to generate quality metrics for the VGI data, including metrics that evaluate the VGI against authoritative reference sources.

Schade et al. [28] propose a cross-validation mechanism to overcome VGI’s credibility challenge. The main idea in their work was to aggregate VGI from multiple sources such as

Twitter, Flickr, OpenStreetMap, etc., and process these VGI data to determine their relevance in

a given context. Among the validation techniques, k-fold cross-validation is a common approach used for VGI verification and validation [12,143,144]. Additionally, spatiotemporal analysis has been used to evaluate the quality of VGI. For example, De Longueville et al. [124] extract spatial information from VGI such as contributors’ location and place names to assess the relevance of the content to the 2009 forest fire around Marseille, France. They then performed a temporal analysis to estimate the temporal accuracy of the content compared to the actual event.

Ostermann & Spinsanti [145] recommend using spatial analysis to determine the correlation between the spatial information attached to the content (e.g., geo-tagged tweets), extracted from the content (e.g., geocoded place names), and associated with contributors’ profiles (e.g., a user’s location). The information can then be used to rate the content based on the distance from the contributor’s location to the event location and evaluate the credibility of the VGI.

Table 2.5. Research works that examined VGI quality assessment based on accuracy and credibility metrics.

Quality-as-accuracy VGI Source Quality-as-credibility VGI Source

Fan et al. [137] OSM Bishr & Mantelas [140] and An experiment Bishr & Kuhn [141] involving local citizens

Jackson et al. [136] OSM, OSMCP Goodchild & Li [142] N/A

De Brito Moreira et al. An experiment Schade et al. [28] Twitter and Flickr [138] involving students

Haklay [99] OSM De Longueville et al. [124] Twitter

Girres & Touya [135] OSM Ostermann & Spinsanti [145] Twitter and Flickr

However, some of the VGI quality evaluation methods reviewed employ a manual process to evaluate the quality of VGI, which is time-consuming. Moreover, they cannot function

effectively in real-time when facing extreme events [34]. To minimize manual intervention, automated methods have been discussed in the literature. For example, Imran et al. [44] adopt supervised machine learning to evaluate the credibility of tweets based on their textual content.

Although such automatic approaches are promising in analyzing a large amount of data in a timely manner, a challenge still exists in developing classification models that can recognize and interpret specific patterns in VGI and, thus, improve its use in disaster management [34].

2.3.2.3 Data Distribution

In terms of VGI dissemination, a number of solutions have been developed, as listed in Table

2.3. All the SDI-based platforms support standard service-based data dissemination using OGC’s

WMS. In contrast, VGI-centered systems distribute data using ad hoc, i.e., non-standard, approaches, for example, by using an open data format such as JSON. The two hybrid platforms also utilize service-based approaches.

The hybrid platform by Díaz et al. [54] uses OGC services such as WFS, WMS, and WCS to publish VGI. Although this approach facilitates sharing of user-generated information, it does not cover the temporal dimension of VGI, which is crucial in disaster management activities.

This issue is addressed by the other hybrid platform by Schade et al. [28]. They propose a new approach for VGI data management, called VGI Sensing, which uses OGC’s SWE framework to publish VGI. As a part of OGC’s Interoperability Testbed-10 (OWS-10), Bröring et al. [146] also investigated a framework for integrating VGI into SDI through the use of OGC’s SWE and

WFS standards. However, these platforms do not support dissemination of quality information, which is a major concern especially in the context of VGI-SDI integration [48]. While Cornford et al. [147] and Devaraju et al. [148] do suggest how OGC’s SWE framework and the UncertML specification [149] can be used to provide quality information (e.g., uncertainty) about sensor

observations, there is currently no interoperable approach for the dissemination of VGI data and related quality information.

2.4 Chapter Summary

This chapter reviewed different approaches to the design and development of disaster management systems. In this context, prominent web-based disaster management systems were grouped into three platform generations and evaluated in terms of their functionality and enabling infrastructure. This was followed by a summary of related works and an investigation of their technical characteristics in term of enabling infrastructures and functions. Finally, the functions and tools that are currently missing in the disaster management platforms were highlighted. They include:

• A lack of an interoperable VGI discovery and access mechanism that considers quality-

related parameters as well as spatiotemporal and textual criteria.

• A lack of an effective quality control model and an appropriate set of quality metrics for

evaluation of VGI in an automated manner.

• A lack of interoperable mechanism for the dissemination of VGI that considers data

quality.

Consequently, this research aims to provide a framework that meets the needs identified in this chapter. In this respect, it aims to design and develop a framework that:

• Enables a hybrid information sharing paradigm that supports incorporation of VGI into

SDIs,

• Offers an automated quality control procedure that considers VGI-tailored quality

metrics, and

• Provides a flexible system architecture that supports interoperability and extensibility 61

through standard compliance and modularization.

CHAPTER THREE: METHODS

This chapter presents the methodology of this research that aims to address the research questions outlined in Chapter 1 (see Section 1.4). It follows the main objective of this research: the development of a robust workflow to facilitate collecting, validating, and sharing VGI, published on social media, that is relevant, credible, and actionable for emergency response. In this context, Section 3.1 outlines a set of quality requirements and business goals to guide the development of disaster management platforms (Research Objective 4). Based on the functional and architectural requirements, Section 3.2 introduces a conceptual overview of the proposed framework, which is called hereafter as the “VGI framework” (Research Objectives 1, 2, and 3).

Section 3.3 describes a set of quality metrics specifically designed to evaluate the quality of VGI

(Research Objective 2). It includes spatial, temporal, semantic, and credibility factors. The approaches proposed to evaluate spatial and semantic dimensions of VGI are further explained in

Sections 3.4 and 3.5. Afterwards, Section 3.6 presents a technical architecture that allows the integration of VGI framework’s components into an easy-to-access, easy-to-use, and easy-to- maintain system (Research Objective 4). Specifically, it identifies architectural patterns and design principles in creating (geo)processing workflows (Section 3.6.1) and presents a development strategy for building effective disaster management platforms (Section 3.6.2). The chapter ends with an overview of the software components used to develop the system components.

3.1 Definition of Disaster Management Platform Attributes to Guide the Development

To guide platform development, a set of platform quality attributes/requirements and business goals was identified [150,151]. After retrieving an initial list, the requirements were prioritized according to their importance for disaster management as shown in Figure 3.1. In addition to

using the requirements in the framework design process, they were also helpful for evaluation of the consequences of architectural decisions, and identification of system architecture limitations/risks [152].

During the framework/platform design phase “first priority” was given to following quality attributes:

• Scalability/extensibility – it refers to the ability of the system to cope and perform under

an increased or expanding workload.

• Open systems/standards – it pertains to the use of open systems and standards to reduce

the cost of using commercial systems and to facilitate communication between the

developed systems and other systems.

• Interoperability – it indicates the ability of the system to exchange data with a variety of

other systems and devices.

These attributes ensure that systems built using the VGI framework will work together efficiently, that a set of services developed using this framework will be coherent, and at the same time will address legitimate disaster management issues such as crisis information sharing

[151].

The “secondary” design priorities were:

• Performance – it concerns with the runtime performance of the system.

• Flexibility – it indicates the ability of the system to adapt to changing requirements and

technologies, which makes it easier to maintain and allows a consistent treatment of data

and functionality.

• Integrability – it pertains to the ease of integration of various components of the system

within cost in a timely manner. 64

Figure 3.1. VGI framework business goals.

• Security – it refers to security requirements on the system in terms of user authentication

and multi-level access permissions.

• Ease-of-installation – it concerns with the cost of installing copies of a system in terms of

deployment, configuration, and operation.

• Ease-of-use – it refers to the ease with which an end-user can operate the system.

• Portability – it concerns with the ability of the system to be able to run on various

platforms regardless of the host hardware environment and operation system.

These criteria focus on system capability and quality. Furthermore, there are additional “low- level priorities”, such as distributed development, and ease-of-repair, along with a number of quality metrics (see Figure 3.1) that should ideally be satisfied to achieve all business goals.

3.2 Conceptual Design

Figure 3.2 illustrates the conceptual overview of the VGI framework, which was developed based on the functional and architectural requirements discussed above. The building blocks of the proposed framework are:

• VGI Broker, to find and collect VGI,

• VGI QC (Quality Control), to evaluate quality of VGI,

• VGI Publisher, to publish quality-assessed VGI, and

• VGI Discovery, to advertise VGI metadata in open catalogues.

These four components are explained in the following subsections.

Figure 3.2. A conceptual workflow for discovery, assessment, and dissemination of VGI. 3.2.1 VGI Broker

The VGI Broker provides a service interface to find and collate user-generated content from various social media platforms. Many social media platforms provide public APIs for clients to interact with them using their specific request-response message system. Although they usually adopt REST (REpresentational State Transfer)-based interfaces and support popular data formats

(e.g., JSON or XML), there is no uniform description of service interfaces and data encodings

[42]. Hence, the VGI Broker module connects the platform user to different APIs, enabling data retrieval from multiple social media platforms via a single service interface. It translates a single query to multiple API-based queries and handles different request-response data formats. The retrieved data is then stored in the VGI Repository based on the data models designed for each platform (e.g., Twitter, Flickr, etc.).

The APIs typically allow clients to collect data using one or several of the following query constraints:

• A time period during which the messages were posted,

• A geographical region for the messages with geospatial information attached (i.e., 67

geotagged messages), or

• A set of keywords/hashtags that must be present in the messages.

Data collection strategies impact the data obtained from social media platforms. For instance,

Olteanu et al. [153] discuss that the datasets obtained from Twitter using a keyword- and location-based query may have different characteristics. They also suggest that (a) “collecting everything and then post-filtering” is an ineffective sampling approach and (b) part of data filtering must be done at query time. This is because if clients use a broad/blank query to collect everything, at some point they start losing tweets because of Twitter’s restricted data access policy31 according to which it provides only a random sample of 1% of all tweets through its

API. However, it needs to be noted that pre-filtering of data based on geotags32 will result in a significant loss of information since only a minority of social media messages is currently geotagged – e.g., 2% of crisis-related messages according to Burton et al. [154]. Hence, using a keyword-based sampling approach is a logical choice to collect social media messages33. The next filtering step is done using the VGI QC module.

3.2.2 VGI QC

Having VGI data stored in the repository allows the VGI QC module to perform quality control checks on the data and to generate quality-related metadata that are stored in the VGI Repository.

This is a crucial step especially in the context of disaster management, where VGI can potentially be used along with authoritative data for decision-making, since public authorities are

31 Most large data providers follow the same policy and offer restricted access to a limited amount of their data through an API. 32 It refers a geographical location attached to a posting on social media websites such as tweets on Twitter, photos on Flickr and Instagram, and videos on YouTube. 33 A time-based query can also be used along with a set of keywords to specify a temporal window when collecting data from an archive of past messages.

reluctant to use VGI in disaster response operations due to data quality concerns [12,27,31]. The quality evaluation also enables quality-based data retrieval, which is missing in current social media platforms. The VGI QC module manages the quality control procedures in the proposed framework. The quality control workflow and algorithms are discussed in detail in Section 3.3.

3.2.3 VGI Publisher

The VGI Publisher module is a data service that disseminates quality-assessed VGI following a

DaaS (Data as a Service) approach. It allows clients to access and retrieve VGI based on spatiotemporal and quality parameters in an interoperable manner. This means that instead of dealing with each social media platform and their different request-response paradigms, one only needs to communicate with the VGI Publisher, which provides a single interface and a common data encoding for data dissemination for all different media streams.

3.2.4 VGI Discovery

Finally, the VGI Discovery module acts as a catalogue service to discover, browse, and query metadata about the available VGI datasets published by the VGI Publisher. It offers spatiotemporal and quality-based search options to the clients. A search directed to the VGI

Discovery module returns a metadata document, which includes information about data such as time, location, and quality, as well as a link to the data itself.

3.3 Quality Evaluation Metrics

In GIScience, spatial data quality elements are used to ensure that the production of

(authoritative) spatial data is in line with the well-established principles and guidelines (e.g., ISO standards) [155–157]. Examples of such quality standards concern “lineage”, “positional accuracy”, “attribute accuracy”, “logical consistency”, “completeness”, and “temporal accuracy”. Although some of these quality metrics may seem independent of a specific

application, they need to be defined with respect to the particular context of data (or map) use to ensure that the data is fit-for-purpose [34,158,159].

Fitness-for-purpose is a critical issue for quality assurance of VGI [34,158,159], whereby VGI demands a somewhat different approach to quality assessment. This difference emerges from (1) the different procedures undertaken to produce authoritative data and VGI, (2) the socio- technical nature of VGI systems, and (3) the heterogeneity of VGI (see [158] for a detailed discussion). Therefore, several approaches have been proposed by researchers to fill this gap, including “crowdsourcing”, “social”, “geographic”, “domain”, “instrumental observation” and

“process-oriented” approaches [142,158] (see Table 3.1 for a brief description). These approaches are not necessarily used in isolation for a given use case, but rather combined for real-world applications [158].

This research has adopted three approaches to ensure that the quality assurance fits the disaster management purpose:

• “Crowdsourcing” to evaluate user rating and to measure quality of a contribution (e.g.,

retweets),

• “Geographic” analysis to evaluate the spatial quality of the information, and

• “Domain” analysis to ensure the relevance of the information to a given context, i.e., a

disaster.

Consequently, the set of quality metrics includes: positional nearness and cross-referencing as

“geographic” analysis approaches, temporal nearness and semantic similarity as “domain” approaches, and credibility as a “crowdsourcing” approach.

Table 3.1. VGI quality evaluation methods and the metrics used in this research (adapted from Goodchild & Li [142] and Haklay [158]).

VGI QC methods Description The proposed metrics

Crowdsourcing It relies on the principle of abundance where quality assurance Credibility emerges from “repeated verification” by multiple contributors.

Social It builds on the principle of abundance where there is a need N/A for “experienced contributors” (i.e., gatekeepers or moderators) to verify others.

Geographic It uses “geographical knowledge” to evaluate the validity of the Positional nearness, cross- information that is received by contributors. referencing

Domain In addition to geographical knowledge, it uses a “specific Temporal nearness, knowledge” that is relevant to the domain in which information semantic similarity is collected.

Instrumental It relies on the use of (technological) “equipment” to lessen the N/A observation subjective aspects of data collection by a human that might make an error.

Process-oriented It relies on volunteers who go through some “training” before N/A starting the data collection process, which itself is highly “structured”.

It should be noted that this research does not strive to define indefectible quality evaluation.

Rather it aims to identify a set of quality metrics that are suitable for VGI quality assessment and to measure the usefulness of the metrics in disaster management (Research Objectives 2 and 5).

The proposed metrics are:

• Positional nearness, which exploits the spatial dimension of social media messages to

measure their geographical proximity to the (social media) activity center of a given

disaster.

• Temporal nearness, which considers the time difference between the time of posting

social media messages and the event time (duration).

• Semantic similarity, which analyzes textual content of social media messages to evaluate

their relevance to a given disaster.

• Cross-referencing, which uses spatial information conveyed by social media messages to

evaluate the geographical relationship between messages from different platforms.

• Credibility, which evaluates users’ reactions to social media messages through an

assessment of credibility factors, for instance how many times a message is re-posted

(i.e., retweet for Twitter).

The following subsections describe the proposed quality metrics in more details.

3.3.1 Positional Nearness

Equation (3.1) is used to calculate the positional nearness score (PNS), where ∆! is the distance

(in kilometers) between each contribution (e.g., a tweet) and the centroid34 of all contributions

(e.g., all tweets) for a given context, which can be calculated as the mean center using Equation

(3.2).

1 ! > ∆! → !"# = 1 ∧ ! < ∆! → !"# = ! (3.1) ∆!

! ! ! ! ! = ! , ! = ! (3.2) ! ! !!! !!!

34 There are numerous ways to estimate a centroid, the arithmetic mean, weighted mean, center of minimum distance, center of greatest intensity, etc. For simplicity, the arithmetic mean has been used for this work as an initial starting point.

! is a scalar defined by the standard distance deviation of the set of contributions. Equation (3.3) is used to calculate the two-dimensional equivalent of a standard distance deviation, where !! is the distance between each point, ! and the mean center and ! is the total number of points. In essence, this model gives greater weight to contributions closer to the center of the set of contributions.

! (! )! ! = ! (3.3) ! − 2 !!!

In order to calculate the distances in Equations 3.1-3.3, the geographical location of each contribution is required. The location information can be attached to social media messages as a pair of geographical coordinates (longitude, latitude) also known as “explicit” spatial information. This information can be obtained from GPS or IP address when location-based services (LBS) are used [160], depending on the social media client application (mobile vs. desktop). However, studies shown that only a small portion of contributions has the location component attached, for example only around 2% of tweets are geo-tagged, while more than

10% of tweets include references to places in the text [161]. The place names (toponyms) that are mentioned in the text are also known as “implicit” spatial information. Munro [162] and

Gelernter & Balaji [163] discuss that a disaster-related message that has implicit spatial information is more valuable than a message that does not have a location component.

However, there can be a fundamental difference between explicit and implicit spatial information in terms of the geographic location they point to35. The explicit location indicates where a

35 In this context, the difference between explicit and implicit spatial information refers to the distance between the spatial objects they represent.

message is posted or the location of its author, while the implicit location is the location that the author writes about in the message. For instance, if a user is in the vicinity of an event and reports his/her observations using a LBS-enabled mobile device, it is likely that the distance between the explicit and implicit spatial information is small – this type of user is called

“eyewitness” in the context of a disaster [153]. Otherwise, the level of difference may vary, depending on the users’ location and the platform they use. For example, the distance between explicit vs. implicit location is relatively large for a user in Canada compared to a user in the

Philippines reporting a disaster happening in the Philippines. Contributions with a high relative distance between explicit and implicit spatial information are considered responses to “cyber events” [164].

In the case study, presented later, both explicit and implicit locations are considered in the evaluation of positional nearness. To do so, the metadata of social media messages are parsed to extract the attached geographical coordinates and geoparsing techniques are applied to extract and geocode the place names. Section 3.4 describes the techniques used to geoparse the place names embedded in social media messages. Having location elements extracted from social media messages, the following rules are applied to estimate the location to calculate the PNS:

• If only the explicit or implicit element is available, then the available one (i.e., explicit or

implicit) is considered as the location.

• If both explicit and implicit elements are available, then there are two scenarios to

consider:

o If at least one of the elements represents a location outside the boundaries of the

affected region, then the element representing a location that is closer to the

disaster area is considered as the location.

o If both elements represent locations inside the boundaries of the affected region,

then the mean of the coordinates of both elements is considered as the location.

3.3.2 Temporal Nearness

The creation date and time of each contribution (e.g., a tweet) is compared to the event time (or start/end of the event if a duration) to determine the number of days since the event actually happened (∆!). Equation (3.4) is used to calculate the temporal nearness score (TNS). As ∆! increases TNS decreases.

1 ∆! < 1 → !"# = 1 ∧ ∆! > 1 → !"# = (3.4) ∆!

It should be noted that an event can be a particular point in time as for an earthquake or a timespan (duration) as for a typhoon. Since the focus of this research is on Typhoon Ruby, the scale of measurement is “day” – the scale could be “hour” for an earthquake. So if the creation date and time of a contribution is within the period of the event, the full temporal nearness score is then assigned to the contribution (!"# = 1). Otherwise, it is compared to start/end of the event and the time difference is calculated in days.

3.3.3 Semantic Similarity

Each contribution (e.g., a tweet) is compared to a pre-defined dictionary of disaster related words and then Equation (3.5) is used to calculate the semantic similarity score (SSS), where !! is the word similarity score calculated for each word in a contribution and ! is the average number of words with a score of greater than zero in a contribution.

! ! !!! = !!! ! (3.5) !

Since there always is a portion of social media messages that is irrelevant or spam [165,166], this so-called noise is filtered out beforehand. The “related” contributions are then analyzed using

natural language processing (NLP) techniques and the semantic similarity score is calculated.

Section 3.5 describes the approach used to filter the social media messages, create the reference dictionary, calculate word similarity score (!!), and estimate !.

3.3.4 Cross-referencing

The spatial extent in the form of an axis-parallel minimum-bounding rectangle (MBR) is calculated from all contributions for each social media platform (e.g., the Twitter dataset).

Afterwards, the point-in-polygon operation is performed for each contribution (e.g., a tweet) on each social dataset MBR (e.g., the Twitter dataset). Equation (3.6) is then used to calculate the cross-referencing score (CRS), where !! is the number of bounding boxes that a contribution falls within and ! is the total number of bounding boxes/media streams (see Figure 3.3 as an example).

!! !"# = (3.6) !

Figure 3.3. An imaginary location of a tweet that falls into the (imaginary) spatial extents of Twitter, Flickr, and Google Plus.

It should be noted that spatial extent in terms of social media contributions is a vague concept.

This thesis uses it here as a measure of nearness. It should also be noted that there are many ways to represent nearness. This research has chosen nearness to mean, for simplicity, “something is nearest if it falls within the intersection of all contribution MBRs”.

3.3.5 Credibility

A set of credibility factors was defined and the maximum value of each factor within all the contributions (e.g., all the tweets) is calculated for each social media platform (e.g., Twitter) for a particular event (e.g., Typhoon Ruby). For example, the Twitter’s API allows a client to collect a number of credibility factors for each tweet. They include: verification36 (of the tweeter), the tweeters followers’ count37, how many times the tweet has been “favorited”38, and the retweet

39 count . Equation (3.7) (a) is then used to calculate a credibility score for each factor (!"!), where !!" is the value of each factor ! for a contribution ! and !! is the maximum value of each factor within all ! contributions. The total credibility score (!") is calculated using Equation

(3.7) (b). ! is the number of factors used to assess credibility.

! !!" !"! a !"! = , b !" = (3.7) !! ! !!!

Table 3.2 lists the credibility factors used in the case study of this research to evaluate the VGI framework.

36 A verified Twitter account formally validates the identity of the person or company that owns the account. 37 The number of followers this account currently has. With more followers, a Twitter account gains more attention, thus increasing its popularity. 38 Approximately how many times a Tweet has been favorited by Twitter users. Favoriting a Tweet indicates that a user liked a specific Tweet. 39 Number of times this Tweet has been retweeted. Retweeting means a reposting or forwarding message on Twitter.

Table 3.2. Suggested credibility factors for well-known social media platforms.

Social media platform Credibility metrics

Twitter Verification (of a tweeter).

The follower count of a tweeter.

How many times a tweet has been “favorited”.

How many times a tweet has been retweeted.

Flickr How many times a photo has been “viewed”.

Google Plus How many times a post has been “re-shared”.

How many times a post has been “plus-oned”40.

How many times a post has been “replied” to.

Instagram How many times a photo/video has been “liked”.

How many times a photo/video has been “commented” on

The follower count of an Instagram user

3.3.6 Quality Score

Finally, the VGI QC module calculates a total quality score (QS) for each contribution (e.g., a tweet) summing individual quality scores calculated for each metric (Equation (3.8)). Summation is necessary since individual scores can be zero. For instance, positional nearness score (PNS) is zero when a contribution is without a spatial reference (i.e., geographical coordinates), including both explicit and implicit.

!" = !"#! + !"#! + !!!! + !"#! + !"! (3.8)

40 A “plus-one” or “+1” indicates that a user liked a specific post on Google Plus.

All quality scores will range between zero and five, with zero indicating comparatively low quality and five indicating a comparatively high-quality contribution for disaster management.

Quality control is performed as an iterative process and, therefore, quality scores will evolve over time, as more data is added to the VGI Repository. The iterative nature of the QC process stems from the fact that (ideally) new data is added to the system during a disaster, regularly if not continuously. Accordingly, the quality scores need to be updated since:

• PNS calculation is dependent upon !, which is estimated based on the location of all

contributions.

• TNS algorithm uses the times when a disaster starts and ends, which are not always

known41.

• SSS employs a supervised classification model to identify noises, which can be updated if

new annotated data is added to the system. It also estimates ! to calculate the score,

which can be a different value if new data is incorporated in the estimation process.

• Similar to PNS, CRS uses the location of all contributions to estimate the spatial extent.

• CS calculation is based on the credibility factors such as the number of re-post (e.g.,

retweet), which can change over time.

This will, in turn, result in an update to QS (see Table 3.3 for a summary). To enable quality- based queries, all the quality scores (i.e., PNS, TNS, SSS, CRS, CS and QS) for each contribution are stored in the VGI Repository.

41 Although duration of an event can sometimes be predicted, for example for typhoons, it can still be different in reality.

Table 3.3. A summary of quality evaluation metrics and their characteristics.

QC metric Description Evolving

PNS Considers spatial nearness of each contribution to the center mean Yes, as it depends on ! of all contributions.

TNS Considers temporal nearness of each contribution to the duration Yes, as it depends on ∆! of an event.

SSS Considers the relevance and semantic similarity of each Yes, as it depends on ! and contribution to disaster-related content. supervised learning

CRS Considers the number of social media’s bounding boxes (i.e., Yes, as it depends on spatial extents) each contribution falls into. geographical coordinates

CS Considers the credibility factors of each contribution. Yes, as it depends on (potentially changing) credibility factors

QS Considers PNS, TNS, SSS, CRS, and CS to calculate the total Yes, as it depends on PNS, quality score. TNS, SSS, CRS, and CS

3.4 Geoparsing

As discussed in Section 3.3.1, it is necessary to geoparse the place names embedded in social media messages to apply the quality assessment procedures that are based on location, i.e., positional nearness and cross-referencing. Geoparsing refers to the process of identifying the implicit spatial information (e.g., place names) in the text and associating them with geographical coordinates [163,167].

Figure 3.4 summarizes the major steps of the geoparsing process used in this study. First, NLP techniques are used to recognize and extract named entities. In this step, only “location” entities are selected and others such as “person” and “organization” entities are filtered out. The place

names are then georeferenced using a geocoder. Finally, the results are stored in a database for further processing. The following sections describe the details of each step.

Figure 3.4. Geoparsing procedure workflow. 3.4.1 Place Name Extraction

As shown in Figure 3.4, the first step in extracting place names is tokenization, which breaks text

(e.g., a sentence) into distinct tokens (e.g., single words). Note that a token can be a word, phrase, symbol, or another meaningful element, but for this work it is intended to refer to some

English word. A part-of-speech (POS) tagger then classifies the tokens into lexical categories such as nouns, verbs, adjectives, etc. and labels them accordingly. Finally, a named-entity recognition (NER) engine is used to identify the noun phrases that refer to specific types of entities, such as “persons”, “organizations”, and “locations”. For instance, in “Apple was founded by Steve Jobs and two others in 1976 in Cupertino.” “Apple” is tagged as

“organization”, “Steve Jobs” is tagged as “person”, and “Cupertino” is tagged as “location”. For the purpose of this study, only “locations” are selected in this step and other types of entities are discarded. One may argue that entities classified as “organizations”, should also be selected since

they are a type of POIs (points of interest) and thus can be geocoded. For example, “University of Calgary” is an organization name in the following tweet that can be geocoded:

“Celebrating #ucalgary50 with just a few of our closest friends @ University

of Calgary”

However, an initial investigation on a (disaster-related) Twitter dataset revealed that organization names mentioned in the messages represent either international humanitarian organizations such as Red Cross42 and ShelterBox43 or UN (United Nations) agencies such as UNICEF44. The locations identified for these organizations did not contribute much to situational awareness.

Hence, “organizations” entities are filtered out for the purpose of this research.

To realize the place name extraction pipeline, there is a need for a suite of NLP tools. However, named entity recognition is a difficult task especially when dealing with social media messages, because of their short, noisy, and unstructured nature [163,167,168]. For instance, Ritter et al.

[169] and Liu et al. [170] report that the use of NER methods usually have a 85-90% success rate on longer texts but a success rate of only 30-50% for tweets. There is a need to perform an evaluation on existing NLP software libraries, such as Stanford NER [171] and DBpedia

Spotlight [172], to explore their success rates. Given the data, the software package that provides the best results should be selected as the place name extractor tool. Chapter 4 (Section 4.2) reports an experiment undertaken to select the most appropriate place name extractor for this study.

42 http://www.redcross.org/ 43 http://www.shelterbox.org/ 44 http://www.unicef.org/

3.4.2 Geocoding

The next step in geoparsing is to assign geographical coordinates to the extracted place names, also known as geocoding (see Figure 3.4). The geocoding process consists of three main steps:

• Parsing the input data (i.e., place name),

• Querying the reference database (e.g., street network), and

• Assigning geographical coordinates, which can be either obtained directly from the

reference database or calculated using interpolation techniques [173].

There are several online solutions that provide easy-to-use geocoding services to users, including end-users and developers, and at a low cost45 [174]. Examples include Google’s46 geocoding service and the OSM-based Mapzen’s47. Moreover, they function with a high level of availability, scalability, and performance. However, since they use different reference databases and geocoding algorithms, the coordinates they return may be different [173,175–177]. This highlights the need for evaluating the quality of the results of existing online geocoding services in terms of match rate and similarity [173,175].

An important factor of geocoding accuracy is place name ambiguity. Place name ambiguities refer to the fact that locations can share a single place name [178], for example “London,

Ontario” vs. “London, England”. For instance in the case study’s test dataset, around 16% of the geocoded place names do not fall inside the Philippines boundary (the case study location), meaning that 16% of the Filipino place names are also place names in other countries. To

45 Most online geocoding providers offer free-of-charge plans but limit the number of requests that can be sent in a specific period of time, for example 10 requests per second for Google’s. 46 https://developers.google.com/maps/documentation/geocoding/intro 47 https://mapzen.com/projects/search/

minimize the error, thus disambiguating the place names, the country code48 can be incorporated into the geocoding requests, which in this research is “PH” for the Philippines.

Another solution to deal with place names ambiguity issue employed by modern geocoders is that they usually return more than one (candidate) match for a given query. They use heuristics such as population or user’s location49 to rank the results [178], which is based on a probability score calculated for each candidate. For instance, the probability score is represented as “location type” and “confidence” score in the geocoding response returned by Mapzen and Google, respectively. Therefore, it makes sense to use the pair of coordinates with the highest level of confidence to georeference the text when using online geocoding APIs. The geocoder selection process for the case study of this research is described in Chapter 4 (Section 4.3).

3.5 Content Analysis for Semantic Similarity Assessment

Studies have shown that a fraction of social media messages is not relevant to a particular disaster, because of the data collection method used50 [153], or presence of irrelevant messages that have the same hashtag or keywords as the relevant ones [180] for example. The irrelevant messages do not contribute to situational awareness [166,181,182] and, therefore, they should be discarded from the SSS calculation (!!! = 0). This filtering process can be done using crowdsourcing, keyword-based filtering, or automatic text classification techniques [25].

Moreover, there is a need to process the “relevant” messages based on their linguistic features to evaluate their similarity to a given disaster [183].

48 Online geocoders support two-letter country codes defined in ISO 3166 standard to restrict the results to a certain geographical region. See [179] as an example. 49 This means the region from which the geocoding request is sent. 50 For example, when following “collect everything, and then post-filtering” approach as discussed in Section 3.2.1.

Figure 3.5 depicts the steps undertaken in this research to achieve these goals (i.e., filtering irrelevant messages and performing semantic similarity assessment). First, machine-learning algorithms are used to build a text classification model. This model is then used to identify irrelevant messages. Afterwards, NLP techniques are applied to perform a semantic similarity assessment between the relative messages and a disaster-related dictionary of words. In the end, the results (i.e., SSS) are stored in the database. Individual steps are described in further detail in the following subsections.

Figure 3.5. Content filtering and semantic similarity evaluation workflow. 3.5.1 Text Classification

As shown in Figure 3.5, the first step in content analysis is to train a text classifier to remove irrelevant messages prior to any further analysis. There is a wide variety of (supervised and unsupervised learning) techniques proposed in the literature to automate text classification (see

[25] for a survey). Since classification algorithms based on supervised learning achieve results with higher quality [25,184], a supervised learning approach has been adopted in this study to

perform the text classification task. All supervised classification methods use a set of human- labeled data (i.e., training dataset) to build a model. In this study, the classification methods used a set of annotated tweets that Imran et al. [44] collected using the AIDR platform during

Typhoon Ruby in the Philippines in 2014. It contains 9,675 tweets that are categorized into two main groups: relevant and irrelevant. Relevant tweets are also coded considering their content using a set of categories, including “Request for Help/Needs”, “Humanitarian Aid Provided”,

“Infrastructure Damage”, and “Other Relevant Information”51 (see Table 3.4).

Table 3.4. Categories used for classifying tweets.

Category Description Example

Requests for Help/Needs Tweets referring to need of #DSWD to send more relief to Ruby-hit areas something (e.g. food, water, or http://t.co/n4Kci4kfsj shelter) or someone (e.g. volunteers, or doctors)

Humanitarian Aid Tweets referring to Churches, Schools Take in Typhoon Hagupit Provided humanitarian efforts by Evacuees http://t.co/JZenZyo63c humanitarian/emergency response organizations

Infrastructure Damage Tweets referring to the status of SAN REMIGIO-Per Brgy.Capt Segarino, some critical infrastructures light material houses near shoreline in Brgy Tambongon damaged by big waves #RubyPH

#mediangbayan

Other Relevant Tweets not referring to any of Official list of localities: Typhoon Ruby Information the previous categories but still (Hagupit) Storm Surge Advisory disaster-related http://t.co/ngAruzotjc

51 Volunteers performed the labeling process: 3-5 volunteers examined each tweet and the final label is based on majority voting.

Not Informative Tweets not contributing to Stay safe everyone ! #RubyPH situational awareness

To prepare the tweets for the training process, a set of standard text preprocessing operations are performed:

• First, all non-words, including URLs, user mentions, punctuations, and special characters,

are removed.

• Each tweet is then split up into single words or “tokens” – this process is called

tokenization.

• Afterwards, to avoid too many variables, capital letters are changed to lowercase and

stop-words52 are removed.

• Finally, the stemming process is performed to reduce all words to their root (e.g.,

considering “provides”, “providing”, and “provision” as equivalent) to simplify further

analysis.

The vectorization process is then performed on the remaining tokens to create a set of feature vectors using the “Term Frequency - Inverse Document Frequency” (TF-IDF) weighting method, which favors the term that is frequent in a document but infrequent in a collection of documents

[185]. In the end, the feature vectors are used to train the classifier using machine-learning algorithms.

Many machine-learning algorithms are available, however, the choice depends on the specific problem settings [25]. For example, Imran et al. [186] used Naïve Bayes, Zhou & Zhang [187]

52 The most common words, including “a”, “an”, “the”, “are”, “as”, etc., that do not contain valuable information.

applied SVM (support vector machine), Imran et al. [44] employed Random Forest, and Hashemi et al. [188] proposed J48 for text classification. Others [182,189] investigated the effectiveness of ensemble methods in classification tasks. The main principle behind ensemble methods is to weigh several individual classifiers, and combine them to obtain a classifier that outperforms each of them separately (see [189] for a survey). For example, Verma et al. [182] report a successful use of this method for text classification. However, in the end, the most appropriate classifier should be selected based on the application settings and, therefore, the use of different datasets, different baselines, and different performance measures may lead to selection of a different classifier [25]. For this study, a performance evaluation was conducted on the machine learning algorithms reviewed to select the most appropriate text classifier. The results are reported in Chapter 4 (Section 4.4).

3.5.2 Similarity Assessment

Once the relevant messages were distinguished using the text classification model, they are compared with a reference dictionary of disaster-related terms to assess their semantic similarity.

CrisisLex [153] was adapted to build the reference dictionary for this research. It is a lexicon of

380 crisis-related terms that frequently appeared in relevant tweets during different types of disasters between October 2012 and July 2013 in the US, Canada, and Australia [153]. Since there are two-part terms (i.e., bigrams) such as “flood crisis” and “flood victims” in CrisisLex, they were converted into single words (i.e., unigram) and the duplications were removed. Given the example provided, “flood”, “crisis”, and “victims” are final results of the conversion process.

This process generated a set of 288 single words that are used as the reference dictionary. The semantic similarity assessment procedure begins with a preprocessing step where NLP methods are used to remove stop-words and all non-words (i.e., URLs, user mentions, punctuations, and

special characters), change capital letters to lowercase, and tokenize the message. Each word of a message is then compared with the reference dictionary and a word similarity score is calculated.

An approach proposed by Li et al. [190] has been adopted to develop the word similarity evaluation algorithm. The similarity ! (!!, !!) between words !! and !! is modeled as a function of path length (!) and depth (ℎ) using Equation (3.9).

! !!, !! = ! ! . !(ℎ) (3.9)

! ! = !!!" (3.10)

!!! − !!!! (3.11) ! ℎ = !!! + !!!!

! and ! are used to scale the contribution of shortest path length and depth, respectively. The values of ! = 0.2 and ! = 0.45 were used, as proposed by Li et al. [190], who selected these parameters by investigating the performance of the model against human common sense. In this context, they calculated word similarity on a benchmark word set [191] with human ratings.

Then, by changing the value of ! and !, they computed a set of correlation coefficients between the computed semantic similarity values and the human ratings and finally selected the ones with the highest correlations. Hence, they were able to select the optimal values of ! and !.

Here, WordNet [192] is used as a hierarchical semantic knowledge base in which words are organized into synonym sets (synsets) along with semantics and relation pointers to other synsets

[190]. To find a path between two words, each traverses the lexical tree towards the top until the two paths meet. The synset at the point where the two (shortest-distance) paths meet is called the

Lowest Common Ancestor (LCA). The minimum path length between two words is calculated by counting the number of synset links along the path between them. The depth is obtained by

counting the levels from the LCA to the top of the lexical tree53. For example, as shown in Figure

3.6, the shortest path between “boy” and “girl” is “boy-male-person-female-girl” and the minimum path length is 4. Since the minimum path length between “boy” and “teacher” is 6, one could say “girl” is more similar to “boy” than “teacher” to “boy”. Considering the “boy-male- person-female-girl” example, the synset of “person” is the LCA for words “boy” and “girl” and its depth in the hierarchy is 2 (“person-being-entity”) [190].

Figure 3.6. Hierarchical semantic knowledge base (adopted from Li et al. [190]). During word similarity evaluation each word of a message is compared with each word in the reference dictionary. Since each word can map to multiple synsets, there may be several paths

53 The depth is taken into account to model the fact that LCA nodes closer to the top of the lexical tree indicate more abstract semantics and thus less similarity than the ones farther away from the top of the tree [190].

between each pair of words. In this case, the synsets with the shortest path between them are selected as the most similar pair of synsets. This shortest path is then used to calculate the word similarity score.

On the other hand, since a word from a message can be similar to more than one word in the reference dictionary, which may result in multiple similarity scores, the maximum score is considered as the word similarity score (!! in Equation (3.5). For example, when comparing the word “lake” with the reference dictionary, the algorithm returns 201 words with a score of greater than zero such as “medical”, “floods”, and “stream” with similarity scores of 0.074,

0.200, and 0.635, among others. The most similar word to “lake” is “water” with the similarity score of 0.775.

To calculate the semantic similarity score for an entire message, there is a need to calculate the average number of words with a score of greater than zero in a message (! in Equation (3.5). To do so, the word similarity evaluation is performed on a large sample of tweets (i.e., the training dataset used in Section 3.5.1) while recording the word similarity score and counting the number of words with the score of greater than zero for each tweet. Since count data usually follows a

Poisson distribution [193], data is analyzed using the Poisson regression model and the mean of the count distribution is considered as the average number of words with a score of greater than zero, i.e., !. Once !! and ! are estimated, the semantic similarity score can be calculated using

Equation (3.5) from Section 3.3.3.

3.6 Technical Design

As mentioned in the research objectives (Section 1.4), this research employs open standards/specifications (e.g., OGC web services) and follows a SOA-based development approach (e.g., SaaS and DaaS) to build the VGI framework. The following subsections explore

this strategy. First, they explain the design principles used to build different system components as services and to integrate them in a (geo)processing workflow. Then, they present the methods for implementing VGI framework’s modules in such a way that they meet the platform quality requirements and business goals outlined in Section 3.1.

3.6.1 Service Design Principles

A service-based development approach enables developers to model a process as a workflow of services in which services can be replaced by other services, or reused in other workflows to ensure architectural flexibility. Following this strategy, the VGI framework’s components are considered as a set of processes to be modeled as workflows of services. In such workflows, it is important to consider the design principles for creating, integrating, and publishing the services involved.

3.6.1.1 Creating Services

When designing services, the goal is to maintain an appropriate balance between multiple quality requirements and business goals, depending on their priorities [67,68]. For example, in this study, performance factors, including scalability and performance, and flexibility factors, including flexibility and integratability, are two sets of high-priority quality requirements that should be met (see Section 3.1). Service granularity is one of the important design principles to support the balance. Fine-grained services provide elementary, non-decomposable operations as single services and, therefore, can be reused in other workflows, or integrated with other services to create new applications. This characteristic of them promotes flexibility. On the other hand, coarse-grained services provide a group of operations wrapped in single services, which improves performance. In this research, the spectrum of granularity levels was considered to find

the right balance between flexibility and performance factors when designing the VGI framework’s components.

Another important principle in creating services is the service design strategy, which can follow a top-down or bottom-up approach. A top-down approach starts by designing the service interface description including the supported operations, (web) access protocols, and input and output parameters. Then, the business logic (e.g., algorithms) behind the service is implemented.

Following a bottom-up approach, a web service is developed based on the existing application and then the service interface description is generated. The two design approaches are generally used in parallel to develop models and scenarios that are reusable and efficient [73]. In this work, this mixed approach has been adopted in developing of the VGI framework: a bottom-up approach is used to develop processing services where there is a need to wrap existing applications and expose them as standard web services, and a top-down approach is used otherwise and for developing data services.

3.6.1.2 Connecting Services

Once the services are created, they should be integrated into a workflow of services to accomplish a task, for example running a quality control procedure. There are three main service-chaining patterns to manage such workflows: (1) transparent pattern, which requires full user interaction to manage the service chain, (2) translucent pattern in which a middleware module manages the service chain, and (3) opaque pattern, which exposes the service chain as a single service instance [74].

To select an appropriate service-chaining architecture, it is necessary to consider the platform’s quality requirements, i.e., flexibility and performance factors in this research. Since users fully control the service-chain in a transparent pattern, it increases the flexibility of the service chain –

e.g., services can be replaced with other services or be reused in other workflows. However, due to users’ intervention, the service chain is prone to human error and it may get difficult to manage with increasing the number of services involved. On the other hand, using an opaque chaining pattern improves performance but decreases services reusability and architectural flexibility. Translucent patterns establish a middle ground: they strike a balance between flexibility and performance. They may not perform as quick as opaque services, but they provide service reusability. Hence, for the platform discussed in this work, a translucent pattern has been adopted to design and implement (geo)processing workflows.

To realize this architectural pattern, several service chaining design principles should be considered [75], including (i) the workflow control pattern, which concerns with the management and execution of a service chain (ii) the data interaction pattern, which describes how data are transferred between services involved in a workflow, and (iii) the communication pattern, which focuses on message exchange mechanisms between services in the workflow – see Section 2.1.1 for a discussion. For the purpose of this research, a centralized chaining method has been followed to design the workflow of services and an engine-controlled approach, specifically a service-based method (i.e., management service), has been employed to manage the execution of the workflow. Additionally, data referencing has been used to “move” data between services, and asynchronous mechanisms have been adopted for time-consuming

(geo)processing workflows.

3.6.1.3 Publishing Services

The VGI framework aims to collect, evaluate, and share VGI during a disaster. Interoperable tools can facilitate this process [76], and open standards play a key role in the establishment of interoperable systems. These are of high-priority quality requirements for developing disaster

management platforms (see Section 3.1). This research, as mentioned earlier, follows a SOA- based approach for providing functionality and data as a service. Web services, also known as

W3C services, implement SOA principles to create standard-compliant service interfaces (e.g.,

WSDL), data encodings (e.g., XML), and message exchange mechanisms (e.g., SOAP), to eventually provide interoperability. In GIScience, however, SDIs aim to provide interoperability through a set of open standards/specifications – for example OGC standards.

From a functional perspective, W3C and OGC services represent different standards, which may result in incompatibilities when reusing or integrating services into other workflows54 [196–198].

This is because:

• Unlike W3C services, each OGC service implements a specific standard designed to

handle a certain type of data (e.g., a WFS expects vector data, whereas a WCS expects

coverage data55),

• W3C services use only XML and SOAP for communication, while OGC services may

use either KVP encoding and HTTP GET, or XML and HTTP POST for interaction, and

• OGC services can return binary data as well as XML documents, whereas W3C services

primarily rely on pure XML response documents56 [196,197].

54 For example, workflow management systems such as Taverna (http://www.taverna.org.uk/) and service orchestration engines such as Apache ODE (http://ode.apache.org/) rely on a SOAP interface and a WSDL description to build complex workflows of services and to execute them, whereas OGC WPS must be wrapped in an additional WSDL layer if it is to be combined seamlessly with W3C workflows such as WS-BPEL [194]. Difficulties can arise with this process as the design of OGC WPS assumes some human intervention, whereas W3C services do not [195]. The outcome is that WPS endpoints may not always translate directly, nor completely, to WSDL execution endpoints. 55 Spatial information representing space/time-varying phenomena. 56 However, there are several approaches to extend data handling capabilities of W3C services, thus enabling binary data exchange (e.g., transferring images). See Zhang et al. [199] for a detailed discussion.

In order to have an effective, fully interoperable solution, both W3C and OGC interfaces should be considered when designing spatial services [197]. Such a solution can be implemented by adding SOAP/WSDL support to OGC services, which will facilitate (i) dissemination of spatial resources (i.e., data and functions) to a larger community of users [198], and (ii) the incorporation of W3C standards into OGC services such as WS-Security, WS-Policy, etc. [200].

Table 3.5 lists the summary of challenges and proposed solutions for integrating OGC and W3C services [196–198].

Table 3.5. A summary of challenges and proposed solutions for integrating OGC and W3C services.

Challenge Solution

Data handling A common backend approach through which a W3C service and an OGC service provide two separate gateways (i.e., a W3C and an OGC compliant interface) to the same server (i.e., backend).

Functionality mapping A one-to-many mapping where each data layer of an OGC service is exposed as a W3C service.

Metadata management The use of W3C’s WS-Metadata-Exchange specification, which describes a standard format to encapsulate OGC services’ metadata inside WSDL.

OGC has also been involved in developing recommendations and guidelines for adding

SOAP/WSDL support to existing and future OGC services [198,201]. Currently, OGC services such as WPS 1.0 [84], WFS 2.0 [202], WCS 2.0 [203], and SOS 2.0 [204] support SOAP and

WSDL protocols. This facilitates discovery of and access to spatial resources and enables seamless integration of W3C and OGC services into service chains.

In order to implement the VGI framework, this research deploys spatial data and analysis tools using OGC standards. Moreover, it has adopted solutions listed in Table 3.5, the OGC

guidelines, and best practices to add SOAP/WSDL support to increase sharing and reusability of data and services.

3.6.2 Service Development Strategy

This section presents a technical architecture that allows the integration of VGI framework’s components based on the design principles discussed above (Research Objective 4). The proposed architecture (Figure 3.7) exploits SOA design and delivery approaches and SDI principles and provides a “reference” framework for building service-oriented applications, for example disaster management platforms. This means that the architecture can be adapted to build a platform based on different use case requirements. For example, here the service and data layers are used to build the VGI framework, since the focus of this work is to contribute to the development of the backend infrastructure57 of disaster management platforms. However, one can further develop this work and implement a web-based portal where users can register and contribute to disaster response operations by adding their observations to the system or rating others’ contributions – these components are classified as frontend modules58 (presentation and application layers).

The data layer is focused on databases, metadata registries for services, and remote data services.

Within the VGI framework, the VGI Repository uses spatial databases such as

PostgreSQL/PostGIS to store and manage VGI and services’ metadata.

57 It refers to server-side components such as service layer and data layer. 58 It refers to client-side components that end-users interact with directly.

Figure 3.7. A reference architecture incorporating proposed VGI framework components (highlighted elements are the focus of this work). The service layer contains a set of web services that provides capabilities to search, access, and analyze spatial data. The web services are grouped based on SDI service types:

• Discovery service (e.g., CSW), to search and provide access to available spatial data and

services,

• Download service (e.g., WFS or WCS), to access spatial data at the geographic feature

level in vector formats such as GML, KML, or GeoJSON,

• View service (e.g., WMS), to visualize data in map form, and

• Processing service (e.g., WPS), to execute statistical and geo-computational models.

These services are accessible through standard OGC and W3C service interfaces. During platform implementation three service types, including discovery, download, and processing, are used to develop the system components, as outlined below.

3.6.2.1 VGI Broker Services

The VGI Broker module is developed as a translucent processing workflow. It is implemented using the WPS 1.0.0 standard, and, based on the number of brokers, consists of several service instances (i.e., a WPS instance per broker). A WPS instance (i.e., WPS Manager) is used to manage the interactions between clients and the brokers. Since data collection is a long-duration process, an asynchronous communication pattern is used to implement the request-response mechanism. The client can initiate the process by sending a query, containing the broker types

(i.e., the name of social media services) and a set of keywords or hashtags, to the Manager WPS instance (WPS 1). The Manager WPS then translates the query to social media-specific queries and invokes the brokers. Each broker then runs independently to search a social media platform to find, retrieve, and store VGI data in the VGI Repository.

Figure 3.8. The VGI Broker as WPS instances. 3.6.2.2 VGI QC Services

The VGI QC module, also implemented as a set of WPS services, performs quality assessment through chaining of WPS instances in a multi-step pattern, i.e., a workflow (see Figure 3.9). As discussed in Section 2.1.1.2, a translucent chaining method is used to handle the interactions between services in the workflow. For example, as shown in Figure 3.9, WPS 1, WPS 2, and

WPS 6 act as management services. A service instance is implemented for each of the five quality metrics (i.e., PNS, TNS, SSS, CRS, and CS). Since geoparsing is a main step in the evaluation of PNS (see Section 3.3.1), place name extraction and geocoding functions are also developed as WPS instances. Likewise, text classification and similarity assessment operations are implemented as individual WPS services and incorporated into the semantic similarity evaluation workflow. Note that it was chosen to create individual services for specific tasks (i.e., fine-grained services) to improve flexibility and reusability of system components as discussed in Section 3.6.1.1. Finally, another service instance is used to calculate the overall quality score

(QS, see Section 3.3.6).

100

In order to reduce the amount of data transferred between services, a mixed data interaction pattern is used where data is passed by reference between the QS service (WPS 1) and the five quality metric services (WPS 2, WPS 5, WPS 6, WPS 9, and WPS 10), and by value between the services deployed in geoparsing and content analysis workflows. To realize data referencing, all the WPS services involved have access to a common data store, i.e., the VGI Repository.

The following steps describe the quality control procedure (see Figure 3.9):

• The client can begin the quality assessment process by sending a query containing the

name of the social media platform (e.g., Twitter) and the message ID (e.g., tweet ID),

which is called the “original query” hereafter, to the QS instance (WPS 1), which then

passes the request to the PNS service (WPS 2).

• The PNS service retrieves the message from the database and sends it to the WPS 3 to

extract the place names.

• Once the WPS 3 returned the place names to the PNS service, it sends them to the

Geocoding service (WPS 4) to assign geographical coordinates to them.

• The PNS service then calculates the positional nearness score and sends it back to the QS

WPS along with the place names and their associated geographical coordinates to be

stored in the database.

• Afterwards, the QS WPS sends the original query to the TNS service (WPS 5) to

calculate the temporal nearness score. The TNS WPS retrieves the temporal information

from the database, performs the calculations, and returns the results to the QS service.

• Next, the QS WPS sends the original query to the SSS service (WPS 6). The SSS service

retrieves the message from the database and sends it to the Text Classification service

(WPS 7) to determine if the messages are relevant to a particular disaster. 101

• If a message is flagged as irrelevant, a score of zero is assigned to that. Otherwise, it is

sent to the Similarity Assessment service (WPS 8) to evaluate its semantic similarity

based on the reference dictionary. The SSS service then calculates the semantic similarity

score and returns the results to the QS WPS.

• Thereafter, the QS service sends the original query to the CRS service (WPS 9), which in

turn retrieves the MBR information from the database, calculates the score, and sends the

results to the QS service.

• Then, the QS WPS sends the original query to the CS service (WPS 10). After retrieving

credibility information from the database, it calculates the score and returns the results to

the QS WPS.

• Finally, the QS WPS calculates the total quality score and store all the quality score in the

database.

Figure 3.9. The VGI QC processing workflow. The numbers on the arrows represent the order of execution within the workflow.

102

3.6.2.3 VGI Publisher Services

The VGI Publisher has been developed as a download service. This research has adopted the

OGC’s Sensor Web Enablement (SWE) framework [86] to model VGI as observation data collected by humans and publish the observation data as a service. To do so, it has employed the

Sensor Observation Service (SOS) 1.0.0 [91], which is a standard web service interface for providing access to observations (i.e., VGI), and Observations and Measurements (O&M) 1.0.0

[92], which is a data model for encoding observations, to enable distribution of VGI together with the quality metrics. This way one is able to retrieve VGI data based on spatiotemporal parameters and quality-related metrics.

The O&M defines “observation” as an event, whose value (e.g., 25°C) is an estimation of an

“observed property” (e.g., room temperature) of a “feature of interest” (e.g., ENE229), captured using a “procedure” (e.g., temperature sensor). Although the data model was initially designed and tested for in-situ and remote sensor observations, it can be adapted to support different use cases including VGI [28,146]. As a result, heterogeneous VGI is transformed into a standardized model and format. Therefore, the first step is to perform a mapping between the VGI data elements and the components of the O&M data model (see Table 3.6). Next, VGI quality measures should be incorporated into the data model. To do so, it might be necessary to add a new relation to the database model to store quality information. This allows the system to assign the outcomes of the VGI QC procedure (i.e., quality scores) to social media messages.

103

Table 3.6. Mapping the VGI data elements to O&M

O&M Element Typical SOS VGI Publisher

Observation 25°C Social media message (e.g., a tweet)

Observed property Room temperature A disaster event (e.g., Typhoon Ruby)

Feature of interest ENE229 Location of the event (e.g., Philippines)

Procedure Temperature sensor Citizens (e.g., tweeters)

The SOS is further developed not only to publish quality information, besides VGI itself, but also to support data requests based on quality filters59. Based on the SOS standard, “observed property” and “offering” are mandatory request parameters while “procedure” and “feature of interest” are optional to use. Observed property identifies the target phenomena, for example

Typhoon Ruby. Offering or observation offering is equivalent to a “layer” in WMS. It is a collection of observations that are grouped by SOS based on a number of parameters such as the sensor systems that report the observations, time period(s) for which observations may be requested, or geographical region that contains the sensors [91]. In this research, an offering refers to a collection of social media messages from a particular platform (e.g., tweets from

Twitter). Procedure refers to the instrument (i.e., sensor) that collects observations. Here, it indicates who provide the observations (i.e., social media users). Feature of interest indicates the position of the instrument for in-situ sensors, while represents the location of the target for

59 The servers implementing the SOS standard have a built-in support for spatiotemporal filtering. Consequently, SOS servers should be further developed to support additional features such as quality-based filtering.

104

remote sensors. In the context of this research, it refers to the location attached to or described in social media messages as discussed in Section 3.3.1.

The VGI Publisher is designed in such a way that clients are able to search and retrieve information from the database (i.e., VGI Repository) by using the name of the (disaster) event

(e.g., Typhoon Ruby) as the observed property, the social media platform (e.g., Twitter) as the offering, and the geographical area in which the event is happened (e.g., Philippines) as the feature of interest. The results can be further customized by using time, bounding box, and quality filters. Table 3.7 lists the query parameters supported by VGI Publisher.

Table 3.7. The query parameters supported by VGI Publisher to search and filter VGI

Query Parameters Usage Example

Observed property (mandatory) Typhoon Ruby

Offering (mandatory) Twitter

Feature of interest (optional) Philippines

Procedure (optional) http://twitter.com/mepoorazizi

Event time (optional) 2015-12-10T01:40:00/2015-12-12T10:20:00

Bounding box (optional) 120.1,12.3,120.9,13.5

Quality (optional) Level 3

3.6.2.4 VGI Discovery Services

Finally, VGI Discovery is developed as a discovery service. The OGC’s CSW 2.0.2 [81] has been adopted to develop a standard web service to publish metadata and advertise the available

VGI published by the VGI Publisher. To enable support for quality information, the CSW has been extended to return available (VGI) data services based on spatial, temporal, and quality 105

parameters to a client. From a consumer perspective, the VGI Discovery acts as an entry point to the platform. The client performs service discovery using the VGI Discovery service to find available VGI data service instances that provide the desired social media messages (e.g., VGI related to Typhoon Ruby with average quality). The service then returns a list of data services

(i.e., instances published by VGI Publisher), which can directly be accessed by the client to obtain the data (i.e., VGI).

3.7 Implementation

This research employs FOSS tools and libraries, open standards/specifications, and free/open data to implement the proposed framework. Following a service-based development approach, the VGI framework’s components are implemented as OGC-compliant web services. VGI

Repository is implemented using PostgreSQL/PostGIS60 where a database is deployed and a data model is designed for each social media dataset to store and manage the incoming data stream.

VGI Broker is implemented as a set of WPS instances using GeoServer WPS framework61. The number of brokers is dependent upon the number of social media platforms, i.e., a broker per platform. The brokers are implemented as web-accessible applications using social media platforms’ Python APIs. They are then wrapped using GeoServer WPS and exposed as a set of standard WPS instances. The WPS server is also configured to support asynchronous execution and report status information.

VGI QC is implemented as a workflow of services using GeoServer WPS framework. With exception of the text classification model, all the VGI QC components are implemented as web-

60 http://www.postgresql.org/ 61 http://geoserver.org/

106

accessible applications using Python. Besides Python libraries, some third-party tools are also used to develop the modules. For example, PostGIS spatial functions are invoked within the

Python applications to perform geometric computations, such as calculating distances between pairs of points and point-in-polygon tests. In addition, the CLAVIN’s REST server62 is used as the place name extractor engine, and Mapzen Search, an online geocoder, is employed as the geocoder in the geoparsing pipeline. They are then wrapped and exposed as WPS instances using

GeoServer WPS. The text classification model is built using Weka63 and then used in the content analysis pipeline using Weka’s Java API as a GeoServer WPS instance. To implement the VGI

QC pipeline as an iterative process, a cron job64 can be configured. Otherwise, the pipeline should be run on a regular basis to calibrate the metrics, which depends on how often new data is added to the VGI Repository.

For the VGI Publisher, istSOS65 server is employed to enable data access and retrieval based on spatiotemporal and quality attributes. Finally, to develop VGI Discovery, the pycsw66 libraries have been used and extended to enable quality-supported metadata publishing and discovery.

Note that the processing services (i.e., VGI Broker and VGI QC) are implemented in such a way that the logic is written in Python, and GeoServer WPS is used as a wrapper to expose it as a standard web service. In this context, the WPS server acts as a gateway, which enables standard communication with the backend (Python-based) applications. It actually accepts the Execute request, parses the query, and sends it to the corresponding Python-based web application using

62 https://clavin.bericotechnologies.com/ 63 http://www.cs.waikato.ac.nz/ml/weka/ 64 Cron is a time-based job scheduler utility in Unix-based operating systems such as Linux. 65 http://istsos.org/ 66 http://pycsw.org/

107

HTTP handlers. After getting the result, the WPS server prepares it as a standard WPS response and sends it back to the client. The reason of choosing such an approach is the fact that there are multiple servers that implement WPS, with different quantitative and qualitative characteristics such as performance or ease-of-use. The separation of the logic, written as Python web applications, and the interface, developed as WPS instances, enables developers to select a WPS framework that fits a particular application domain or use case best.

3.8 Chapter Summary

This chapter described the methods used to tackle the research objectives (see Section 1.4). First, it introduced a conceptual framework (i.e., VGI framework) for discovery and use of disaster- related VGI, which was developed from a set of functional and architectural requirements. In this regard, and to achieve the first research objective, VGI Broker was developed as a standard method that enables searching and retrieving VGI from multiple social media platforms. The development of VGI QC was an effort to address the second research objective. In this context, a quality control model was developed, a set of quality metrics for VGI quality assessment was identified, and a (geo)processing workflow was implemented to automate the quality control procedure. Then, VGI Publisher and VGI Discovery modules were introduced to achieve the forth research objective. VGI Publisher provides an interoperable mechanism to share “standard

VGI”, and VGI Discovery advertises available VGI data sources in open registries to facilitate

VGI discovery. Afterwards, to accomplish the fifth research objective, this chapter presented a

“reference” technical architecture to incorporate the VGI framework’s components. Following this, it discussed different architectural patterns and described the design strategy used to create, integrate, and publish the system components. Finally, it outlined implementation details and illustrated how a client interacts with the system.

108

CHAPTER FOUR: EXPERIMENTS AND RESULTS

The proposed methods for developing a framework for discovery and use of VGI in disaster management platforms were discussed in the previous chapter. In this context, Chapter 3 presented a conceptual design of the VGI framework, introduced a set of quality control metrics, and described system design principles and development strategies to build the platform. To demonstrate how the VGI framework works and to assess its performance, a real-world application has been developed using social media data related to Typhoon Ruby. The following sections briefly describe the disaster event and implementation details of the VGI framework, and then discuss the results of experiments undertaken.

4.1 The VGI Framework in Action

Typhoon Ruby was a catastrophic typhoon, which ranked as the most intense tropical cyclone of

2014 [205,206]. During the typhoon 18 people lost their lives and significant damage to private and public property and infrastructure (~ $114 million USD) occurred [207]. The typhoon entered the Philippines on December 4, 2014, made first landfall over Eastern Samar on

December 6, 2014 with wind speeds reaching maximum velocity of 175 km/h (kilometer per hour), and exited the country on December 10, 2014 as a tropical storm [208].

User-generated content was collected from Flickr, Google Plus, Instagram, and Twitter using the

VGI Broker between December 4 and December 17, 2014 based on a set of predefined search parameters, hashtags and keywords (see Table 4.1). For this case study, disaster-related messages along with their associated metadata were retrieved, which contains additional information about the messages such as information about the user who posted the message, the time when the message was posted, and the author’s location. A data model was designed for each social media

109

dataset and four PostgreSQL/PostGIS databases were deployed to store and manage the incoming data streams.

Table 4.1 lists the Python APIs used to develop the brokers and the search parameters used to invoke them. It should be noted that the search parameters were chosen through an initial investigation of each social media platforms’ public stream to find a relevant sample of data with relatively little noise.

Table 4.1. Python APIs used to develop the VGI Broker, search parameters, and the number of collected contributions from each social media stream.

Search Parameters (hashtag/keyword) Python API Number of Contributions

Flickr Keyword: Typhoon Hagupit, Typhoon Ruby flickrapi 2.067 69

Hashtag: rubyph, hagupit

Google Plus Keyword: Typhoon Ruby, Typhoon Hagupit googleapiclient 1.3.168 935

Instagram Hashtag: typhoonruby, typhoonhagupit python-instagram 6,022 1.3.069

Twitter Keyword: Typhoon Hagupit, Typhoon Ruby twitter 1.15.070 110,249

Hashtag: rubyph, hagupit

Following data collection, VGI QC processing was undertaken. Several third-party software libraries and tools were employed to preprocess the streaming data and implement the quality evaluation measures of the QC module:

67 https://pypi.python.org/pypi/flickrapi/2.0 68 http://google.github.io/google-api-python-client/ 69 https://github.com/Instagram/python-instagram 70 https://pypi.python.org/pypi/twitter

110

• PostGIS spatial functions were used to perform geometric computations such as

calculating geodesic distance71 between pairs of points in positional nearness score (PNS)

calculation or point-in-polygon tests in cross-referencing score (CRS) calculation.

• The CLAVIN’s REST server was selected as the place name extractor engine over two

other NLP software libraries: (i) Stanford NER [171], and (ii) DBpedia Spotlight [172].

In the selection process, their performances on social media messages were compared,

and their success rate were analyzed when performing the task of place name extraction –

see Section 4.2 for a detailed report.

• The Mapzen’s Search service was chosen as the geocoder in the geoparsing pipeline

through a set of experiments on three online geocoding service providers: Google,

Nominatim72, and Mapzen (see Section 4.3 for details).

• The text classification model was built using Weka data mining software suite and then

used in the content analysis pipeline using Weka Java API as a GeoServer WPS instance.

A mixed use of Random Forest and SVM learning algorithms as an ensemble method

(i.e., Voting) was selected to train the classifier. During the selection process, the

performance of different learning algorithms, including Naïve Bayes, SVM, J48, and

Random Forest, and ensemble methods, including Stacking, Boosting, Bagging, and

Voting, was compared on social media data (see Section 4.4 for details).

After performing the quality control process, a VGI Publisher instance was deployed using istSOS platform to provide access to quality-assessed VGI. The reason behind choosing istSOS

71 It refers to the shortest (great-circle) distance between two points on the surface of a sphere or spheroid. 72 http://wiki.openstreetmap.org/wiki/Nominatim

111

was that it not only provides spatiotemporal filtering but also has a built-in support for quality- based search. Finally, pycsw was employed to setup a VGI Discovery instance. The pycsw server was further developed to enable quality-supported publishing and discovery.

The following sections report the results of a set of experiments undertaken to select the most appropriate place name extractor, geocoder, and text classification method for the VGI QC workflow. Afterwards, they demonstrate how the score calculation is accomplished for each quality metric. Finally, the characteristics of contributions from each social media platform are evaluated.

4.2 Place Name Extractor Selection

Named-entity recognition (NER) software applications follow lookup-based, rule-based, or machine learning based approaches to extract implicit spatial information (e.g., place names) from text [163,167,209] – see Table 4.2 for a brief description of each method.

Table 4.2. A brief description of place name extraction methods

NER Approach Description

Lookup-based methods analyze text and match each candidate word or a set of words against some external data source, such as a gazetteer, an ontology, or a database. The quality of the results is Lookup-based method dependent upon the external reference data. Lookup-based methods are language dependent, unless multilingual data sources are used to promote language independence (see [209] for more details).

Rule-based methods use some heuristics, such as population size or level of spatial hierarchy, and a set of rules, such as regular expressions or context-free grammars. Both heuristics and the rules Rule-based method are usually encoded in a domain-specific language, which results in language dependency. However, they can be developed to handle multilingual text (see [209] for more details).

112

Machine learning-based methods are built using a training (annotated) dataset containing the text associated to the expected set of related places. Then, they run on (unseen or unlabeled) documents Machine Learning-based method to decide on the association of places and documents. The quality of the results is dependent on the quality of annotated text (see [209] for more details).

This section reports results on the experiment undertaken to investigate how well NER tools can identify place names in social media messages. The first system evaluated was the Stanford NER engine, which uses a machine learning based approach, specifically conditional random fields

73(CRFs), to identify named-entities in text [171]. The Stanford NER (v. 3.5.2) interface provided by the NLTK suite74 in Python was used along with a 3-class model that recognizes

“person”, “organization”, and “location” entities. Among the classes, location was only considered.

The second system was DBpedia Spotlight, which employs gazetteer-based lookups to detect

DBpedia entities75 in text and considers quality measures such as “confidence” and “support” to deal with ambiguity [172]. DBpedia Spotlight comes with two implementations: Lucene76-based and statistical. Here the statistical implementation of DBpedia Spotlight 0.7 was used, which is claimed to be faster, more accurate, and easier to configure compared to the Lucene-based implementation [210]. DBpedia Spotlight provides a REST interface that allows users to

73 They are a class of statistical modeling methods that consider context (e.g., neighboring samples) in prediction tasks. For example, in NLP, they predict a sequence of labels using a sequence of input samples. 74 http://www.nltk.org/ 75 The DBpedia project is a community-driven project that aims to extract structured information from Wikipedia and to make this information accessible on the Web. DBpedia entities refer to the named-entities extracted from Wikipedia. 76 https://lucene.apache.org/

113

customize the annotation task through confidence77 and support78 parameters. To tag the entities, confidence and support were set to 0.5 and 0, respectively, and “place”, “geography”, and

“location” classes were chosen from DBpedia, Freebase, and Schema.org knowledge bases.

The last system was CLAVIN, which is a GeoNames79-based geoparser that extracts geographic entities from text and employs heuristics to deal with place name ambiguity (e.g., population size) and fuzzy search to handle misspelled and alternative names. CLAVIN-REST v. 0.3.0 was used, which provides a REST interface to tag “location” entities in text. Table 4.3 summarizes key features of the selected NER systems.

Table 4.3. Key features of the NER systems compared in this study (adapted from Derczynski et al. [168]).

Feature Stanford NER DBpedia Spotlight CLAVIN

Approach Machine Learning- Lookup-based method Machine Learning-based and based method rule-based methods

Language English English English

Domain Newswire Unspecified Unstructured text

Supported number of 3,4, or 7 320 1 classes of entities

Knowledge base CoNLL, ACE DBpedia, Freebase, GeoNames Schema.org

Access (How-to-use) Java Web Service Java / Web Service

License GPLv2 Apache License 2.0 Apache License 2.0

77 Confidence refers to the degree of confidence in performing disambiguation process. 78 Support specifies how prominent the entity is. It is based on the minimum number of hyperlinks a DBpedia resource has to have in other resources in order to be annotated. 79 A worldwide geographical database: http://www.geonames.org/

114

To prepare the test dataset, 1,000 random tweets were manually selected from the tweet dataset that VGI Broker collected during Typhoon Ruby. Then, a data cleaning process was performed to remove URLs, user mentions, and special characters. Finally, place names were manually annotated in tweets to create a baseline to evaluate the place names identified by the selected

NER engines.

Three accuracy measures were used to evaluate the performance of the NER operation [211]:

• Precision (P), which is the percentage of correctly annotated place names to the total

number of annotated place names in the test dataset (Equation (4.1)).

• Recall (R), which is a ratio of the percentage of correctly annotated place names to the

total number of place names in the test dataset (Equation (4.2)).

• F-measure (F1), which is the harmonic mean of precision and recall, used to find an

optimal solution for both precision and recall (Equation (4.3)).

!" (4.1) ! = !" + !"

!" (4.2) ! = !" + !"

2!" (4.3) !1 = ! + !

As shown in Table 4.4, !" (True Positive) indicates the number of correctly labeled place names; !" (False Positive) denotes the number of cases where other entities were incorrectly labeled as place names; and !" (False Negative) implies the number of place names that were not labeled.

115

Table 4.4. A contingency table or confusion matrix used to evaluate the accuracy of classification tasks [211].

Correct Incorrect

Selected True Positive False Positive (labeled) (TP) (FP)

Not selected False Negative True Negative (not labeled) (FN) (TN)

Table 4.5 lists the results of the evaluation. Since CLAVIN achieved a better F1 score, it was selected as the place name extractor. It is apparent that accuracy scores for Stanford NER and

CLAVIN are very close. This is because CLAVIN uses Stanford NER in its place name extraction pipeline.

Table 4.5. The performance of NER engines (best scores are underlined).

NER Engine P R F1

Stanford NER 93% 89% 91%

DBpedia Spotlight 90% 87% 88%

CLAVIN 96% 88% 92%

4.3 Geocoder Selection

The three online geocoding services Google, Mapzen, and Nominatim, explored in this thesis, provide APIs that convert addresses and points of interest (POIs) into geographic coordinates.

However, since they have different characteristics, the geographical coordinates they return may be different (see Table 4.6).

116

Table 4.6. Characteristics of online geocoding services used this study (adapted from Chow et al. [173])

Google Geocoding Service Mapzen Search Nominatim

License Proprietary Open source Open source

Reference TeleAtlas, proprietary OpenStreetMap, GeoNames, OpenStreetMap database datasets OpenAddresses, Quattroshapes

Data type Licensed & limited VGI Open data & VGI Open data & VGI

API limitation* 10 requests/sec 6 requests/sec 1 requests/sec

* This study only examined the free version of the products.

This experiment used the test dataset prepared for evaluation of NER engines in Section 4.2.

Then, the unique place names were manually extracted from the dataset and grouped into two categories:

• Local place names, including 63 Filipino place names.

• Mixed place names, including 38 Filipino and foreign place names.

The local place names were then selected as the test dataset to evaluate the results returned by geocoding services – local place names are referred as place names hereafter. It should be noted that an authoritative dataset (e.g., parcel addresses) has been mostly used as a baseline to evaluate the performance of geocoders [175,177,176,173]. However, using such a dataset does not make sense when dealing with social media messages, since people usually mention place names or POIs in their messages and it is unlikely to see a complete postal address in a tweet, for example.

Two quality metrics were chosen for evaluating the quality of geocoding results [175]:

• Match rate, which indicates the percentage of the number of geocoded place names to the

total number of place names.

117

• Similarity, which measures the pairwise geodesic distances between pairs of geocoded

points.

For the first test (i.e., match rate evaluation), the place names were geocoded and a point-in- polygon test was conducted to check if the returned geographical coordinates fall into the

Philippines boundary. Although geocoders usually return more than one (candidate) match for a given query to deal with place names ambiguity issue, for this experiment the geocoders was configured to return only one result per place name.

The results suggest that Mapzen performed the best, with overall match rate of 100% including

84% of “good” matches, followed by Nominatim, and then Google (see Table 4.7). “Good” matches indicate the proportion of place names correctly geocoded, meaning that the returned geographical coordinates fall inside the Philippines boundary. “Ambiguous” matches represent the proportion of geocoded place names incorrectly geocoded, meaning that the geographical coordinates returned for a Filipino place name fall outside the country’s boundary. “Unmatched” implies the proportion of unsuccessful geocoding cases.

Table 4.7. Performance of the geocoders.

Match Level Match Rate (%)

Google Mapzen Nominatim

Good 77% 84% 79%

Ambiguous 22% 16% 21%

Unmatched 1% 0 0

The results also imply the existence of place name ambiguity error. To minimize this error, the country code can be incorporated into the geocoding requests to restrict the results to a specific

118

country. Following this approach, the country code for the Philippines, “PH”, was used as an input parameter and the performance of geocoders was evaluated. The results confirm the effectiveness of the disambiguation approach because all geocoders performed similarly, with overall match rate of 100%, which means that all the geocoded points fall inside the Philippines boundary.

Table 4.8. Performance of the geocoders when using the country code.

Match Level Match Rate (%)

Google Mapzen Nominatim

Good 100% 100% 100%

Ambiguous 0 0 0

Unmatched 0 0 0

The second test (i.e., similarity evaluation) evaluated similarity among geocoding services on geocoded points. To do so, pairwise geodesic distances between pairs of geocoded points were calculated. The results reveal that distances between the place names geocoded by Mapzen and

Nominatim are all zero. This can be partially explained due to the fact that both services use

OpenStreetMap data as their main data source80. So it can be concluded that the similarity between Mapzen, Nominatim, and Google is dependent more upon the data sources they use rather than the geocoding method used.

The similarity of Google and Nominatim’s database was then assessed. In this respect, the geodesic distances between pairs of geocoded place names were calculated and duplications

80 See the following links for more details: https://mapzen.com/documentation/search/data-sources/, http://wiki.openstreetmap.org/wiki/Nominatim

119

were removed, which resulted in 51 unique rows of data. Given the data, the minimum distance value is around 20.8 m, for a place called “Batangas” and the maximum distance value is around

769.58 km, for a place called “Quezon” (see Figure 4.1).

Figure 4.1. The geocoded place names by Google and Nominatim. Lines connect the points with the same place name. Table 4.9 lists the places names with distances greater than 100 km for positions returned by

Google and Nominatim (~ 21%). An investigation on the data reveals that the place names listed in Table 4.9 have at least one of the following characteristics:

• They are the name of large municipalities or provinces such as “Dolores” and “Quezon”.

120

In this case, geocoders return a point that can be anywhere within the area.

• Misspelled names such as “Philippine”, where there is an “s” missing at the end.

• There are several places with similar names but located at different geographical

locations such as (i) Dolores, Eastern Samar, (ii) Dolores, Quezon, and (iii) Dolores,

Abra.

• The use of general names. For example, the general name “Samar” appears in official

name registers in “Northern Samar”, “Easter Samar”, and “Samar Island” for the

Philippines. “Mindo” and “island of Samar” are other instances of using unofficial place

names in social media messages.

Table 4.9. The geocoded place names that have a distance value of greater than 100 km.

Place name Place name representing Place name representing Distance (km) the coordinates returned the coordinates returned by Google by Nominatim

Quezon Province of Quezon Quezon, Cagayan Valley 769.577

Mindo Tablas Island Mindono Airport, Vigan 579.056

island of Samar Samar Island Samal, Daro del Norte 541.990

Dolores Dolores, Quezon Dolores, Eastern Samar 493.186

Philippine Middle of the Ocean Province of Benguet 416.450

Aurora Aurora, Isabela San Francisco, Quezon 413.188

Legazpi Legazpi City Metro Manila 330.954

East Visayas Visayas Island East Visayas Region 281.194

Mindanao Mindanao Island Mindanao Island 144.313

121

Cavite Cavite City Guimba 133.341

Atimonan River Middle of the Ocean Atimonan River, Atimonan 125.641

Finally, a paired t-test was performed to investigate the similarity of geocoded points returned by

Google and Nominatim. The results reveal that the geocoding services returned equivalent geographical coordinates and there is no significant difference between the longitudes

(! = 0.975, d.f. = 50, ! = 0.334) and latitudes (! = 0.579, d.f. = 50, ! = 0.565) – see Figure

4.2.

Longitude Latitude 126 18

●

● 125

124

123 12 Degree Degree

122

121 ●

●

120 Google Nominatim Google Nominatim Geocoder Geocoder

Figure 4.2. Longitude (right) and latitude (left) of geocoded points returned by Google and Nominatim. In summary, the results of the geocoding experiments suggest that Mapzen is the logical choice to be used as the geocoder, since it performed better in the first test (i.e., match rate). Moreover,

122

from a qualitative perspective, the platform’s quality requirements and business goals require the system components to be open, flexible, and integratable, and Mapzen meets these requirements.

4.4 Text Classifier Selection

In a set of experiments the performance of machine learning methods on text classification tasks was evaluated. In this respect, the Weka software suite was used to compare four different classifiers, i.e., Naïve Bayes, SVM, Random Forest, and J48, to select the classifier that achieves the best performance on social media messages. Table 4.10 lists a brief description of each algorithm.

Table 4.10. A brief description of machine learning algorithms used in this experiment.

Algorithm Description

Naïve Bayes It is a probabilistic classifier based on Bayes’ theorem, which has strong independence assumptions. It assumes that each feature is independent of the rest, that is, the value of a feature has no relation to the value of another feature. The main advantage of Naïve Bayes is that it can process a large number of variables. It is suitable for large-scale prediction of complex and incomplete data. The main potential drawback of this method is that it requires independence of attributes. However, this method is considered to be relatively robust (see [212] for more details).

SVM It is defined over a vector space where the problem is to find a decision surface (i.e., a hyperplane) that best separates the data points in two classes by finding the maximum margin, which is the largest possible distance between instances of different classes. The main advantage of this approach is that it may lower the generalization error, making this model resistant to overfitting in classification tasks. Moreover, it is very effective when working on high- dimensional spaces and when the data is sparse. However, SVM models can be computationally intensive (see [212] for more details).

J48 It is a Java implementation of the C4.5 decision tree algorithm. A decision tree is a hierarchical model composed of a set of decision rules used to predict outcome of a set of input variables. J48 uses an entropy-based measure as the selection criteria to construct a decision tree using a top- down, recursive-splitting technique. Once created, the tree is pruned to minimize overfitting and, therefore, to achieve a better classification accuracy for new data. The main advantage of

123

decision trees is that they are easy to construct and interpret, and are able to model complex relationship between variables. However, they are susceptible to noisy data (see [212] for more details).

Random Forest It is a classifier composed of multiple decision trees. A Random Forest trains each decision tree independently using a randomly selected subset of samples and a subset of variables. Each decision tree predicts a label for each sample. The prediction of a Random Forest is the result of the votes from all decision trees. The advantages of Random Forest are as follows: they handle missing values and unbalanced datasets; they are robust against overfitting; and they run efficiently on large datasets. One of the drawbacks of this method is that for data including categorical variables with different number of levels, Random Forest are biased in favor of those attributes with more levels (see [212] for more details).

This experiment used a set of annotated tweets that Imran et al. [44] collected using the AIDR platform during Typhoon Ruby in 2014. It contains 9,675 tweets that are categorized into two main groups:

• Relevant, including 2,933 tweets (~ 30% of the sample)

• Irrelevant, including 6,742 tweets (~ 70% of the sample)

Relevant tweets are also assigned to categories with respect to their content, including “Request for Help/Needs”, “Humanitarian Aid Provided”, “Infrastructure Damage”, and “Other Relevant

Information”, as described in Section 3.5.1.

Three performance metrics were used to assess the results of text classification:

• Precision (P), which is the percentage of correctly classified messages (i.e., tweets) to all

classified messages.

• Recall (R), which is a ratio of all correctly classified messages to all relevant messages.

• F-measure (F1), which is the harmonic mean of precision and recall, used to find an

optimal solution for both precision and recall.

124

The P, R, and F1 scores are calculated using Equations (4.1), (4.2), and (4.3), respectively, where

!" indicates the number of relevant messages that are correctly classified; !" denotes the number of irrelevant messages that are classified as relevant; and !" is the number of relevant messages that are classified as irrelevant.

Because of the class imbalance in the test dataset (30% relevant vs. 70% irrelevant), the experiment was conducted using two datasets:

• The original dataset, which includes 9,675 tweet samples grouped into five categories: (1)

“Request for Help/Needs” (71 samples), (2) “Infrastructure Damage” (260 samples), (3)

“Humanitarian Aid Provided” (323 samples), (4) “Other Relevant Information” (2,279

samples), and (5) “Not Informative” (6,742 samples).

• A resampled dataset, which contains an equal number of samples (967) in each class.

The k-fold cross-validation technique (k=10) was used to evaluate the performance of classifiers on the original and resampled datasets. In this approach, the dataset is partitioned into k equally sized sets. Each set is used as validation data to test a model that was trained on the remaining k−1 parts. The validation process is then repeated k times and the k results are finally combined to produce a single error estimate.

The results of the experiment suggest that the classifiers performed better on the resampled dataset, except for Naïve Bayes (see Table 4.11). Given the data, SVM and Random Forest outperformed others81, when using the resampled dataset, and achieved a precision, recall, and

F1 score of 85%.

81 These findings are consistent with previous analyses on the performance of classification algorithms, which suggest that Random Forest and SVM are the best family of classifiers [213]. 125

Table 4.11. The performance of machine learning algorithms on the original and resampled datasets (high precision, recall, and F1 scores are underlined).

Original Data Resampled Data

Performance Naïve SVM J48 Random Naïve SVM J48 Random Metric Bayes Forest Bayes Forest

Precision 74% 71% 68% 70% 66% 85% 79% 85%

Recall 65% 73% 71% 73% 65% 85% 79% 85%

F-Measure 68% 69% 69% 70% 65% 85% 79% 85%

In the next experiment, since both SVM and Random Forest achieved the best scores, their performance was evaluated as ensemble classifiers. To do so, they were combined using

Bagging, Boosting, Stacking, and Voting methods [214]. These methods create multiple classifiers, compare the outputs, and combine them into a single composite classifier, based on different rules such as weighted average, major voting, or maximum probability, to achieve a higher accuracy. The resampled dataset was used to train the classifiers and 10-fold cross- validation was employed for error estimation.

As listed in Table 4.12, the Voting approach achieved slightly better results compared to others with an F1 score of 86%82, so it is selected as the text classifier.

82 A one-way Analysis of Variance (ANOVA) indicates that, given the data, there is no significant difference between the classifiers’ F1 score, ! 6,63 = 0.297, ! = 0.936.

126

Table 4.12. The performance of ensemble methods (high scores are underlined).

P R F1

Random Forest 84.6% 85.1% 84.8%

SVM 85.5% 85.5% 85.5%

Boosting (Random Forest) 84.7% 85.2% 84.9%

Bagging (Random Forest) 84.7% 85.2% 84.9%

Bagging (SVM) 84% 83.5% 83.9%

Stacking 85.8% 85.9% 85.8%

Voting 85.9% 86.1% 86%

4.5 Evaluation of VGI Quality Score Calculation and Metrics

Given a tweet example, this section first demonstrates how quality scores are calculated using current implementation of the VGI QC model. Then, based on the Typhoon Ruby case study, it evaluates the characteristics of contributions from each dataset, including Flickr, Google Plus,

Instagram, and Twitter, and presents results for quality control scores. For simpler presentation, values of the five quality scores were classified into Levels 1 to 4 representing values between

0.00 and 0.25 (Level 1), 0.25 and 0.50 (Level 2), 0.50 and 0.75 (Level 3), and 0.75 and 1.00

(Level 4), respectively – with 0 being a bad score value and 1 being a good score. Similarly, the total QS values, ranging from 0 to 5, were classified as Level 1 to Level 5 corresponding to values between 0 and 5 with a step value of 1. Consequently, a higher valued level (score) corresponds to a higher quality contribution, which is assumed to have better utility for disaster management.

127

4.5.1 How VGI QC Works

Using an example tweet published during Typhoon Ruby in the Philippines, this section presents the corresponding score result for each quality metric, with their equations given in Chapter 3.

Thereby the implementation of the metrics utilizes the parsing, geocoding, and classification algorithms selected above.

Figure 4.3. A tweet sent during Typhoon Ruby in the Philippines during December 2014. Positional Nearness Score (PNS) – The example tweet in Figure 4.3 was sent at 7:04 PM of

December 8, 2014 from Ormoc City, Eastern Visayas, Philippines with a recorded longitude of

124.608714° and a latitude of 11.009535° (i.e., explicit location). The geoparser also identified a place name (i.e., implicit location) in the message, i.e., “Biliran”. The coordinates returned by the geocoding algorithm indicate that the implicit location is inside the Philippines (longitude =

124.494202°, latitude = 11.587409°). The centroid of all the tweets collected is located at the longitude of 104.890883° and Latitude of 14.848960° and an approximate standard distance deviation (!) of 1,316 km was observed. Since the distance between the mean of explicit and implicit coordinates (longitude = 124.551458°, latitude = 11.298472°) and the centroid of all tweets is around 2,167.20 km, the PNS value for the given tweet is 0.61 (i.e., Level 3). 128

Temporal Nearness Score (TNS) – Since the event occurred between the December 4th and 10th of 2014 and the creation time of the tweet falls into that period, the TNS value is 1 (i.e., Level 4).

Semantic Similarity Score (SSS) – The text classification model classified the tweet as a relevant message belonging to “Infrastructure Damage” category. After performing the preprocessing step, the similarity assessment algorithm compares each word of the tweet to the reference dictionary and calculates a maximum score for each one of them (see Table 4.13). Now, there is a need to estimate the value of ! to calculate the score. Count data usually follows a Poisson distribution [193]. So, the data was analyzed using the Poisson regression model (! ≪ 2!!!").

However, overdispersion is often observed when using simple parametric models (e.g., the

Poisson regression model), because there is usually a greater variability in data than is implied by the model [193]. An overdispersion test confirmed that the data is overdispersed (! = 0.117) and, consequently, a negative binomial distribution was used instead (! = 4.627687 !" =

95% , ! = 7.16). The mean of the count distribution (!) is considered as !. Given ! as

4.627687, the similarity score is 1.27 for the given tweet. However, this value should be normalized to keep all the scores between 0 and 1. Considering the minimum and maximum scores as 0.0 and 2.60, the SSS is 0.49 after normalization (i.e., Level 2).

Table 4.13. The extracted words from the tweet and maximum word similarity scores.

Word in Tweet Maximum Word Similarity Score

Biliran 0

displacement 0.670

Flooding 1

Number 0.991

129

damaged 0.586

houses 0.998

Aftermath 0.978

Typhoon 0.670

Hagupit 0

Commisaid 0

Cross-referencing Score (CRS) – Given the mean of explicit and implicit coordinates of the tweet, the CRS algorithm indicates that it falls within all contributions’ bounding boxes – including Flickr’s, Google Plus’s, Instagram’s, and Twitter’s. So, the CRS value is 1 for the given tweet (i.e., Level 4).

Credibility Score (CS) – As shown in Figure 4.3, the message has been retweeted three times and favorited once. Considering the values in Table 4.14, the CS is zero for the given tweet (i.e.,

Level 1).

Table 4.14. The credibility factors for the given tweet and their maximum values among all contributions.

Credibility Factors Followers’ Retweets’ Verification of Favorites’ Count Count Tweeter Count

Values for the tweet 531 3 0 1

Maximum value of all 21,433,505 8,006 1 1,441 tweets

Finally, having all the scores calculated, the total quality score for the given tweet is 3.08 (i.e.,

Level 4).

130

4.5.2 Characteristics of Collected Contributions for Positional Nearness

A total number of 117,272 contributions were collected from the four different data sources:

Flickr, Google Plus, Instagram, and Twitter, that referenced Typhoon Ruby [215]. As listed in

Table 4.15, of these contributions only 2,440 (~ 2%) were geo-tagged with explicit spatial information (i.e., geographical coordinates). It appears that for this case study there were only two explicitly geo-tagged contributions from both Flickr and Google Plus. Hence, Google Plus provided the lowest proportion of explicitly geo-tagged content (~ 0.2%), followed by Twitter (~

0.6%), then Flickr (~ 3.0%), with Instagram providing a significantly83 greater proportion of geo- tagged contributions (~ 33.1%). This can be partially explained by the fact that Instagram’s users can only publish photos and videos via their mobile phones, whereby Instagram may deduce spatial information from the devices’ GPS or IP addresses.

However, after performing the geoparsing workflow, it was apparent that more than 39% of the contributions (43,599) convey implicit spatial information (i.e., place names). Of the four social media streams, Google Plus had the highest proportion of implicitly geo-referenced content (~

48%), followed by Twitter (~ 38%), then Flickr (~ 30%), with Instagram providing a significantly84 lower proportion of geo-referenced content (~ 12%). This might be because of the way that Google Plus functions. Unlike the other social media platforms, it allows users to share longer textual content, which may increase the likelihood of having place names in text.

83 A 2-sample test for equality of proportions was carried out indicating a significant difference between the two tested proportions !! = 26,757, d.f. = 1, ! ≪ 0.001 . In the first proportion, the data streams of Twitter, Flickr, and Google Plus were conflated, since they provided fewer than five contributions with coordinates, which results in an unreliable solution. There was also no significant difference between the count proportions for those services. 84 A pairwise comparison of proportions indicates that there is a significant difference between Instagram and three other social media streams, !! = 1,747.6, d.f. = 3, ! ≪ 0.001.

131

As described in the previous chapter (Section 3.3.1), the spatial information component (i.e., geographical location) of social media messages is estimated based on both explicit and implicit spatial information. The results of the positional nearness workflow reveal that more than 41% of social media messages (45,753) have an estimated geographical location. It appears that Google

Plus provided the highest proportion of geo-referenced content (~ 48%), followed by Instagram

(~ 41%), Twitter (~ 39%), and Flickr (~ 33%).

Table 4.15. The number of geo-referenced contributions from each social media stream.

Flickr Google Plus Instagram Twitter

Number of explicitly geo- 2 2 1,996 708 referenced contributions

Number of implicitly geo- 20 450 729 42,400 referenced contributions

Number of geo-referenced 22 452 2,479 42,880 contributions as a result of the geoparsing workflow

Total number of contributions 66 935 6,022 110,249

Table 4.16 shows the number of contributions for each positional nearness score (PNS) according to their source. PNS Level 1 contributions are not implicitly or explicitly geo- referenced. Figure 4.4 shows the spatial distribution of all the retrieved data from social media, while Figure 4.5 depicts the spatial distribution of social media distributions with an estimated location within the Philippines boundaries.

132

Figure 4.4. Spatial distribution of all social media contributions for Typhoon Ruby (each circle represents a certain number of social media contributions).

Figure 4.5. Spatial distribution of social media contributions with estimated location in the Philippines. 133

Table 4.16. Counts of Positional Nearness Score (PNS) values for each media stream.

Flickr Google Plus Instagram Twitter

PNS Level 1* 44 484 3,631 67,650

Level 2 4 67 19 8

Level 3 12 89 1 38,614

Level 4 6 295 2,371 3,977

* Level 1 contributions are not (implicitly or explicitly) geo-referenced.

4.5.3 Characteristics of Collected Contributions for Temporal Nearness

During the data collection period, social media messages were streaming at an average rate of

2,129 contributions per day. As shown in Figure 4.6, of the contributions collected, most were published between December 4, 2014 and December 10, 2014, when the event itself took place in the Philippines. It is evident from Figure 4.6 that prior to the arrival of the typhoon in the

Philippines, there seems to exist already a considerable volume of contributions on Twitter. This means also that any of these data from Twitter were not collected, since the VGI Broker was deployed on December 4th.

The temporal nearness evaluation indicates that more than 76% of all collected contributions were streamed during the event, which were assigned the highest temporal nearness score (i.e.,

Level 4). Level 1 contributions were all published several days before or after the event. Of the four contributing streams, Flickr had the highest proportion of Level 4 TNS contributions at

98.5%, followed by Instagram at 82.4%, then Twitter at 76.6%, and finally Google Plus at 56.9%

(see Table 4.17). A pairwise comparison of proportions indicates that there is a significant difference in temporal performance between social media streams, !! = 331.6, d. f. = 3, ! ≪

0.001. For instance, Instagram photo and video contributions were posted, predominantly,

134

during the event, whereas Twitter contributions were also published prior to the typhoon as forms of warnings, and then post typhoon, as messages of hope to those affected.

Figure 4.6. Temporal dynamics of users’ contributions on social media. Note that, as shown in Figure 4.6, there is a spike on the number of tweets on December 12,

2014. This is because of the fact that Typhoon Ruby dissipated on that day and a series of tweets related to the recovery phase was published. A closer look into the data revealed that around

80% of the tweets (~ 4,500) published on December 12th were about donations, which misclassified as “Not Informative” by the semantic similarity algorithm (see Figure 4.7).

135

Table 4.17. Counts of Temporal Nearness Score (TNS) values for each dataset.

Flickr Google Plus Instagram Twitter

TNS Level 1 0 65 29 1,746

Level 2 1 338 1,033 24,041

Level 385 0 0 0 0

Level 4 65 532 4,960 84,462

4.5.4 Characteristics of Collected Contributions for Semantic Similarity

A post-event analysis of social media messages reveals that the most frequently employed hashtags/keywords used to describe the event were rubyph, hagupit, typhoonruby, typhoonhagupit, and philippines. To evaluate the semantic similarity of contributions to Typhoon

Ruby, a two-step verification mechanism was implemented. First, the crisis-related contributions

(i.e., the “relevant” messages) are identified using the text classifier and then compared to a reference dictionary of crisis-related words to calculate the semantic similarity score (SSS).

Figure 4.7 and Figure 4.8 show the spatial and temporal dynamics of “relevant” social media contributions. The proportion of relevant contributions was around 65% during the event.

Contributions classified as Level 1 in Table 4.18 matched few of the reference words exactly – they actually had partial hashtag/keyword matches, as this was necessary to enter the event database. Level 2 contributions had at least one match with a reference-word.

Twitter’s content was semantically more similar to the reference dictionary since 14% of the collected data was classified as Level 3 or 4. This is, however, not significantly86 more than the

85 Note that given the model used (see Equation (3.4)) Level 3 is not possible.

136

contributions for the data retrieved from Flickr (~ 9.1% achieved Level 3 or 4). Only few

Instagram data contained enough similar words to the reference dictionary to achieve Level 3 or

4 (~ 0.3%), which also holds for Google Plus contributions (~ 0.2%).

20,000

All Contributions "Relevant" Contributions

15,000

10,000 Contributionsper day

5000

0 Dec 4 Dec 5 Dec 6 Dec 7 Dec 8 Dec 9 Dec 10 Dec 11 Dec 12 Dec 13 Dec 14 Dec 15 Dec 16 Dec 17 2014

Figure 4.7. Temporal dynamics of all social media contributions and relevant contributions.

86 A 4-sample test for equality of proportions along with a pairwise comparison was used to determine if there were significant differences in the semantic quality of contributions between social media streams, χ! = 1,086.8, d. f. = 3, p ≪ 0.001.

137

Figure 4.8. Spatial distribution of relevant contributions (see Figure 4.4 for spatial distribution of all contributions).

Table 4.18. Counts of Semantic Similarity Score (SSS) values of each dataset.

Flickr Google Plus Instagram Twitter

SSS Level 1 21 914 5,993 51,926

Level 2 39 18 19 42,954

Level 3 0 2 5 15,072

Level 4 6 1 5 297

4.5.5 Characteristics of Collected Contributions for Cross-referencing

The cross-referencing process focused on the spatial component of the contributions; measuring how many times each contribution did fall within the MBR (minimum-bounding rectangle) of

138

each social media stream. Contributions that were assigned to Level 1 had either no spatial reference or did not fall in a bounding box other than its own. As can be seen from Figure 4.9, more than 39% of contributions were located in the bounding box of at least three other datasets.

Google Plus had the highest proportion of Levels 3 or 4 contributions at 46.7%, followed by

Twitter at 38.5%, then Instagram at 38.1%, and finally Flickr at 33.3% (see Table 4.19). A 4- sample test for equality of proportions reports that there is a significant difference in cross- referencing across the four social media streams !! = 27.8, d. f. = 3, ! ≪ 0.001. A pairwise comparison of proportions indicates that Google Plus contributions are more likely to be within, or near the geographic intersection of the social media data sources compared to the data collected from Twitter and Instagram.

139

Table 4.19. Counts of Cross-Referencing Score (CRS) values of each dataset.

Flickr Google Plus Instagram Twitter

CRS Level 1 44 483 3,555 67,373

Level 2 0 15 170 452

Level 3 4 75 597 5,486

Level 4 18 362 1,700 36,938

4.5.6 Characteristics of Collected Contributions for Credibility

Given the different characteristics of the social media platforms (e.g., data model), a different set of factors for credibility assessment was selected for each media stream as described in Section

3.3.5. Depending on the social media source, Level 1 contributions have no shares, likes, comments, or followers. As indicated in Table 4.20 only very few contributions obtained a high credibility score. A 2-sample test for equality of proportions87 reports that there is a significant difference in credibility across the four social media streams, !! = 112.4, d.f. = 1, ! ≪ 0.001.

The data suggests that Twitter offers more credible information.

87 Flickr and Google Plus and Instagram were conflated, as were Levels 2 to 4, to ensure that the !! approximation would be correct.

140

Table 4.20. Counts of Credibility Score (CS) values of each dataset.

Flickr Google Plus Instagram Twitter

CS Level 1 64 926 6,018 108,060

Level 2 0 6 1 2,183

Level 3 0 2 2 6

Level 4 2 1 1 0

4.5.7 Overall Scores per Social Media Stream

Table 4.21 presents a summary of the final quality score calculations for each dataset. Given the data, most of the content published on the social media services fell in Levels 2 and 4. Few contributions were identified as Level 5, the highest quality. Indeed contributions without any

(explicit or implicit) geographical reference can only reach a quality level of 3, since they do not receive the score for positional nearness and cross-referencing (i.e., PNS = 0, CRS = 0). A 4- sample test for equality of proportions indicates that there are significant differences in the quality of each social media data stream, !! = 128.9, d.f. = 3, ! ≪ 0.001. Given the data, it appears that Instagram provides the highest quality data88 with 22.9% of the data falling in

Levels 4 or 5, followed by Flickr (17.3%), Google Plus (17.2%), and Twitter (9.1%).

88 A pairwise comparison of proportions suggests that the proportion of high-quality Instagram contributions is significantly higher than the data retrieved from Google Plus, Flickr, and Twitter.

141

Table 4.21. Counts of total Quality Score (QS) values of each dataset.

Flickr Google Plus Instagram Twitter

QS Level 1 0 257 611 17,213

Level 2 43 269 3,046 50,634

Level 3 17 248 986 23,374

Level 4 0 161 1,377 19,028

Level 5 6 0 2 0

4.6 Chapter Summary

This chapter reported the use of the VGI framework in a real-world case study. The VGI framework was deployed during the Typhoon Ruby in December 2014 and its performance was evaluated with a focus on the VGI QC and quality metrics. In this context, the quality of social media messages collected from Flickr, Google Plus, Instagram, and Twitter were assessed in terms of spatial, temporal, semantic, and credibility factors. The results suggest that Instargam provides the largest proportion of high quality data during the event among others.

142

CHAPTER FIVE: EVALUATION AND DISCUSSION

The previous chapter reported the development of an application prototype, the VGI framework, and analyzed the use of VGI in a real-world use case. Using the case study, this chapter aims to evaluate and discuss the usefulness of the methods proposed in this research (Research Objective

5). In this regard, first, it evaluates the quality assessment model and metrics (VGI QC). This is followed by an evaluation of the VGI framework in terms of the quality requirements and business goals set out in Section 3.1.

5.1 Evaluation of Quality Control Metrics

The current set of quality assessment metrics are to be regarded as an experimental set with their primary purpose being to demonstrate the utility of the proposed framework for quality-based

VGI data retrieval. In this section, the Typhoon Ruby case study is used to evaluate the metrics and discuss which ones may work, and which require rework or refinement.

5.1.1 Positional Nearness

Although the Philippines was the center of the crisis, there was a substantial amount of data streaming from other parts of the world (see Section 4.5.2). These (global) contributions have been referred to as a response to a “cyber event” by social media [164]. This means that they are not reports of the crisis itself, but rather, reactions to its coverage in social media. Therefore, such contributions should receive lower positional nearness scores as they may have a negative effect on attempts to localize the crisis using social media messages [164]. However, although global contributions may point to a location that is far away from a disaster-affected area89, they

89 For example, in our case study, the mean center of all “explicitly” geotagged contributions is in Myanmar (97.98608° E 15.66537° N), 500 km southeast of Naypyitaw (the capital), which is more than 2,000 km away from the center of the crisis.

143

may carry geographically meaningful information (e.g., place names) about the ongoing event.

For this reason, this research used both explicit and implicit spatial information to estimate the positional nearness score (PNS) – see Chapter 3 (Section 3.3.1). An analysis on the data revealed that incorporation of implicit spatial information, also known as place names, increased the number of “geo-referenced” contributions by around 95% (from 2,440 to 45,833). As a result, the proportion of contributions with a high PNS score (Levels 3 and 4) increased from 1.6% to

38.7%90.

It was also found that more than 93% of Levels 3 and 4 contributions fall within a 100 km buffer from the typhoon track or “footprint”. This confirms that contributions with a high positional nearness score were either posted from the vicinity of the disaster-affected region or report a location around that area. For example, the average distance to the typhoon track is around 60 km for (PNS) Levels 3 and 4 contributions. Further investigations indicate that contributions that are spatially closer to the affected area (higher PNS) have higher total quality scores. For instance, the average distance to the typhoon track for high quality contributions (Levels 3-5 of

Total QS) is around 75 km.

Instagram provided the highest proportion of explicitly geotagged contributions (~ 33.1%). This might be because Instagram contributions generally contain photos and videos related to the typhoon – not text messages as with Twitter. Hence, Instagram contributors are more likely to have created the photos or videos in the vicinity of the event, whereas the other contributions can in reality be created from anywhere in the world. Google Plus had the highest proportion of

90 In an experiment, the PNS was calculated based only explicit, and both explicit and implicit spatial information. A 2-sample test for equality of proportions was carried out indicating a significant difference between the two proportions !! = 40,170, d.f. = 1, ! ≪ 0.001 . Social media contributions were conflated, as were Levels 3 to 4, to ensure that the !! approximation would be correct.

144

implicitly geo-referenced content (~ 48%). This might be because of the way that Google Plus functions as a social networking website. Compared to Twitter, a micro-blogging service with a

140-character limit, and Instagram and Flickr, both photo/video sharing services, Google Plus allows users to post both short and medium length textual content directly to their stream, and share content (e.g., photos, videos, or news-articles) from third-party websites. This increases the likelihood of having place names in text.

However, there are still two problems that need further investigation:

• First, it is difficult to apply this metric for events with a global response as these data

sources will be biased by the coordinate system used to store geographic location, and

may generate geographic (arithmetic) centers that are far away from the crisis. Other

metrics such as the center of mean distance, or the center of intensity may be more

appropriate.

• Second, there are very few Google Plus and Flickr contributions with an “explicit” spatial

reference (i.e., attached geographical coordinates). It would be desirable that each media

stream has a substantial number of geo-tagged contributions for both the nearness and

cross-referencing metrics. This research proposed a method to extract “implicit” spatial

information (i.e., place names or POIs) to enrich the spatial component of users’

contributions.

However, estimating the location of social media messages is still a challenging task since explicit and implicit spatial information may point to different, and even very distant,

145

geographical locations91. The question then arises as to what type of location information is more relevant when studying a disaster. Although the positional nearness algorithm introduced in this work favors locations within the Philippines boundary in the location estimation process, the estimation approach, in the end, depends on the spatial extent of the particular event and the use case of the VGI, which requires the model to be calibrated accordingly. For example, if the main goal is to localize an event and perform damage analysis, then more weight should be assigned to the locations within the area of interest. However, if the purpose is to spread information for increasing situational awareness, then I believe it does not really matter if a contribution is posted from the center of a crisis or a distant location (e.g. an emergency control center in the country’s capital), as far as it is “informative”.

5.1.2 Temporal Nearness

One of the reasons for having contributions with lower temporal nearness scores is that the event monitoring started two days before the typhoon hit the Philippines and data collection continued until after it exited the country. This means that the datasets contain social media contributions from different phases of the disaster, preparedness through response, and then recovery. The differences in temporal patterns between social networks, as seen in Figure 4.6, may have two different reasons:

• Because contributors prefer to use different social media services during different phases

of an event, and

• Because of the different spatial distribution patterns of each of the social media datasets

91 The tweets were classified into local and global tweets and distances between their implicit and explicit location were calculated. The average distance for local contributions is 35 times smaller than for global contributions (310 km vs. 10,596 km).

146

due to worldwide responses to the “cyber event”.

This latter point suggests that the metric should be combined with positional nearness score to identify the spatiotemporal center, or moving center, of an event. An additional question to consider is: what timespan should be used to best calculate the scores? In the case study, data were evaluated in terms of “days” since the end of the event. This temporal unit seemed to work adequately as the typhoon was moving slowly given the spatial extent of the event area.

However, other types of events, like an earthquake or an accident, of smaller spatial scale, may require adjustment of the temporal unit.

5.1.3 Semantic Similarity

The semantic similarity assessment suggested that contributions from Twitter had the highest level of similarity to the reference dictionary with 14% of tweets classified as Level 3 or 4. This can partially be explained by the fact that the text classifier used for the semantic similarity assessment was trained using a sample of tweets. Hence, Twitter may be favored by the semantic similarity calculation used. Moreover, contributions from Flickr and Instagram are photos and videos, which may contain a short length of text as photo/video title or descriptions for example.

Given the data, Flickr contributions had the lowest volume of characters with an average of 43 characters approximately. Hence, given the content types and character volume per contribution,

Flickr may be underestimated by the semantic similarity algorithm used. This indicates the need to weigh contributions for each social media stream and also for each media type (image, text, etc.).

147

Instagram had the highest proportion of “Not Informative” contributions (~ 89%)92, followed by

Twitter (37%), then Flickr (20%), and finally Google Plus (19%). This is reasoned by the fact that a tweet sample was used to train and build the text classification model while Instagram actually is a photo-based and not text-based platform. Moreover, short length and unstructured nature of social media messages poses a challenge to the text classification process [50]. This issue resulted in a misclassification of more than 80% of the tweets posted on December 12th

2014, as discussed in Section 4.5.3. Hence, different (VGI-specific) classifiers need to be built and evaluated using annotated data from different social media platforms to further investigate the issue.

An issue that was observed with the current data is that sometimes words similar to the reference-words were contained in the message but with different or incorrect spellings.

Consequently, a different word-matching approach can be introduced that measures similarity using character-based distance for example, such as the Hamming Distance [216].

Finally, a further challenge is to interpret different languages and terminologies in contributions, which is crucial when a disaster happens in a multilingual country or where people use vernacular and alternative terms to describe an event [217]. A promising approach to minimize this issue is to use a data dictionary that covers different languages or terms in a given context, similar to CrisisLex, which was used in the semantic similarity pipeline.

92 A 4-sample test for equality of proportions along with a pairwise comparison indicates that Instagram provided the highest proportion of irrelevant content, !! = 2,164, d.f. = 3, ! ≪ 0.001.

148

5.1.4 Cross-referencing

Incorporation of implicit spatial information significantly93 improved the proportion of contributions with a high level of CRS (Levels 3 and 4) – from 0.1% to more than 38%.

However, there are some situations that raise some concern about how the concept of nearness should be defined. For example, in this research, it was chosen to use an MBR (minimum- bounding rectangle), but with worldwide social media contributions, such as Twitter and

Instagram, this may be of little value. Additionally, if there is few data obtained from a selected social media stream, the bounding boxes may potentially be very small, or in a different geographic location than the event location. In such situations, the metric could be improved, or replaced by a density-based metric, e.g. a standard deviational ellipse [218], a 95% minimum convex polygon [219], or perhaps a core home range as used in animal home range estimation

[220].

5.1.5 Credibility

This metric is fairly straightforward and allows for weighing each contribution with its peers.

However, it is interesting to observe that very few contributions have a credibility level of three or higher. This indicates two things:

• Either the credibility criteria are too demanding, and it may be difficult to develop civic

credibility within the context of a crisis because of bandwagon effects [221],

• Or the media’s hierarchical network structure (i.e., network centrality and connectivity

[222]) does not allow a more equal distribution of contributions among the credibility

93 A 2-sample test for equality of proportions suggests that there is a significant difference in CRS between social media platforms, !! = 44,667, d.f. = 1, ! ≪ 0.001. Contributions were conflated, as were Levels 3 to 4, to ensure that the !! approximation would be correct.

149

levels.

A problem that was observed is that the maximum values for credibility factors reached, such as the number of followers of a user or the number of likes of a photo, are not usually given to contributions by ordinary citizens’ contributions, but rather to messages posted by media outlets, public figures, and celebrities. For example, a well-known media outlet has 21,433,505 followers on Twitter while the average number of followers was around 6,462. Similarly, a photo posted by a celebrity on Instagram received 11,309 likes, which is the maximum value among

Instagram’s contributions, while the average number of likes was around 17.

This raises some questions about the usefulness of the credibility factors and their relationship to the final credibility score. For this reason, a Pearson’s product-moment correlation coefficient was computed to assess the relationship. The results indicate that there are strong positive correlations between the factors and credibility score for each media stream, except for Twitter’s favorite count and follower count, which exhibit only small positive correlations with the CS

(see Table 5.1). Therefore, increases in credibility scores were correlated with increases in level of credibility factors of contributions. This finding confirms the usefulness of the selected credibility metrics. However, modifications can be applied to the model to improve the CS estimation. For instance, social media messages can be grouped into several categories based on their publishers, such as public, media outlets, and celebrities. Each group can then be evaluated separately to estimate the maximum values for credibility factors and calculate the score.

150

Table 5.1. Correlations between the selected credibility factors and the credibility scores calculated for each social media stream.

Social Media Platform Pearson’s Correlation Coefficient

Twitter Verification (of a tweeter), ! 110247 = 0.710, ! ≪ 0.001.

Follower counts, ! 110247 = 0.269, ! ≪ 0.001.

Favorite counts, ! 110247 = 0.148, ! ≪ 0.001.

Retweet counts, ! 110247 = 0.681, ! ≪ 0.001.

Flickr View counts, ! 64 = 1, ! ≪ 0.001.

Google Plus Re-share counts, ! 933 = 0.779, ! ≪ 0.001.

Plus-one counts, ! 933 = 0.873, ! ≪ 0.001.

Reply counts, ! 933 = 0.728, ! ≪ 0.001.

Instagram Like counts, ! 6020 = 0.765, ! ≪ 0.001.

Comment counts, ! 6020 = 0.688, ! ≪ 0.001.

Follower counts, ! 6020 = 0.706, ! ≪ 0.001.

5.1.6 Total QS

It is somewhat difficult to assess the overall quality scores with this initial set of (un-refined metrics) and for this particular case study, because of the problems mentioned above: positional nearness and cross-referencing appear to have limited utility if a dataset is global. Second, location information is double weighted as there are two location-depended metrics, leading to a limitation that contributions without any spatial reference can only score 3 at most. Third, the semantic similarity scores are low, with 13% of contributions at Levels 3 and 4, which highlights the need for VGI-tailored semantic similarity assessment models.

151

A Pearson’s product-moment correlation coefficient was computed to assess the relationship between the quality scores. The results show a strong correlation between PNS and CRS, weak correlations between PNS, TNS, SSS, and CRS, and almost no correlations between CS and other quality scores. The reason behind this might be because of the fact that PNS and CRS are both spatial metrics, while other metrics are of different types (e.g., temporal (TNS) vs. textual

(SSS) vs. social (CS)). It is also apparent that PNS and CRS had a strong positive correlation with Total QS; TNS and SSS had a moderate positive correlation with Total QS; and the CS had no correlation with Total QS (see Table 5.2). The strong correlation between the location- dependent metrics (i.e., PNS and CRS) and Total QS can be partially explained due to the fact that the current model favors spatial information. It could also imply that contributions that are spatially closer to center of the crisis are of high quality. However, to verify the latter hypothesis requires further investigations. A lack of correlation between the CS and Total QS might be because of the way that the current model estimates the credibility score, which requires further refinements.

Table 5.2. Correlations between the quality scores.

PNS TNS SSS CRS CS Total QS

PNS 1.00 0.104 0.150 0.981 -0.049 0.926

TNS 1.00 0.136 0.104 0.060 0.368

SSS 1.00 0.180 -0.101 0.405

CRS 1.00 -0.054 0.935

CS 1.00 -0.003

Total QS 1.00

152

An initial analysis on the data revealed that correlation between the non-spatial quality scores

(i.e., TNS, SSS, and CS) and Total QS increases when removing location-dependent scores from the Total QS calculation (Equation (3.8)) – see Table 5.3. For example, the correlation between

CS and Total QS changed from no correlation to a weak correlation (-0.003 to 0.118), and the correlation between TNS and Total QS changed from moderate to strong (0.368 to 0.770). It is also evident that removing one of the spatial scores from the calculations results in a decrease in correlation between the remaining score and Total QS. For instance, as shown in Table 5.3, the correlation between PNS and Total QS decreased from 0.926 to 0.774 once the CRS was discarded.

Table 5.3. Correlations between the quality scores when removing location-dependent metrics (i.e., PNS and CRS).

Correlation Without PNS Without CRS Without PNS & CRS

PNS-Total QS N/A 0.774 N/A

TNS-Total QS 0.492 0.564 0.770

SSS-Total QS 0.523 0.563 0.772

CRS-Total QS 0.849 N/A N/A

CS-Total QS 0.023 0.042 0.118

This raises the issue of the relative merit of each metric relative to a disaster: should all metrics be weighted equally, or do some contribute greater value with respect to a crisis? If this is the case, then development of an appropriate weighting scheme using the Analytical Hierarchical

Process [223], for example, is necessary.

153

To finally evaluate if the quality evaluation is helpful in general, Table 5.4 to Table 5.7 present the highest-ranked social media contributions. An analysis of the message content confirms that

15 out 20 messages are “informative” – for example, in my opinion, among the messages listed in Table 5.4 to Table 5.7, all the five tweets, four out of five messages from Google Plus, and three out of five photos from Flickr and Instagram convey useful information. This suggests that the VGI framework worked as designed, from VGI data retrieval to data quality evaluation. Only the set of quality metrics needs refinement and further experimentation with different event datasets. This is expected that additional events would have different spatial and temporal characteristics that will enable better evaluation and understanding of metrics correlation and usability.

Table 5.4. A sample of Twitter contributions with the highest total quality score.

Tweet Total QS PNS TNS SSS CRS CS

Typhoon #Hagupit has made landfall in the 3.827 0.708 1 0.864 1 0.254 Philippines, knocking out power and taking down trees. Red Cross disaster teams are ready to go

At least 21 died in Philippines from Tropical 3.826 0.708 1 0.554 1 0.565 Storm Hagupit, says secretary general of the Philippine Red Cross. http://t.co/M1yhlgDhaI

Typhoon Hagupit death toll in Philippines 3.767 0.708 1 0.800 1 0.260 rises to at least 21 people. Millions fled their homes to avoid the storm http://t.co/3b1yz62KvM

Latest track on #Hagupit is bad news 4 3.762 0.733 1 0.778 1 0.250 Philippines this wknd. Looks 2 maintain typhoon strength as it impacts Manila.

154

http://t.co/49KHRmMw1M

RT @cnnbrk: At least 21 died in Philippines 3.706 0.708 1 0.554 1 0.444 from Tropical Storm Hagupit, says secretary general of the Philippine Red Cross. http://t.co/M1…

Officials in central Philippines city of Legazpi 3.684 0.732 1 0.418 1 0.534 expect to evacuate about 75,000 people ahead of #TyphoonHagupit http://t.co/Sr9Rezb9l2

Table 5.5. A sample of Google Plus contributions with the highest total quality score.

Text Total QS PNS TNS SSS CRS CS

More than 30 million people risk feeling the 3.690 0.660 1 1 1 0.030 brunt of Typhoon Hagupit as it barrels down on the Philippines, United Nations emergency and humanitarian officials on 5 December 2014 warned as they outlined the Organization-wide preparations being made before the massive storm makes landfall this weekend. At a press briefing in Geneva, Denis McClean, a spokesperson for the UN Office for Disaster Risk Reduction (UNISDR), said the Category 5 typhoon has triggered one of the largest peace time evacuations in Philippines history, noting that the extent of the evacuation “appeared to rival” the one million people evacuated last year when Cyclone Phailin threatened India’s coastline. As a result, he said, schools and local Government offices had been closed and evacuation centres were filling up as people were rapidly moving out of danger zones. While around 10 million residents of the

155

Philippines are at risk of flooding, storm surges and strong winds, “more than 30 million people could feel the impact,” he added. More http://b4in.org/tCim

Typhoon 'Hagupit' Strikes The Philippines - 3.490 1 1 0.490 1 0 Typhoon 'Hagupit' Landfall Typhoon Ruby / Hagupit Approaches...

Typhoon Hagupit begins agonizingly slow trek 3.328 1 1 0.328 1 0 across Philippines, deadly floods likely Typhoon Hagupit, which was once the strongest typhoon of the year, is making landfall as of 11:30 a.m. ET near the northern edge of the Philippine island of Samar, close to the community of Dolores. The storm is the equivalent of a Category 3 hurricane, with maximum sustained winds of about 125 miles per hour. Samar was also the landfall location of Super Typhoon Haiyan in 2013, which killed at least 6,000 people, and displaced more than 100,000. However, that storm hit the island's southern edge. Typhoon Ruby will make six landfalls from today to Monday. Please visit PRC's social media accounts for updates #RubyPH — Philippine Red Cross (Philippine Red Cross) December 6, 2014 Read more... http://buff.ly/1tY5Mi0

Contacting the Embassy About Typhoon 3.320 1 1 0.167 1 0.153 Hagupit Issues. You can alert us to U.S. citizens affected by the storm, including yourself, by email at [email protected] or by visiting https://tfa.state.gov selecting “2014 Typhoon Hagupit” and providing as much

156

information as possible.

Donate to help Typhoon Hagupit victims. 3.269 1 1 0.266 1 0.003 Habitat for Humanity has launched an appeal to assist families...

Table 5.6. A sample of Flickr contributions with the highest total quality score.

Caption Photo Total QS PNS TNS SSS CRS CS

A family waits inside Mercedes 4.184 1 1 1 1 0.184 Municipal Hall in Guinan, which is now a Typhoon evacuation shelter

A family waits inside Mercedes 4.180 1 1 1 1 0.180 Municipal Hall in Guinan, which is now a Typhoon evacuation shelter

Emergency responders during 4.088 1 1 0.892 1 0.196 Typhoon Ruby/ Hagupit in communication center in

Tacloban, the Philippines

Emergency responders during 4.079 1 1 0.892 1 0.187 Typhoon Ruby/ Hagupit in communication center in

Tacloban, the Philippines

The Philippines: Typhoon 2.792 0.667 1 0 1 0.125 Hagupit barrels through the country

157

Table 5.7. A sample of Instagram contributions with the highest total quality score.

Caption Photo Total QS PNS TNS SSS CRS CS

Urgent Call for Help for Typhoon 4.004 1 1 1 1 0.004 Ruby Victims (SAN FRANCISCO, CA)—Give2Asia today announces the launch of the Philippines Typhoon Recovery Fund to meet the urgent needs of the 2 million people affected by Typhoon Ruby this week. To help support the worst hit communities in Eastern Samar, please donate to Give2Asia’s Philippines Typhoon Recovery Fund at www.give2asia.org/ptrf. According to the United Nations Office for the Coordination of Humanitarian Affairs (OCHA), 1.7 million people have been moved to evacuation centers. The storm’s trajectory, which mirrored Typhoon Yolanda’s, caused an additional $28 million in damage to Eastern Samar, an area already struggling to recover from the previous typhoon to pass through the area. The OCHA’s initial estimates suggest that between 30-60% of homes in Eastern Samar have been severely damaged, and as people begin to return home, they will need emergency and transitional shelter.

158

My prayers are with my friends and 3.753 1 1 0 1 0.753 followers in the Philippines. I was there only 1 week ago and it's a country I love, full of amazing people so am really sad to see the effects of Typhoon Hagupit which is still going strong. Please stay safe

This photo shows a man climbing 3.433 1 1 0 0.75 0.683 his house after it was wrecked by #TyphoonRuby. Let us all pray for those who are affected by this typhoon especially in Tacloban and Borongan, Eastern Samar (which is the hometown of my Grandmother on my Mom's side) and those who have lost their loved ones, so that they may find strength and hope to move on from this tragedy. Dear guardian angels and dear God, please watch over us. Thank you. God bless us all! #rubyph

Disaster Response: Typhoon 3.302 1 1 0.302 1 0 Hagupit/Ruby made landfall on Samar, an island in the Visayas, Philippines, late in the evening on 6 December. Habitat for Humanity is sheltering evacuated families in Leyte, Cebu and Surigao del Sur provinces of the Philippines. Staff and materials are prepositioned and we will begin to support affected families to repair their homes. @habitatforhumanity is appealing for funds to assist families in this

159

time of crisis. Your generous donation will help them rebuild their lives and gain access to decent shelter. Click here to donate: http://goo.gl/BKXk7Y #typhoonhagupit #disasterresponse

Our teams on the ground are rapidly 3.289 1 1 0.286 1 0.003 preparing emergency kits for distribution in the # Philippines . These include equipment to provide clean water, emergency shelter kits for home repairs, hygiene equipment, essential household items and tents to help teams set up Child Friendly Spaces as quickly as possible to protect children from the chaos of an emergency situation. Read our latest news on # Typhoon # Hagupit here: http://bit.ly/1wCfJHU #TyphoonHagupit #Hagupit #BePreparedPH #SaveTheChildren #SaveChildrenAus #Emergency #Disaster

5.2 Evaluation of the Technical Framework

This section discusses both qualitative and quantitative criteria to verify whether the system

design successfully meets users’ needs and to evaluate how well it performs under different

conditions.

To implement the architecture, a FOSS (Free and Open Source Software) strategy was employed

for two reasons. First, it minimizes the potential cost required to implement, modify, or

160

customize the system. Second, it facilitates the free adoption of the platform by organizations tasked with disaster management responsibilities [224,225]. With respect to the properties of the architecture it can be concluded that:

• The proposed framework follows open geospatial and World Wide Web standards (i.e.,

OGC and W3C), which facilitate machine-machine and human-machine interactions in

an interoperable manner. For instance, GeoServer WPS, pycsw, and istSOS instances,

implementing WPS 1.0.0, CSW 2.0.2, and SOS 1.0.0, were used to deploy the VGI

framework for the case study. An interoperability test using OGC TEAM engine94

confirmed the standard compliance of the VGI framework’s components (see Figure 5.1).

• It allows developers to design and build system components (i.e., services) using various

technologies and tools (e.g., FOSS or proprietary). The architecture presented in this

work is an abstract or high-level framework in such a way that it is independent from the

technology enablers. It is actually developed based on open standards/specifications and

it is not dependent on the technology used to implement it. Hence, different technologies

and tools can be employed to meet different application requirements. For example, OGC

web services can be implemented using different programming languages and still be

standard-compliant95. In the end, it is developers' choice which software tools and

libraries to use for implementation purposes. This highlights the fact that as long as the

architecture is standard-compliant it is flexible with respect to the technology to employ

for implementing it.

94 A tool developed by OGC that evaluates standard compliance of the servers that implement OGC standards (http://cite.opengeospatial.org/teamengine/). 95 For instance, GeoServer, PyWPS, and ZOO are three OGC-compliant frameworks written in Java, Python, and C/C++, respectively.

161

• The architecture supports a service-based development approach, which provides the

flexibility necessary to allow changes or customization of the system. A service-based

design and development approach was employed in such a way that tasks were broken

into small pieces and each piece was developed as a standard web service. This facilitates

changing/improving the small pieces (i.e., system components), and ensures that the

integration of improvements to the system does not affect other architectural components.

• It provides a flexible deployment solution where the system components can be easily

“plugged into” existing systems, and allows deployment in both local and distributed

(e.g., cloud) environments. As discussed above, a standards-compliant, service-based

development approach allows for different deployment strategies. Therefore, services can

be distributed on different servers to handle heavy (geo)processing tasks. For example, in

the case study, the geoprocessing workflows were developed in such a way that a central

component, i.e., a web service, is used to control the interaction between the different

services involved in the workflows. Using this approach facilitates workflow

management in a distributed environment.

Consequently, regarding the primary quality attribute requirements and business goals [150,151] that drove system development (see Section 3.1), it is concluded that quality attributes such as open systems/standards and interoperability are met in the proposed framework. Moreover, the secondary priorities, flexibility, integrability, ease-of-installation, and functionality are supported through the adoption of “best practices” throughout the design process. Lower level priorities such as portability and ease-of-repair are also met (see Table 5.8).

162

Figure 5.1. Part of the results of standard compliance tests performed to ensure interoperability of system components.

163

Table 5.8. Quality attributes requirements and business goals addressed by the VGI framework.

Framework Characteristics Quality Requirements / Business Goals

OGC and W3C compatible service Interoperability interfaces and data encodings Integrability

Flexibility

Ease-of-use

Service-based development approach Portability

Integrability

Ease-of-installation

Flexibility

Ease-of–repair

Functionality

FOSS development strategy Portability

Flexibility

Ease-of-repair

Functionality

However, as outlined, there is still a need to assess the framework in terms of practical technical performance and usability. This will be necessary to understand how the system responds in real emergency operations to ensure that system operators are able to put the VGI framework to good use. Usability metrics could include efficiency, satisfaction of the platform, effectiveness, perceived ease of use, flexibility of deployment, and quality of documentation. Performance metrics might include quality of service (QoS) criteria such as performance and scalability to determine efficiency and effectiveness of the platform under specific conditions.

164

As described in Chapter 3, the VGI framework was implemented as a set of WPS, SOS, and

CSW service instances integrated in a geoprocessing workflow. So far, a performance evaluation has been carried out on WPS servers, including 52°North, deegree, GeoServer, PyWPS, and Zoo, and SOS servers, including 52°North, deegree, and MapServer. The result of the WPS servers’ performance evaluation suggests that there was no significant difference in system response time

(F(4, 220) = 0.739, P = 0.566), nor data volume returned (F(4, 220) = 1.071, P = 0.372)

– response document data volume varied between WPS because each WPS structured their response document differently – regardless of which WPS was used. The architecture continued to perform well under all load tests undertaken. Any failures to respond to a client request using the architecture were due to a WPS failure rather than an architectural failure. From a developer’s perspective, there were no limitations encountered that would suggest circumspection in the use of the architecture for development of geoprocessing workflows. The full evaluation results can be found in Poorazizi & Hunter [226].

The result of a performance evaluation on SOS servers indicates that deegree performed the best, especially when data requests cover larger areas, and longer temporal intervals. 52°North SOS showed competitive performance across all types of queries, but does not scale quite as well as deegree SOS. Poorazizi et al. [227] report the evaluation results in further details.

To conclude, when selecting an appropriate (spatial) data/processing server, it is important to consider both quantitative and qualitative metrics. The importance of each metric can be weighted based on different application requirements. Generally speaking, from a user’s perspective, performance is one of the most important factors when choosing a web-based application, while from developers’ perspective, qualitative factors such as perceived ease of installation and configuration, variety of development languages, quality of documentation, and

165

accessibility of support may be more critical. To choose a server, this research suggests starting an evaluation process with a basic set of questions that are linked to the evaluation criteria. The questions could be “Who is the user of the system?” “What should the end-user be able to do with the system?” “What programming languages are developers comfortable with for develop of the system?” “How complex are the back-end processes?” “How should the system function, synchronous or asynchronous?” “What is the architecture used to design the (geo)processing workflow?” “What is the expected number of users?” In the end, the most appropriate server should be selected based on a trade-off between quantitative “performance” metrics and qualitative “ease-of-use” metrics for a specific application or use case. This may lead to the selection of different servers for different applications.

5.3 Chapter Summary

This chapter reported the results of an evaluation of the framework proposed in this research, as an effort to achieve the fifth research objective (see Section 1.4). To do so, first, it evaluated the suitability of VGI quality metrics and discussed the strength and weaknesses of the metrics in disaster response using the Typhoon Ruby case study reported in Chapter 4. Afterwards, it assessed the framework from a technical perspective, according to the quality requirements and business goals outlined in Section 3.1, to verify whether the system design successfully meets users’ needs and to evaluate how well it performs under different conditions. The next chapter will present a summary of findings of this work, highlight its main contributions, and provide an outlook.

166

CHAPTER SIX: CONCLUSIONS

This thesis presented research on the development of a framework for effective use of VGI in disaster management platforms. The main focus was to develop a robust workflow to facilitate collecting, validating, and sharing VGI, published on social media, that is relevant, credible, and actionable for emergency response. This final chapter, first, discusses the key findings of this thesis with respect to the research questions identified in Chapter 1 (Section 1.4). It then provides a summary of main contributions of this thesis and proposes possible directions for future research.

6.1 Revisiting the Research Questions

6.1.1 Research Question 1

What specific information values does VGI present that can complement

authoritative spatial information in a disaster? What are the strengths and

weakness of VGI in this context?

The results of the case study on Typhoon Ruby indicated that the VGI framework was able to detect crisis-related social media messages that provide timely information regarding the incident locations and impacts of the ongoing event. For example, it was observed that informative contributions were posted mostly during the event and were spatially closer to the center of the crisis. This highlights the thematic (semantic), temporal, and spatial information values that VGI provides. It also indicates the potential of VGI as a complementary source of information that can help citizens, decision makers, and disaster responders to answer “where”, “when”, and

“what” questions. For instance, using the data provided by the VGI framework, disaster responders could filter infrastructure damage reports, extract the incidents time and location,

167

perform an initial damage assessment, and prioritize the relief operation accordingly. As a result, the use of VGI can speed up disaster response operations, since extraction of useful information from official data sources such as satellite imagery could take hours to days to be accomplished.

However, VGI is by nature subjective, which introduces quality and credibility issues. Moreover,

VGI is not a representative sample of people affected by a disaster, since it is subject to several biases [228]. This is mainly because of the digital divide or the inequalities created by technology. Poverty, location, literacy, and gender inequalities are among the main driving factors [2]. As a result VGI is sparse in less populated or socio-economically less developed regions. There are also a number of legal issues arising from VGI, for example intellectual property rights, data ownership and copyright, and liability issues, which are amongst the main barriers in integration of VGI and authoritative data [229].

6.1.2 Research Question 2

What would be a typical (geo)processing workflow that allows collecting,

validating, and sharing VGI in an automated way? What specific design

requirements should be considered to develop the system components?

To respond to this question, this research introduced a conceptual framework, the VGI framework, for discovery and use of disaster-related VGI, developed from a set of functional and architectural requirements. The VGI framework consists of four components: (i) a VGI brokering module to provide a standard service interface to retrieve VGI from multiple resources; (ii) a VGI quality control component to evaluate the spatiotemporal relevance and credibility of VGI; (iii) a VGI publisher module, which uses a service-based delivery mechanism

168

to disseminate VGI; and (iv) a VGI discovery component, which acts as a catalogue service to find, browse, and query metadata about available VGI datasets.

A set of quality attributes/requirements and business goals was outlined to guide the platform development and to evaluate the consequences of architectural decisions. Examples of such quality attributes include compliance to open standards, interoperability, flexibility, and scalability and performance. A service-based development strategy was followed to implement the system components. In this context, different architectural patterns, such as granularity of services, service-chaining architectures, and service dissemination mechanisms, used to create, integrate, and publish the components were discussed. For the prototype, services with different levels of granularity (e.g., Geocoding WPS as a fine-grained service and Semantic Similarity

WPS as a medium-grained service) were developed. They were then published as standard

OGC/W3C service instances and integrated in a centralized workflow to maintain a balance between different quality attributes, for example performance and flexibility (reusability). The evaluation results indicate that the developed platform enables interoperability, performs well under different conditions (e.g., high workloads), and is flexible in terms of component integration for new or existing disaster management platforms.

6.1.3 Research Question 3

Considering a massive amount of VGI from multiple social media platforms,

what mechanism can facilitate discovery of and access to VGI from different

social media platforms?

A VGI brokering approach was proposed as an effort towards addressing this question. The VGI

Broker provides a single service interface to find and collate user-generated content from various

169

social media platforms using a keyword-based filtering approach. It translates a single query to multiple API-based queries and handles different request-response data formats. The VGI Broker is implemented as a set of WPS services and, therefore, improves the interoperability between clients and the heterogeneous Web 2.0 services (i.e., social media platforms). Moreover, since it is developed as a (centralized) workflow of services, additional brokers can be easily added to the platform to enable support for more VGI resources.

6.1.4 Research Question 4

What strategy can be used to tackle the verification, i.e., credibility, of VGI

from social media streams? What quality control metrics are suitable to

evaluate VGI credibility? What will the quality evaluation tell us about VGI?

This research introduced a set of VGI-tailored quality metrics and a quality control mechanism to cope with the credibility issue of VGI. It was discussed that it is necessary to re-define traditional

(spatial) data quality metrics with respect to the particular characteristics of VGI to ensure that the data is fit-for-purpose. In this context, “crowdsourcing”, “geographic”, and “domain” analysis approaches were adopted to derive the metrics that consider users’ ratings on VGI, measure spatial quality of VGI, and evaluate the relevance of VGI to a given disaster. The quality metrics were developed using spatial analysis, machine learning, and natural language processing techniques and then incorporated into a quality control workflow called VGI QC.

Unlike similar platforms that filter out the alleged irrelevant VGI, the VGI QC implements a ranking mechanism that calculates a relative score for VGI in terms of each quality metric, and aggregates the individual scores to estimate overall quality of VGI. This allows for further

170

refinement of the metrics and, consequently, the quality scores over time, since VGI QC is an iterative process.

The results of the case study on Typhoon Ruby indicate that the highest-ranking social media contributions provided disaster-related information, meaning that they are fit-for-purpose. This suggests that the VGI QC, with relatively simple parameters and approaches, worked well. The results also highlight the differences between VGI from the four social media streams – Flickr,

Google Plus, Instagram, and Twitter. These differences include different types of media (text, images, etc.), different spatial and temporal contribution patterns, and different credibility information. A strong positive correlation between the spatial quality scores (i.e., PNS and CRS) and total quality scores (i.e., Total QS) and almost no correlation between credibility scores (CS) and total quality scores were observed. This highlights the need for further experimentation with different event datasets, comparison of the results, and re-evaluation of the metrics. It will, in turn, allow answering the following questions with more confidence: What metrics do need further refinement? Do some metrics contribute greater value with respect to a certain disaster event? If yes, should a weighting scheme be developed for different event types?

6.1.5 Research Question 5

How can quality-assessed VGI be shared in a way that increases availability of

VGI and, eventually, facilitates incorporation of VGI and authoritative data in

disaster management platforms?

This research proposed the VGI Publisher and VGI Discovery modules to address this research question. VGI Publisher provides a standard-based data publication mechanism for information sharing. It employs the sensor web enablement (SWE) framework to model VGI as observation

171

data collected by humans and publish the observation data as a service. This, on the one hand, promotes the rates of sharing and availability of VGI in the form of standards-based geospatial services in SDI-compatible environments. For example, VGI can be accessed in web-based visualization platforms to produce crisis web-maps or can be used in disaster alert and notification services that send situation updates to subscribed citizens on their mobile devices.

On the other hand, since authoritative spatial data is (ideally) published by SDI services in disaster management platforms, a standards-based VGI dissemination facilitates interactions between SDI- and VGI-based systems, thus providing better interoperability. As a result, VGI can be used as a complementary data source with authoritative data in new or existing disaster management platforms.

The use of metadata and catalogue services is essential in SDIs in order for spatial data and services to be properly discovered [230]. Following this approach, VGI Discovery implements a standard web service that publishes metadata, and describes and advertises the available VGI published by the VGI Publisher. This way clients can find available VGI data service instances that provide the desired social media content.

6.2 Summary of Contributions

The primary objective of this research was to develop a framework for the effective use of VGI in disaster management platforms. This research provided the following main contributions with respect to the research objectives:

• A novel conceptual framework, i.e., the VGI framework, was proposed that provides the

essential components for collection, quality assessment, and dissemination of VGI for use

in disaster management platforms.

• A new VGI brokering approach was proposed that facilitates VGI search and retrieval

172

from different social media platforms using a single-access standard-based web service

interface.

• A novel VGI quality control model was proposed that includes a set of quality metrics

specifically designed to evaluate the quality of VGI in terms of spatial, temporal,

semantic, and credibility factors.

• A new geoparsing mechanism was introduced that identifies and geocodes place

names in social media messages.

• A new hybrid text processing approach was introduced that classifies social media

messages and evaluates their relevance to the context of a disaster.

• An effective VGI sharing mechanism was proposed that enables standard publication of

VGI and allows for VGI search and retrieval based on spatial, temporal, textual, and

quality parameters.

• A technical architecture, a foundation, and a guideline were presented that enables

integration of VGI framework’s components into an easy-to-access, easy-to-use, and

easy-to-maintain service-oriented system. Although the development strategy and design

principles were employed to implement the VGI framework, they are generalizable and

can be applied to develop (geospatial) service-oriented applications in general as

demonstrated in Poorazizi et al. [231].

6.3 Future Work

The concepts and methods proposed in this thesis can be further investigated and explored in various ways. Four primary areas of research are proposed here.

The first concerns information values provided by VGI. This research has shown how the various dimensions of VGI – including spatial, temporal, semantic, and credibility – can

173

contribute to the validation process, but ignored the “visual” dimension. The visual dimension refers to the visual content (i.e., images and videos) of VGI in contrast to the text-dimension that formed the base for the quality assessment in the current prototype. Analysis of VGI’s visual content can provide contextual information and contribute to situational awareness, as the old saying goes “a picture is worth a thousand words”. This would be extremely valuable to interpretation and evaluation of VGI collected from photo/video sharing services such as Flickr and Instagram, those media contributions that were evaluated only based on semantics contained in VGI’s textual content. Hence, research into algorithms for automatic classification of images and videos of media streams is required so that the current VGI quality evaluation mechanism can be improved. An interesting direction in this context would be the use deep-learning frameworks, such as TensorFlow96, for developing image/video recognition services.

The second direction concerns natural language processing. This research employed machine- learning algorithms to identify disaster-related VGI and applied a semantic similarity assessment approach to calculate a score for related VGI items based on their similarity to a reference dictionary of disaster-related terms. The machine learning algorithms were trained using a sample of annotated tweets collected during Typhoon Ruby. Now, there is a need to test if the text classification model performs well on another disaster dataset such as flooding. It is also necessary to evaluate the performance of the text classification model on VGI from other sources such as Flickr, Google Plus, and Instagram, which was ignored in this research due to a lack of data for the Typhoon Ruby case study. The following question needs to be answered in the end:

“Do we need to train a classifier for each specific type of disaster or VGI?”

96 https://www.tensorflow.org/

174

This requires further research in the field of machine learning and natural language processing to find techniques that maintain the accuracy of a text classifier trained on one dataset when performing on a dataset with different characteristics, for example domain adaptation methods

[232].

The reference dictionary used in the semantic similarity assessment procedure was developed based on CrisisLex, which is a lexicon of disaster-related terms that frequently appeared in relevant tweets during different types of disasters between October 2012 and July 2013 in the

US, Canada, and Australia [153]. Hence, there is a need for further development of CrisisLex through analyzing contributions from

• Different social media platforms,

• Different geographical regions, and

• More recent disaster events.

A particular problem related to analyzing geographical regions is the processing of different languages: it is necessary to develop language-based reference dictionaries for the text classifier so that the current solution can be adapted for international (non-English) use cases.

The third area of research relates to geoparsing. One of the challenges of spatial analysis of social media messages is the low number of “explicitly” geo-tagged contributions. To enrich the spatial dimension of VGI, this research proposed a geoparsing approach that extracts and geocodes place names embedded in the content of social media messages, i.e., “implicit” spatial information. This process is a key priority for geo-referencing global contributions (i.e., messages sent from outside of the affected area), since their explicit location may point to a place far away from the center of the crisis event and, therefore, most likely is not of much value for disaster responders on the ground. The current geoparsing method was able to identify place

175

names within text with an F1 score of 92% and to geocode the place names with a success rate of

100%. However, geoparsing is still a challenging task because of place names ambiguity (e.g., several places with similar names), misspelling, and vernacular terms (e.g., unofficial place names). Perhaps the process can best be improved by (i) tuning the name entity recognition algorithms, (ii) improving geocoding methods, and (iii) enriching reference geographical databases. An alternative approach is to encourage people to report the impact location using a universal addressing system such as What3Words97, which provides a unique three-word phrase for every nine-square-meters spot on Earth. This would facilitate both sharing and interpreting location in text, but requires further research on how to utilize its functionality and promote its usage within social media platforms.

The final area of proposed research concerns the integration of other sources of information such as satellite and in-situ sensors into the VGI framework. The combined use of information from different sources, including authoritative and user-generated, would be beneficial to improve the data validation process so as to achieve more realistic results. For example, the result of satellite image analysis (e.g., damage assessment) or in-situ sensor observations (e.g., flood sensors) can be used to evaluate the quality of VGI items in terms of spatial, temporal, and thematic information they convey. As a result, tactical information can be extracted and used for planning and decision-making purposes. There is also a potential to use disaster alerts from satellite observations and in-situ sensor readings as a trigger for specific processes within the VGI framework (e.g., VGI Broker) to be executed. Hence, further investigations are required to

97 http://what3words.com/

176

understand how authoritative data and VGI actually complement each other and how they can be used in conjunction.

177

References

1. Vescoukis, V.; Doulamis, N.; Karagiorgou, S. A service oriented architecture for decision support systems in environmental crisis management. Future Gener. Comput. Syst. 2012, 28, 593–604. 2. IFRC World disasters report 2015: focus on local actors, the key to humanitarian effectiveness; International Federation of Red Cross and Red Crescent Societies: Geneva, Switzerland, 2015. 3. Manfré, L. A.; Hirata, E.; Silva, J. B.; Shinohara, E. J.; Giannotti, M. A.; Larocca, A. P. C.; Quintanilha, J. A. An Analysis of Geospatial Technologies for Risk and Natural Disaster Management. ISPRS Int. J. Geo-Inf. 2012, 1, 166–185. 4. Alamdar, F.; Kalantari, M.; Rajabifard, A. An evaluation of integrating multisourced sensors for disaster management. Int. J. Digit. Earth 2015, 8, 727–749. 5. Altay, N.; Green III, W. G. OR/MS research in disaster operations management. Eur. J. Oper. Res. 2006, 175, 475–493. 6. Zakour, M. J.; Gillespie, D. F. Community Disaster Vulnerability; Springer New York: New York, NY, 2013. 7. Nakagawa, Y.; Shaw, R. Social capital: A missing link to disaster recovery. Int. J. Mass Emergencies Disasters 2004, 22, 5–34. 8. Hystad, P. W.; Keller, P. C. Towards a destination tourism disaster management framework: Long-term lessons from a forest fire disaster. Tour. Manag. 2008, 29, 151–162. 9. Miyazaki, H.; Nagai, M.; Shibasaki, R. Reviews of Geospatial Information Technology and Collaborative Data Delivery for Disaster Risk Management. ISPRS Int. J. Geo-Inf. 2015, 4, 1936–1964. 10. Jongman, B.; Wagemaker, J.; Romero, B. R.; de Perez, E. C. Early Flood Detection for Rapid Humanitarian Response: Harnessing Near Real-Time Satellite and Twitter Signals. ISPRS Int. J. Geo-Inf. 2015, 4, 2246–2266. 11. Spinsanti, L.; Ostermann, F. Automated geographic context analysis for volunteered information. Appl. Geogr. 2013, 43, 36–44. 12. Spinsanti, L.; Ostermann, F. Automated geographic context analysis for volunteered information. Appl. Geogr. 2013, 43, 36–44. 13. Budhathoki, N. R.; Bruce, B. (Chip); Nedovic-Budic, Z. Reconceptualizing the role of the user of spatial data infrastructure. GeoJournal 2008, 72, 149–160. 14. Craglia, M.; Ostermann, F.; Spinsanti, L. Digital Earth from vision to practice: making sense of citizen-generated content. Int. J. Digit. Earth 2012, 5, 398–416. 15. Goodchild, M. F. Citizens as sensors: the world of volunteered geography. GeoJournal 2007, 69, 211–221.

178

16. OpenStreetMap in GIScience; Jokar Arsanjani, J.; Zipf, A.; Mooney, P.; Helbich, M., Eds.; Lecture Notes in Geoinformation and Cartography; Springer International Publishing: Cham, 2015. 17. Meier, P. Crisis Mapping in Action: How Open Source Software and Global Volunteer Networks Are Changing the World, One Map at a Time. J. Map Geogr. Libr. 2012, 8, 89–100. 18. Goodchild, M. F.; Glennon, J. A. Crowdsourcing geographic information for disaster response: a research frontier. Int. J. Digit. Earth 2010, 3, 231–241. 19. Earle, P.; Guy, M.; Buckmaster, R.; Ostrum, C.; Horvath, S.; Vaughan, A. OMG earthquake! can twitter improve earthquake response? Seismol. Res. Lett. 2010, 81, 246–251. 20. Vieweg, S.; Hughes, A. L.; Starbird, K.; Palen, L. Microblogging During Two Natural Hazards Events: What Twitter May Contribute to Situational Awareness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; CHI ’10; ACM: New York, NY, USA, 2010; pp. 1079–1088. 21. Muller, C. l.; Chapman, L.; Johnston, S.; Kidd, C.; Illingworth, S.; Foody, G.; Overeem, A.; Leigh, R. r. Crowdsourcing for climate and atmospheric sciences: current status and future potential. Int. J. Climatol. 2015, 35, 3185–3203. 22. Muller, C. L. Mapping snow depth across the West Midlands using social media- generated data. Weather 2013, 68, 82–82. 23. Takahashi, B.; Tandoc Jr., E. C.; Carmichael, C. Communicating on Twitter during a disaster: An analysis of tweets during Typhoon Haiyan in the Philippines. Comput. Hum. Behav. 2015, 50, 392–398. 24. Crawford, K.; Finn, M. The limits of crisis data: analytical and ethical challenges of using social and mobile data to understand disasters. GeoJournal 2014, 80, 491–502. 25. Imran, M.; Castillo, C.; Diaz, F.; Vieweg, S. Processing Social Media Messages in Mass Emergency: A Survey. ACM Comput Surv 2015, 47, 67:1–67:38. 26. McDougall, K. Using Volunteered Information to Map the Queensland Floods. In Proceedings of the Surveying & Spatial Sciences Biennial Conference 2011; Wellington, New Zealand, 2011. 27. Haworth, B. Emergency management perspectives on volunteered geographic information: Opportunities, challenges and change. Comput. Environ. Urban Syst. 2016, 57, 189–198. 28. Schade, S.; Díaz, L.; Ostermann, F.; Spinsanti, L.; Luraschi, G.; Cox, S.; Nuñez, M.; De Longueville, B. Citizen-based sensing of crisis events: sensor web enablement for volunteered geographic information. Appl. Geomat. 2013, 5, 3–18. 29. Castillo, C.; Mendoza, M.; Poblete, B. Information Credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web; WWW ’11; ACM: New York, NY, USA, 2011; pp. 675–684. 30. Farzindar, A.; Khreich, W. A Survey of Techniques for Event Detection in Twitter.

179

Comput. Intell. 2015, 31, 132–164. 31. Haworth, B.; Bruce, E. A Review of Volunteered Geographic Information for Disaster Management. Geogr. Compass 2015, 9, 237–250. 32. Lindsay, B. R. Social media and disasters: Current uses, future options, and policy considerations; Congressional Research Service Washington, DC, 2011. 33. De Longueville, B.; Annoni, A.; Schade, S.; Ostlaender, N.; Whitmore, C. Digital Earth’s Nervous System for crisis events: real-time Sensor Web Enablement of Volunteered Geographic Information. Int. J. Digit. Earth 2010, 3, 242–259. 34. Hung, K.-C.; Kalantari, M.; Rajabifard, A. Methods for assessing the credibility of volunteered geographic information in flood response: A case study in Brisbane, Australia. Appl. Geogr. 2016, 68, 37–47. 35. Annual Global Climate and Catastrophe Report; Impact Forecasting, 2013. 36. Wright, M. Social media tools proved vital in 2013 floods. Calg. Her. 37. Dymon, U. J. The role of emergency mapping in disaster response. FMHI Publ. 1990. 38. Poorazizi, M. E.; Hunter, A. J. S. A Crisis Map Mashup for Calgary’s 2013 Flood. In; Delta Edmonton South Hotel and Conference Centre, Edmonton, AB, Canada, 2013. 39. Ramsay, M. State of Emergency lifted in Calgary | Globalnews.ca 2013. 40. Calgary Flood Infographic Shows Remarkable Power Of Social Media. Huffington Post. 41. Hughes, A. L.; Palen, L. Twitter adoption and use in mass convergence and emergency events. Int. J. Emerg. Manag. 2009, 6, 248. 42. Díaz, L.; Granell, C.; Huerta, J.; Gould, M. Web 2.0 Broker: A standards-based service for spatio-temporal search of crowd-sourced information. Appl. Geogr. 2012, 35, 448–459. 43. Pierce, D. We’re Drowning in Content. Recommendations Are What We Need. Wired. 44. Imran, M.; Castillo, C.; Lucas, J.; Meier, P.; Vieweg, S. AIDR: Artificial Intelligence for Disaster Response. In Proceedings of the 23rd International Conference on World Wide Web; WWW ’14 Companion; ACM: New York, NY, USA, 2014; pp. 159–162. 45. Tsou, M.-H. Research challenges and opportunities in mapping social media and Big Data. Cartogr. Geogr. Inf. Sci. 2015, 42, 70–74. 46. Reuter, C.; Marx, A.; Pipek, V. Crisis Management 2.0: Towards a Systematization of Social Software Use in Crisis Situations. Int. J. Inf. Syst. Crisis Response Manag. 2012, 4, 1–16. 47. Fast, V.; Rinner, C. A Systems Perspective on Volunteered Geographic Information. ISPRS Int. J. Geo-Inf. 2014, 3, 1278–1292. 48. Elwood, S. Volunteered geographic information: future research directions motivated by critical, participatory, and feminist GIS. GeoJournal 2008, 72, 173–183. 49. Haklay, M. International Encyclopedia of Geography - Quality Assurance of VGI. Po Ve Sham - Muki Haklays Pers. Blog 2014.

180

50. Widener, M. J.; Li, W. Using geolocated Twitter data to monitor the prevalence of healthy and unhealthy food references across the US. Appl. Geogr. 2014, 54, 189–197. 51. Ding, Y.; Fan, Y.; Du, Z.; Zhu, Q.; Wang, W.; Liu, S.; Lin, H. An integrated geospatial information service system for disaster management in China. Int. J. Digit. Earth 2015, 8, 918–945. 52. Sondheim, M.; Gardels, K.; Buehler, K. GIS interoperability. In Geographical Information Systems: Principles, Techniques, Applications and Management (Second Edition); Longley, P. A.; Goodchild, M. F.; Maguire, D. J.; Rhind, D. W., Eds.; John Wiley and Sons: New York, 1999; Vol. 1, pp. 347–358. 53. GSDI Spatial Data Infrastructure Cookbook; Global Spatial Data Infrastructure Association, 2009. 54. Díaz, L.; Granell, C.; Gould, M.; Huerta, J. Managing user-generated information in geospatial cyberinfrastructures. Future Gener. Comput. Syst. 2011, 27, 304–314. 55. Bondi, A. B. Characteristics of Scalability and Their Impact on Performance. In Proceedings of the 2Nd International Workshop on Software and Performance; WOSP ’00; ACM: New York, NY, USA, 2000; pp. 195–203. 56. Hevner, A. R.; March, S. T.; Park, J.; Ram, S. Design Science in Information Systems Research. MIS Q. 2004, 28, 75–105. 57. Handbook of mixed methods in social and behavioural research; Tashakkori, A.; Teddlie, C., Eds.; Sage Publications: Thousand Oakes, CA, 2003. 58. Huang, C.-M.; Chan, E.; Hyder, A. A. Web 2.0 and Internet Social Networking: A New tool for Disaster Management? - Lessons from Taiwan. BMC Med. Inform. Decis. Mak. 2010, 10, 57. 59. Laituri, M.; Kodrich, K. On Line Disaster Response Community: People as Sensors of High Magnitude Disasters Using Internet GIS. Sensors 2008, 8, 3037–3055. 60. Shankar, K. Wind, Water, and Wi-Fi: New Trends in Community Informatics and Disaster Management. Inf. Soc. 2008, 24, 116–120. 61. Below, R.; Wirtz, A.; Guha-Sapir, D. Disaster category classification and peril terminology for operational purposes; Centre for Research on the Epidemiology of Disasters, Catholic University of Louvain and Munich Reinsurance Company, 2009. 62. IFRC About disaster management http://www.ifrc.org/en/what-we-do/disaster- management/about-disaster-management/ (accessed May 4, 2016). 63. Yang, C.; Raskin, R.; Goodchild, M.; Gahegan, M. Geospatial Cyberinfrastructure: Past, present and future. Comput. Environ. Urban Syst. 2010, 34, 264–277. 64. Di, L. A framework for developing Web-service-based intelligent geospatial knowledge systems. Geogr. Inf. Sci. 2005, 11, 24–28. 65. Friis‐Christensen, A.; Ostländer, N.; Lutz, M.; Bernard, L. Designing Service Architectures for Distributed Geoprocessing: Challenges and Future Directions. Trans. GIS 2007, 11, 799–

181

818. 66. Roche, S.; Propeck-Zimmermann, E.; Mericskay, B. GeoWeb and crisis management: issues and perspectives of volunteered geographic information. GeoJournal 2013, 78, 21– 40. 67. Erl, T. Service-Oriented Architecture: Concepts, Technology, and Design; Prentice Hall, 2005. 68. Haesen, R.; Snoeck, M.; Lemahieu, W.; Poelmans, S. On the Definition of Service Granularity and Its Architectural Impact. In Advanced Information Systems Engineering; Bellahsène, Z.; Léonard, M., Eds.; Lecture Notes in Computer Science; Springer Berlin / Heidelberg, 2008; Vol. 5074, pp. 375–389. 69. Granell, C.; Díaz, L.; Gould, M. Service-oriented applications for environmental models: Reusable geospatial services. Environ. Model. Softw. 2010, 25, 182–198. 70. Kim, D.; Lee, S.; Han, S.; Abraham, A. Improving Web services performance using priority allocation method. In International Conference on Next Generation Web Services Practices, 2005. NWeSP 2005; 2005; p. 6 pp. 71. Papazoglou, M. P.; Van Den Heuvel, W.-J. Service-oriented design and development methodology. Int. J. Web Eng. Technol. 2006, 2, 412–442. 72. Kirda, E. Web engineering device independent web services. In Proceedings of the 23rd International Conference on Software Engineering; ICSE ’01; IEEE Computer Society: Washington, DC, USA, 2001; pp. 795–796. 73. Li, X.; Di, L.; Han, W.; Zhao, P.; Dadi, U. Sharing geoscience algorithms in a Web service- oriented environment (GRASS GIS example). Comput. Geosci. 2010, 36, 1060–1068. 74. Stojanovic, Z.; Dahanayake, A.; Sol, H. Modeling and design of service-oriented architecture. In 2004 IEEE International Conference on Systems, Man and Cybernetics; IEEE: The Hague, Netherlands, 2004; Vol. 5, pp. 4147–4152 vol.5. 75. Alameh, N. Chaining geographic information Web services. IEEE Internet Comput. 2003, 7, 22–29. 76. Friis-Christensen, A.; Lucchi, R.; Lutz, M.; Ostländer, N. Service chaining architectures for applications implementing distributed geographic information processing. Int. J. Geogr. Inf. Sci. 2009, 23, 561–580. 77. Zhao, J.; Coleman, D. J. GeoDF: Towards a SDI-based PPGIS application for E- Governance. In Proceedings of the GSDI 2006; Santiago de Chile, 2006; Vol. 9. 78. ISO Information technology -- Vocabulary -- Part 1: Fundamental terms http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=7229 (accessed Aug 7, 2012). 79. Granell, C.; Schade, S.; Hobona, G. Linked Data: Connecting Spatial Data Infrastructures and Volunteered Geographic Information. In Geospatial Web Services: Advances in Information Interoperability; Zhao, P.; Di, L., Eds.; IGI Global: Hershey, PA, 2010; pp. 189–

182

226. 80. Booth, D.; Haas, H.; McCabe, F.; Newcomer, E.; Champion, M.; Ferris, C.; Orchard, D. Web services architecture; W3C, 2004. 81. Lee, C.; Percivall, G. Standards-Based Computing Capabilities for Distributed Geospatial Applications. Computer 2008, 41, 50–57. 82. Nebert, D.; Whiteside, A.; Vretanos, P. OpenGIS® Catalogue Services Specification portal.opengeospatial.org/files/?artifact_id=20555 (accessed Feb 21, 2014). 83. Vretanos, P. A. Web Feature Service Implementation Specification https://portal.opengeospatial.org/files/?artifact_id=8339 (accessed Jul 16, 2014). 84. Evans, J. D. Web Coverage Service (WCS) portal.opengeospatial.org/files/?artifact_id=3837 (accessed Apr 3, 2014). 85. Schut, P. OpenGIS Web Processing Service portal.opengeospatial.org/files/?artifact_id=28772&version=2 (accessed Aug 10, 2014). 86. de La Beaujardiere, J. Web Map Service portal.opengeospatial.org/files/?artifact_id=5316 (accessed Feb 1, 2014). 87. Botts, M.; Percivall, G.; Reed, C.; Davidson, J. OGC® Sensor Web Enablement: Overview and High Level Architecture. In GeoSensor Networks; Nittel, S.; Labrinidis, A.; Stefanidis, A., Eds.; Lecture Notes in Computer Science; Springer Berlin / Heidelberg, 2008; Vol. 4540, pp. 175–190. 88. Schäffer, B. Web Processing Service Best Practices Discussion Paper 2012. 89. Conover, H.; Berthiau, G.; Botts, M.; Goodman, H. M.; Li, X.; Lu, Y.; Maskey, M.; Regner, K.; Zavodsky, B. Using sensor web protocols for environmental data acquisition and management. Ecol. Inform. 2010, 5, 32–41. 90. OpenGIS® Sensor Model Language (SensorML) Implementation Specification; Botts, M.; Robin, A., Eds.; Open Geospatial Consortium Inc.: Wayland, MA, USA, 2007. 91. Henson, C. A.; Pschorr, J. K.; Sheth, A. P.; Thirunarayan, K. SemSOS: Semantic sensor Observation Service. In International Symposium on Collaborative Technologies and Systems, 2009. CTS ’09; 2009; pp. 44–53. 92. Na, A.; Priest, M. Sensor Observation Service http://portal.opengeospatial.org/files/?artifact_id=12846 (accessed Jul 2, 2014). 93. Observations and Measurements - XML Implementation http://portal.opengeospatial.org/files/?artifact_id=41510 (accessed Sep 11, 2014). 94. GPS.gov: Selective Availability http://www.gps.gov/systems/gps/modernization/sa/ (accessed Jun 24, 2016). 95. Djuknic, G. M.; Richton, R. E. Geolocation and assisted GPS. Computer 2001, 34, 123– 125. 96. Gruber, T. Collective knowledge systems: Where the Social Web meets the Semantic

183

Web. Web Semant. Sci. Serv. Agents World Wide Web 2008, 6, 4–13. 97. Lane, N. D.; Miluzzo, E.; Lu, H.; Peebles, D.; Choudhury, T.; Campbell, A. T. A survey of mobile phone sensing. IEEE Commun. Mag. 2010, 48, 140–150. 98. Goodchild, M. F. Citizens as Voluntary Sensors: Spatial Data Infrastructure in the World of Web 2.0. Int. J. Spat. Data Infrastruct. Res. 2007, 2, 24–32. 99. Jiang, B.; Thill, J.-C. Volunteered Geographic Information: Towards the establishment of a new paradigm. Comput. Environ. Urban Syst. 2015, 53, 1–3. 100. Haklay, M. How good is volunteered geographical information? A comparative study of OpenStreetMap and Ordnance Survey datasets. Environ. Plan. B Plan. Des. 2010, 37, 682 – 703. 101. Zielstra, D.; Zipf, A. A comparative study of proprietary geodata and volunteered geographic information for Germany. In 13th AGILE international conference on geographic information science; 2010; Vol. 2010. 102. Haklay, M. E.; Antoniou, V.; Basiouka, S.; Soden, R.; Mooney, P. Crowdsourced Geographic Information Use in Government; Global Facility for Disaster Reduction & Recovery (GFDRR), World Bank: London, UK, 2014; pp. 1–73. 103. Coleman, D. J.; Georgiadou, Y.; Labonte, J. Volunteered Geographic Information: The Nature and Motivation of Produsers. Int. J. Spat. Data Infrastruct. Res. 2009, 4, 332–358. 104. Beaudreau, P.; Johnson, P. A.; Sieber, R. E. Strategic Choices in Developing a Geospatial Web 2.0 Application for Rural Economic Development. J. Rural Community Dev. 2012, 7. 105. Flanagin, A. J.; Metzger, M. J. The credibility of volunteered geographic information. GeoJournal 2008, 72, 137–148. 106. Jaeger, P. T.; Shneiderman, B.; Fleischmann, K. R.; Preece, J.; Qu, Y.; Fei Wu, P. Community response grids: E-government, social networks, and effective emergency management. Telecommun. Policy 2007, 31, 592–604. 107. Kettl, D. F. System under Stress: The Challenge to 21st Century Governance; SAGE, 2013. 108. Wise, C. R. Organizing for Homeland Security after Katrina: Is Adaptive Management What’s Missing? Public Adm. Rev. 2006, 66, 302–318. 109. Granell, C.; Gould, M.; Manso, Á. M.; Bernabé, Á. B. Spatial Data Infrastructures. In Handbook of Research on Geoinformatics; Karimi, H. A., Ed.; Hershey: New York, 2009; pp. 36–41. 110. Annoni, A.; Bernard, L.; Douglas, J.; Greenwood, J.; Laiz, I.; Lloyd, M.; Sabeur, Z.; Sassen, A.-M.; Serrano, J.-J.; Usländer, T. Orchestra: Developing a Unified Open Architecture for Risk Management Applications. In Geo-information for Disaster Management; Oosterom, P. D. P. van; Zlatanova, D. S.; Fendel, E. M., Eds.; Springer Berlin Heidelberg, 2005; pp. 1–17. 111. Mansourian, A.; Rajabifard, A.; Valadan Zoej, M. J.; Williamson, I. Using SDI and web- based system to facilitate disaster management. Comput. Geosci. 2006, 32, 303–315.

184

112. Molina, M.; Bayarri, S. A multinational SDI-based system to facilitate disaster risk management in the Andean Community. Comput. Geosci. 2011, 37, 1501–1510. 113. Giuliani, G.; Peduzzi, P. The PREVIEW Global Risk Data Platform: a geoportal to serve and share global data on risk to natural hazards. Nat Hazards Earth Syst Sci 2011, 11, 53– 66. 114. Weiser, A.; Zipf, A. Web Service Orchestration of OGC Web Services for Disaster Management. In Geomatics Solutions for Disaster Management; Li, A. P. J.; Zlatanova, A. P. S.; Fabbri, P. A. G., Eds.; Lecture Notes in Geoinformation and Cartography; Springer Berlin Heidelberg, 2007; pp. 239–254. 115. Mazzetti, P.; Nativi, S.; Angelini, V.; Verlato, M.; Fiorucci, P. A Grid platform for the European Civil Protection e-Infrastructure: the Forest Fires use scenario. Earth Sci. Inform. 2009, 2, 53–62. 116. Agosto, E.; Dalmasso, S.; Pasquali, P.; Terzo, O. Ithaca worldwide flood alert system: the web framework. Appl. Geomat. 2011, 3, 83–89. 117. Bejar, R.; Latre, M. A.; Nogueras-Iso, J.; Muro-Medrano, P. R.; Zarazaga-Soria, F. J. An architectural style for spatial data infrastructures. Int. J. Geogr. Inf. Sci. 2009, 23, 271–294. 118. Zook, M.; Graham, M.; Shelton, T.; Gorman, S. Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake. World Med. Health Policy 2010, 2, 7–33. 119. Bugs, G.; Granell, C.; Fonts, O.; Huerta, J.; Painho, M. An assessment of Public Participation GIS and Web 2.0 technologies in urban planning practice in Canela, Brazil. Cities 2010, 27, 172–181. 120. McCall, M. Can Participatory GIS Strengthen Local-level Planning? Suggestions for Better Practice. In; Universiti Teknologi Malaysia, Johor Malaysia., 2004. 121. Schulz, A.; Paulheim, H. Mashups for the Emergency Management Domain. In Semantic Mashups; Endres-Niggemeyer, B., Ed.; Springer Berlin Heidelberg, 2013; pp. 237–260. 122. Li, L.; Goodchild, M. F. The Role of Social Networks in Emergency Management: A Research Agenda. Int. J. Inf. Syst. Crisis Response Manag. 2010, 2, 48–58. 123. Mooney, P.; Corcoran, P. Has OpenStreetMap a role in Digital Earth applications? Int. J. Digit. Earth 2014, 7, 534–553. 124. Okolloh, O. Ushahidi, or “testimony”: Web 2.0 tools for crowdsourcing crisis information. Particip. Learn. Action 2009, 59, 65–70. 125. De Longueville, B.; Smith, R. S.; Luraschi, G. “OMG, from Here, I Can See the Flames!”: A Use Case of Mining Location Based Social Networks to Acquire Spatio-temporal Data on Forest Fires. In Proceedings of the 2009 International Workshop on Location Based Social Networks; LBSN ’09; ACM: New York, NY, USA, 2009; pp. 73–80. 126. Genovese, E.; Roche, S. Potential of VGI as a resource for SDIs in the North/South context. Geomatica 2010, 64, 439–450.

185

127. Olson, J. A. Data as a Service: Are We in the Clouds? J. Map Geogr. Libr. 2009, 6, 76–78. 128. Wei-dong, C.; Qing-zhan, Z. Research of GIS services mode based on SaaS. In 2010 2nd International Conference on Information Science and Engineering (ICISE); 2010; pp. 6812– 6814. 129. Castronova, A. M.; Goodall, J. L.; Elag, M. M. Models as web services using the Open Geospatial Consortium (OGC) Web Processing Service (WPS) standard. Environ. Model. Softw. 2013, 41, 72–83. 130. Sahin, K.; Gumusay, M. U. Service oriented architecture (SOA) based web services for geographic information systems. In XXIst ISPRS Congress. Beijing; Chen, J.; Jiang, J.; Baudoin, A., Eds.; ISPRS International Archives of Photogrammetry, Remote Sensing, and Spatial Information Science, 2008; pp. 625–630. 131. Foster, I.; Zhao, Y.; Raicu, I.; Lu, S. Cloud Computing and Grid Computing 360-Degree Compared. In Grid Computing Environments Workshop, 2008. GCE ’08; 2008; pp. 1–10. 132. Arcaini, P.; Bordogna, G.; Sterlacchini, S. Flexible Querying of Volunteered Geographic Information for Risk Management. In Advances in Intelligent Systems Research; Atlantis Press: Milan, Italy, 2013. 133. Brin, S.; Page, L. The anatomy of a large-scale hypertextual Web search engine. Comput. Netw. ISDN Syst. 1998, 30, 107–117. 134. Mandl, T. Implementation and Evaluation of a Quality-based Search Engine. In Proceedings of the Seventeenth Conference on Hypertext and Hypermedia; HYPERTEXT ’06; ACM: New York, NY, USA, 2006; pp. 73–84. 135. Poser, K.; Dransch, D. Volunteered geographic information for disaster management with application to rapid flood damage estimation. Geomatica 2010, 64, 89–98. 136. Girres, J.-F.; Touya, G. Quality Assessment of the French OpenStreetMap Dataset. Trans. GIS 2010, 14, 435–459. 137. Jackson, S.; Mullen, W.; Agouris, P.; Crooks, A.; Croitoru, A.; Stefanidis, A. Assessing Completeness and Spatial Error of Features in Volunteered Geographic Information. ISPRS Int. J. Geo-Inf. 2013, 2, 507–530. 138. Fan, H.; Zipf, A.; Fu, Q.; Neis, P. Quality assessment for building footprints data on OpenStreetMap. Int. J. Geogr. Inf. Sci. 2014, 28, 700–719. 139. De Brito Moreira, R.; Castro Degrossi, L.; De Albuquerque, J. P. An experimental evaluation of a crowdsourcing-based approach for flood risk management. In; Lima, Peru, 2015. 140. Antoniou, V.; Skopeliti, A. Measures and Indicators of VGI Quality: An Overview. In ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences; 2015; Vol. II-3/W5, pp. 345–351. 141. Bishr, M.; Mantelas, L. A trust and reputation model for filtering and classifying knowledge about urban growth. GeoJournal 2008, 72, 229–237.

186

142. Bishr, M.; Kuhn, W. Trust and Reputation Models for Quality Assessment of Human Sensor Observations. In Spatial Information Theory; Tenbrink, T.; Stell, J.; Galton, A.; Wood, Z., Eds.; Lecture Notes in Computer Science; Springer International Publishing, 2013; pp. 53–73. 143. Goodchild, M. F.; Li, L. Assuring the quality of volunteered geographic information. Spat. Stat. 2012, 1, 110–120. 144. Bodnar, T.; Salathé, M. Validating Models for Disease Detection Using Twitter. In Proceedings of the 22Nd International Conference on World Wide Web Companion; WWW ’13 Companion; International World Wide Web Conferences Steering Committee: Republic and Canton of Geneva, Switzerland, 2013; pp. 699–702. 145. Achrekar, H.; Gandhe, A.; Lazarus, R.; Yu, S.-H.; Liu, B. Predicting Flu Trends using Twitter data. In 2011 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS); 2011; pp. 702–707. 146. Ostermann, F. O.; Spinsanti, L. A conceptual workflow for automatically assessing the quality of volunteered geographic information for crisis management. In Proceedings of AGILE; University of Utrecht, Utrecht, Netherlands, 2011; Vol. 2011. 147. Bröring, A.; Jirka, S.; Rieke, M.; Pross, B. OGC® Testbed-10 CCI VGI Engineering Report 2014. 148. Cornford, D.; Williams, M.; Bastin, L. Uncertainty enabled Sensor Observation Services. In; 2010; Vol. 12, p. 9082. 149. Devaraju, A.; Kunkel, R.; Sorg, J.; Bogena, H.; Vereecken, H. Enabling Quality Control of Sensor Web Observations. In Proceedings of the 3rd International Conference on Sensor Networks (SENSORNETS 2014); Lisboa, Portugal, 2014. 150. Williams, M.; Cornford, D.; Bastin, L.; Pebesma, E. Uncertainty Markup Language (UnCertML) https://portal.opengeospatial.org/files/?artifact_id=33234 (accessed May 20, 2015). 151. Barbacci, M. R.; Ellison, R. J.; Lattanze, A. J.; Stafford, J. A.; Weinstock, C. B.; Wood, W. G. Quality Attribute Workshops (QAWs), Third Edition; Software Engineering Institute, Carnegie Mellon University: Pittsburgh, Pennsylvania, 2003. 152. Kazman, R.; Bass, L. Categorizing Business Goals for Software Architectures; Software Engineering Institute, Carnegie Mellon University: Pittsburgh, Pennsylvania, 2005. 153. Kazman, R.; Gagliardi, M.; Wood, W. Scaling up software architecture analysis. J. Syst. Softw. 2012, 85, 1511–1519. 154. Olteanu, A.; Castillo, C.; Diaz, F.; Vieweg, S. CrisisLex: A lexicon for collecting and filtering microblogged communications in crises. In In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media (ICWSM" 14); Ann Arbor, ML, USA, 2014. 155. Burton, S. H.; Tanner, K. W.; Giraud-Carrier, C. G.; West, J. H.; Barnes, M. D. “Right time, right place” health communication on Twitter: value and accuracy of location information. J. Med. Internet Res. 2012, 14, e156. 187

156. Devillers, R.; Jeansoulin, R. Spatial Data Quality: Concepts. In Fundamentals of Spatial Data Quality; Devillers, R.; Jeansoulin, R., Eds.; ISTE, 2006; pp. 31–42. 157. Guptill, S. C.; Morrison, J. L. Elements of Spatial Data Quality; Elsevier, 2013. 158. Shi, W.; Fisher, P.; Goodchild, M. F. Spatial Data Quality; CRC Press, 2003. 159. Haklay, M. Volunteered geographic information, quality assurance. Int. Encycl. Geogr. People Earth Environ. Technol. Forthcoming. 160. Craglia, M.; Shanley, L. Data democracy – increased supply of geospatial information and expanded participatory processes in the production of data. Int. J. Digit. Earth 2015, 8, 679–693. 161. Noulas, A.; Scellato, S.; Mascolo, C.; Pontil, M. An Empirical Study of Geographic User Activity Patterns in Foursquare. In Fifth International AAAI Conference on Weblogs and Social Media; 2011. 162. Wallgrün, J. O.; Hardisty, F.; MacEachren, A. M.; Karimzadeh, M.; Ju, Y.; Pezanowski, S. Construction and First Analysis of a Corpus for the Evaluation and Training of Microblog/Twitter Geoparsers. In Proceedings of the 8th Workshop on Geographic Information Retrieval; GIR ’14; ACM: New York, NY, USA, 2014; p. 4:1–4:8. 163. Munro, R. Subword and Spatiotemporal Models for Identifying Actionable Information in Haitian Kreyol. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning; CoNLL ’11; Association for Computational Linguistics: Stroudsburg, PA, USA, 2011; pp. 68–77. 164. Gelernter, J.; Balaji, S. An algorithm for local geoparsing of microtext. GeoInformatica 2013, 17, 635–667. 165. Crooks, A.; Croitoru, A.; Stefanidis, A.; Radzikowski, J. #Earthquake: Twitter as a Distributed Sensor System. Trans. GIS 2013, 17, 124–147. 166. Gupta, A.; Kumaraguru, P. Credibility Ranking of Tweets During High Impact Events. In Proceedings of the 1st Workshop on Privacy and Security in Online Social Media; PSOSM ’12; ACM: New York, NY, USA, 2012; p. 2:2–2:8. 167. Albuquerque, J. P. de; Herfort, B.; Brenning, A.; Zipf, A. A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management. Int. J. Geogr. Inf. Sci. 2015, 29, 667–689. 168. Gelernter, J.; Mushegian, N. Geo-parsing Messages from Microtext. Trans. GIS 2011, 15, 753–773. 169. Derczynski, L.; Maynard, D.; Rizzo, G.; van Erp, M.; Gorrell, G.; Troncy, R.; Petrak, J.; Bontcheva, K. Analysis of named entity recognition and linking for tweets. Inf. Process. Manag. 2015, 51, 32–49. 170. Ritter, A.; Clark, S.; Etzioni, O.; others Named entity recognition in tweets: an experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics, 2011; pp. 1524–1534.

188

171. Liu, X.; Zhou, M.; Wei, F.; Fu, Z.; Zhou, X. Joint inference of named entity recognition and normalization for tweets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1; Association for Computational Linguistics, 2012; pp. 526–535. 172. Finkel, J. R.; Grenager, T.; Manning, C. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics; Association for Computational Linguistics, 2005; pp. 363–370. 173. Mendes, P. N.; Jakob, M.; García-Silva, A.; Bizer, C. DBpedia spotlight: shedding light on the web of documents. In Proceedings of the 7th International Conference on Semantic Systems; ACM, 2011; pp. 1–8. 174. Chow, T. E.; Dede-Bamfo, N.; Dahal, K. R. Geographic disparity of positional errors and matching rate of residential addresses among geocoding solutions. Ann. GIS 2016, 22, 29– 42. 175. Delmelle, A. P. E.; Páez, A. P. A.; Kanaroglou, P. P. Spatial Analysis in Health Geography; Ashgate Publishing, Ltd., 2015. 176. Roongpiboonsopit, D.; Karimi, H. A. Comparative evaluation and analysis of online geocoding services. Int. J. Geogr. Inf. Sci. 2010, 24, 1081–1100. 177. Goldberg, D. W.; Ballard, M.; Boyd, J. H.; Mullan, N.; Garfield, C.; Rosman, D.; Ferrante, A. M.; Semmens, J. B. An evaluation framework for comparing geocoding systems. Int. J. Health Geogr. 2013, 12, 50. 178. Roongpiboonsopit, D.; Karimi, H. A. Quality Assessment of Online Street and Rooftop Geocoding Services. Cartogr. Geogr. Inf. Sci. 2010, 37, 301–318. 179. Xu, C.; Wong, D. W.; Yang, C. Evaluating the “geographical awareness” of individuals: an exploratory analysis of twitter data. Cartogr. Geogr. Inf. Sci. 2013, 40, 103–115. 180. The Google Maps Geocoding API https://developers.google.com/maps/documentation/geocoding/intro (accessed Mar 10, 2016). 181. Qu, Y.; Huang, C.; Zhang, P.; Zhang, J. Microblogging After a Major Disaster in China: A Case Study of the 2010 Yushu Earthquake. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work; CSCW ’11; ACM: New York, NY, USA, 2011; pp. 25– 34. 182. Huang, Q.; Xiao, Y. Geographic Situational Awareness: Mining Tweets for Disaster Preparedness, Emergency Response, Impact, and Recovery. ISPRS Int. J. Geo-Inf. 2015, 4, 1549–1568. 183. Verma, S.; Vieweg, S.; Corvey, W. J.; Palen, L.; Martin, J. H.; Palmer, M.; Schram, A.; Anderson, K. M. Natural Language Processing to the Rescue? Extracting “Situational Awareness” Tweets During Mass Emergency. In Fifth International AAAI Conference on Weblogs and Social Media; 2011.

189

184. Steiger, E.; Resch, B.; Zipf, A. Exploration of spatiotemporal and semantic clusters of Twitter data using unsupervised neural networks. Int. J. Geogr. Inf. Sci. 2015, 0, 1–23. 185. Hahmann, S.; Purves, R.; Burghardt, D. Twitter location (sometimes) matters: Exploring the relationship between georeferenced tweet content and nearby feature classes. J. Spat. Inf. Sci. 2014. 186. Baeza-Yates, R.; Ribeiro-Neto, B. Modern Information Retrieval: The Concepts and Technology Behind Search; Addison Wesley, 2011. 187. Imran, M.; Elbassuoni, S.; Castillo, C.; Diaz, F.; Meier, P. Extracting information nuggets from disaster-related messages in social media. In; 2013. 188. Zhou, X.; Zhang, L. Crowdsourcing functions of the living city from Twitter and Foursquare data. Cartogr. Geogr. Inf. Sci. 2016, 0, 1–12. 189. Hashemi, S. M.; Sayahi, A.; Nejad, M. B. Evaluation of Naïve Bayes and J48 Learning Algorithm for Text Classification. Int. J. Comput. Sci. Netw. Solut. 2013, 1. 190. Rokach, L. Ensemble-based classifiers. Artif. Intell. Rev. 2010, 33, 1–39. 191. Li, Y.; McLean, D.; Bandar, Z. A.; O’Shea, J. D.; Crockett, K. Sentence similarity based on semantic nets and corpus statistics. IEEE Trans. Knowl. Data Eng. 2006, 18, 1138–1150. 192. Miller, G. A.; Charles, W. G. Contextual correlates of semantic similarity. Lang. Cogn. Process. 1991, 6, 1–28. 193. Miller, G. A. WordNet: A Lexical Database for English. Commun ACM 1995, 38, 39–41. 194. Gardner, W.; Mulvey, E. P.; Shaw, E. C. Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychol. Bull. 1995, 118, 392–404. 195. Granell, C. Robust workflow systems + flexible geoprocessing services = geo-enabled model web? In Geographical Information Systems: Trends and Technologies; Pourabbas, E., Ed.; CRC Press: Boca Raton, 2014; pp. 172–204. 196. Schade, S.; Ostländer, N.; Granell, C.; Schulz, M.; McInerney, D.; Dubois, G.; Vaccari, L.; Chinosi, M.; Díaz, L.; Bastin, L.; Jones, R. Which service interfaces fit the model Web? In; Rückemann, C.-P.; Resch, B., Eds.; IARIA XPS Press: Valencia, Spain, 2012; p. 6. 197. Ioup, E.; Lin, B.; Sample, J.; Shaw, K.; Rabemanantsoa, A.; Reimbold, J. Geospatial Web Services : Bridging the Gap Between OGC and Web Services. In Geospatial Services and Applications for the Internet; Sample, J.; Shaw, K.; Tu, S.; Abdelguerfi, M., Eds.; Springer US, 2008; pp. 72–92. 198. Amirian, P.; Alesheikh, A. A.; Bassiri, A. Standards-based, interoperable services for accessing urban services data for the city of Tehran. Comput. Environ. Urban Syst. 2010, 34, 309–321. 199. Di Giovanni, P.; Bertolotto, M.; Vitiello, G.; Sebillo, M. Web Services Composition and Geographic Information. In Geographical Information Systems: Trends and Technologies; Pourabbas, E., Ed.; CRC Press, 2014; pp. 104–141.

190

200. Zhang, D.; Coddington, P.; Wendelborn, A. Binary Data Transfer Performance over High-Latency Networks Using Web Service Attachments. In e-Science and Grid Computing; Fox, G.; Chiu, K.; Buyya, R., Eds.; IEEE: Bangalore, India, 2007; pp. 261–269. 201. Gartmann, R.; Schäffer, B. OpenGIS Wrapping OGC HTTP-GET and -POST Services with SOAP 2008. 202. OGC OGC Newsletter http://www.opengeospatial.org/pressroom/newsletters/200607/#C2 (accessed Jul 25, 2014). 203. Vretanos, P. A. OpenGIS Web Feature Service 2.0 Interface Standard 2010. 204. Baumann, P. OGC Web Coverage Service 2.0 Interface Standard - XML/SOAP Protocol Binding Extension 2010. 205. Bröring, A.; Stasch, C.; Echterhoff, J. OGC ® Sensor Observation Service Interface Standard; Open Geospatial Consortium Inc., 2012. 206. Rice, D. Super Typhoon Hagupit takes aim on the Philippines http://www.usatoday.com/story/weather/2014/12/03/super-typhoon-hagupit- philippines/19849821/ (accessed Feb 9, 2015). 207. Typhoon Hagupit 2014 http://en.wikipedia.org/w/index.php?title=Typhoon_Hagupit_(2014)&oldid=644324125 (accessed Jan 27, 2015). 208. Pama, A. SitRep No. 27 re Effects of Typhoon “Ruby” (Hagupit); National Disaster Risk Reduction and Management Council: Quezon City, Philippines, 2014; p. 42. 209. Official Gazette Weather bulletin no. 25 (FINAL), tropical cyclone warning: Tropical storm Ruby, issued at 10:30 p.m., December 10, 2014; National Printing Office: Philippines, 2014. 210. Monteiro, B. R.; Davis Jr., C. A.; Fonseca, F. A survey on the geographic scope of textual documents. Comput. Geosci. 2016, 96, 23–34. 211. Daiber, J.; Jakob, M.; Hokamp, C.; Mendes, P. N. Improving Efficiency and Accuracy in Multilingual Entity Extraction. In Proceedings of the 9th International Conference on Semantic Systems; I-SEMANTICS ’13; ACM: New York, NY, USA, 2013; pp. 121–124. 212. Japkowicz, N.; Shah, M. Evaluating Learning Algorithms: A Classification Perspective; Cambridge University Press, 2011. 213. James, G.; Witten, D.; Hastie, T.; Tibshirani, R. An Introduction to Statistical Learning; Springer Texts in Statistics; Springer New York: New York, NY, 2013; Vol. 103. 214. Fernández-Delgado, M.; Cernadas, E.; Barro, S.; Amorim, D. Do We Need Hundreds of Classifiers to Solve Real World Classification Problems? J Mach Learn Res 2014, 15, 3133– 3181. 215. Witten, I. H.; Frank, E.; Hall, M. A. Data Mining: Practical Machine Learning Tools and Techniques, Third Edition; 3 edition.; Morgan Kaufmann: Burlington, MA, 2011.

191

216. Ellis, R.; Gray, M. Thousands evacuate as Typhoon Hagupit lashes Philippines http://www.cnn.com/2014/12/06/world/asia/philippines-typhoon-hagupit- ruby/index.html (accessed Feb 9, 2015). 217. Li, M.; Chen, X.; Li, X.; Ma, B.; Vitanyi, P. M. B. The similarity metric. IEEE Trans. Inf. Theory 2004, 50, 3250–3264. 218. Chan, C. K.; Vasardani, M.; Winter, S. Leveraging Twitter to detect event names associated with a place. J. Spat. Sci. 2014, 59, 137–155. 219. Yuill, R. S. The Standard Deviational Ellipse; An Updated Tool for Spatial Description. Geogr. Ann. Ser. B Hum. Geogr. 1971, 53, 28–39. 220. Dahle, B.; Swenson, J. E. Home ranges in adult Scandinavian brown bears (Ursus arctos): effect of mass, sex, reproductive category, population density and habitat type. J. Zool. 2003, 260, 329–335. 221. Steiniger, S.; Hunter, A. J. S. A scaled line-based kernel density estimator for the retrieval of utilization distributions and home ranges from GPS movement tracks. Ecol. Inform. 2013, 13, 1–8. 222. Moe, W. W.; Schweidel, D. A. Online Product Opinions: Incidence, Evaluation, and Evolution. Mark. Sci. 2012, 31, 372–386. 223. Romero, D. M.; Meeder, B.; Kleinberg, J. Differences in the Mechanics of Information Diffusion Across Topics: Idioms, Political Hashtags, and Complex Contagion on Twitter. In Proceedings of the 20th International Conference on World Wide Web; WWW ’11; ACM: New York, NY, USA, 2011; pp. 695–704. 224. Saaty, T. L. The analytic hierarchy process; McGraw-Hill: New York, NY, 1980. 225. Currion, P.; Silva, C. de; Van de Walle, B. Open Source Software for Disaster Management. Commun ACM 2007, 50, 61–65. 226. Steiniger, S.; Hunter, A. J. S. The 2012 Free and Open Source GIS Software Map – A Guide to facilitate Research, Development and Adoption. Comput. Environ. Urban Syst. 2013, 39, 136–150. 227. Poorazizi, M. E.; Hunter, A. J. S. Evaluation of Web Processing Service Frameworks. OSGeo J. 2015, 14, 29–42. 228. Poorazizi, M. E.; Liang, S. H. L.; Hunter, A. J. S. Testing of sensor observation services: a performance evaluation. In Proceedings of the First ACM SIGSPATIAL Workshop on Sensor Web Enablement; SWE ’12; ACM: Redondo Beach, CA, USA., 2012; pp. 32–38. 229. Granell, C.; Ostermann, F. O. Beyond data collection: Objectives and methods of research using VGI and geo-social media for disaster management. Comput. Environ. Urban Syst. 230. Haworth, B.; Bruce, E. A Review of Volunteered Geographic Information for Disaster Management. Geogr. Compass 2015, 9, 237–250. 231. Nogueras-Iso, J.; Zarazaga-Soria, F. J.; Béjar, R.; Álvarez, P. J.; Muro-Medrano, P. R. OGC

192

Catalog Services: a key element for the development of Spatial Data Infrastructures. Comput. Geosci. 2005, 31, 199–209. 232. Poorazizi, M. E.; Steiniger, S.; Hunter, A. J. S. A service-oriented architecture to enable participatory planning: an e-planning platform. Int. J. Geogr. Inf. Sci. 2015, 29, 1081–1110. 233. Pan, S. J.; Yang, Q. A Survey on Transfer Learning. IEEE Trans. Knowl. Data Eng. 2010, 22, 1345–1359.

193