Fusing Social Media and Traditional Traffic Data for Advanced Traveler Information and Travel Behavior Analysis by Zhenhua Zhang January 15, 2017 A dissertation submitted to the Faculty of the Graduate School of the University at Buffalo, the State University of New York in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Civil, Structural, and Environmental Engineering ACKNOWLEDGEMENTS First, I would like to offer my greatest gratitude to my supervisor, Dr. Qing He, who has supported me throughout my research studies showing their patience and knowledge. His continuous encouragement and guidance on my research works in the past three and a half years help me with useful suggestions to make research plans. Without his help, this dissertation would not have been possible. At the same time, I would give my thankfulness to the committee members: Dr. Adel W. Sadek and Dr. Qian Wang, who accept the invitation to serve on my committee and give valuable devices on my dissertation. Besides, thank them a lot for teaching me a lot of useful knowledge about transportation modelling and demand forcasting in their class. In addition, I thank for the research environment provided by the University at Buffalo in which the students can devote themselves to the research. I can concentrate on my studies all the time without worries about affairs, which are irrelevant to research. Thank very much Ming Ni, who provides the social media data for my research. Thank my friends Lei Lin, Yu Cui and Li Tang who helped me a lot in my previous studies. Finally, thank my family for giving me the material and spiritual help. Their unconditioned care always pushes me forward. Hope they would be happy for my Ph.D. graduation. ii TABLE OF CONTENTS ACKNOWLEDGEMENTS ................................................................................................ II LIST OF FIGURES .......................................................................................................... IX LIST OF TABLES ........................................................................................................... XV ABSTRACT ........................................................................................................................ 1 INTRODUCTION ........................................................................................ 5 1.1 Research background and motivations ................................................................ 5 1.1.1 Research background .................................................................................... 5 1.1.2 Motivations ................................................................................................... 8 1.2 Data description................................................................................................... 9 1.2.1 Loop detector data ........................................................................................ 9 1.2.2 Social media data ........................................................................................ 11 1.2.3 Traffic accident data ................................................................................... 13 1.2.4 Land use data and Google Place Type data ................................................ 14 1.2.5 Trip data of connected vehicle.................................................................... 14 1.4 Contributions ..................................................................................................... 15 1.5 Dissertation organization................................................................................... 18 LITERATURE REVIEW ........................................................................... 21 2.1 Regional traffic flow pattern identification ....................................................... 21 2.1.1 Gaps in large-scale traffic pattern identification ........................................ 21 iii 2.1.2 Pattern identification and anomaly detection ............................................. 22 2.1.3 Dimensionality reductions of traffic data ................................................... 25 2.2 Social media definitions and features................................................................ 26 2.3 Travel behavior and travel motivation identification ........................................ 30 2.3.1 Mobility features of travel behavior based on Twitter ............................... 33 2.2.3 Travel motivation identification ................................................................. 36 2.2.4 Research gaps and opportunities ................................................................ 38 2.3 Traffic accident and social event detection ....................................................... 40 2.3.1 Traffic accident detection based on social media ....................................... 40 2.3.2 Traffic surge and social event detection based on social media ................. 45 2.4 Trip purpose inference ...................................................................................... 49 SPATIAL-TEMPORAL TRAFFIC FLOW PATTERN IDENTIFICATION AND ANOMALY DETECTION ..................................................................................... 55 3.1 Data preprocessing ............................................................................................ 55 3.2 Method .............................................................................................................. 58 3.2.1 Dictionary-based compression ................................................................... 58 3.2.2 Minimum Description Length principle ..................................................... 61 3.2.3 Anomaly degree .......................................................................................... 63 3.3 Numerical examples .......................................................................................... 67 3.3.1 Spatial regional traffic flow pattern ............................................................ 67 iv 3.3.2 Temporal regional traffic flow pattern ....................................................... 72 3.4 A case study ...................................................................................................... 76 3.5 Comparisons ...................................................................................................... 81 3.5.1 Comparisons between different discretization settings .............................. 81 3.5.2 Comparisons with another anomaly detection method............................... 84 3.6 Conclusions and discussions ............................................................................. 86 EXPLORING TRAVEL BEHAVIOR: ABNORMAL MOVEMENTS USING HIGH-RESOLUTION TWEET TRAJECTORY ................................................ 89 4.1 Empirical findings of tweet trajectory ............................................................... 89 4.1.1 Location features ........................................................................................ 89 4.1.2 Movement features ..................................................................................... 92 4.1.3 Clustering features ...................................................................................... 96 4.2 Geo-Mobility method ........................................................................................ 98 4.2.1 Geo clustering ............................................................................................. 98 4.2.2 Abnormal movement detection ................................................................ 103 4.3 Travel motivation decoding ............................................................................ 106 4.4 Conclusions and discussions ........................................................................... 109 THE HUMAN MOBILITY PATTERN STUDY BASED ON SOCIAL MEDIA ........................................................................................................................... 112 5.1 Human mobility study ..................................................................................... 112 v 5.1.1 Twitter displacement and human mobility patterns ................................. 112 5.1.2 Long-distance Twitter displacement ........................................................ 118 5.2 Travel motivation inference ............................................................................ 121 5.2.1 Topic generation ....................................................................................... 121 5.2.2 Topic tokens and their interpretations ...................................................... 125 5.3 Applications and discussions .......................................................................... 129 5.3.1 Individual mobility pattern and travel motivation identification.............. 130 5.3.2 Social event influence identification ........................................................ 132 5.4 Conclusions and discussions ........................................................................... 135 TRAFFIC ACCIDENT DETECTION WITH BOTH SOCIAL MEDIA AND TRAFFIC DATA ............................................................................................................ 138 6.1 Data preprocessing .......................................................................................... 138 6.1.1 Tweet data preprocessing ......................................................................... 138 6.1.2 Token filtering and stemming..................................................................