Czech Technical University in Prague Faculty of Electrical Engineering DIPLOMA THESIS Vojtˇech Kˇr´ıˇzek Intelligent Visualization of data from Multi-agent Simulations Supervisor: Ing. Ondˇrej Vanˇek Department of Cybernetics Prague, 2012 Prohl´aˇsen´ı Prohlaˇsuji,ˇzejsem pˇredloˇzenoupr´acivypracoval samostatnˇea ˇzejsem uvedl veˇsker´epouˇzit´einformaˇcn´ızdroje v souladu s Metodick´ympokynem o dodrˇzov´an´ı etick´ych princip˚upˇripˇr´ıpravˇevysokoˇskolsk´ych z´avˇereˇcn´ych prac´ı. V Praze dne 9. 5. 2012. ::::::::::::::::::::: podpis Acknowledgements I'd like to thank to Ing. OndˇrejVanˇekand Ing. Michal Jakob, Ph.D. for en- abling me to do the work in the web applications field. I'd like also to thank Bc. OndˇrejHrstka for helping me with AgentC platform setup. Finally thanks to Ing. Jan Faigl for providing me LATEX template. Abstract The thesis deals with design and implementation of a web-based application that is used for visualization of large-scale data from multi-agent simulations. An overview of related applications for web-based visualizations and of available technologies is presented. From this overview, system architecture, consisting of modular filter framework on the server side, and utilizing jQuery framework on the client side, is proposed. Filter framework uses K-means, agglomera- tive hierarchical clustering, line clustering, and kernel density estimation algo- rithms, which enable fast-response visualization in a web browser. Static and dynamic visualizations are supported, therefore whole simulation can be re- played. The application is tested on different data sets to evaluate performance and accuracy. Abstrakt Tato diplomov´apr´acese zab´yv´an´avrhem a implementac´ı webov´eaplikace, kter´a vizualizuje velk´e objemy dat z multiagentn´ıch simulac´ı. Je vypra- cov´anpˇrehledpodobn´ych aplikac´ıpro webovou vizualizaci a dostupn´ych tech- nologi´ıch. Na z´akladˇetohoto pˇrehleduje vytvoˇrenn´avrhsyst´emov´earchitek- tury, kter´ase skl´ad´az modul´arn´ıhofiltrovac´ıhoframeworku na serverov´estranˇe a jQuery frameworku na klientsk´estranˇe. Filtrovac´ıframework pouˇz´ıv´aalgo- ritmy K-stˇred˚u,aglomerativn´ıhierarchick´eseskupov´an´ı,ˇc´arov´eseskupov´an´ıa odhad j´adrov´ehustoty, kter´eumoˇzˇnuj´ıwebovou vizualizaci velk´ehomnoˇzstv´ı dat s rychlou odezvou. Je podporov´anastatick´ai dynamick´avizualizace, a proto je moˇzn´eznovu pˇrehr´atuloˇzenousimulaci. V´ykonnost a pˇresnostap- likace je otestov´ana pomoc´ır˚uzn´ych datov´ych soubor˚u. Contents 1 Introduction 1 2 Problem 3 2.1 Specification . .4 2.1.1 Assigned Use Cases . .4 2.2 Requirements . .5 2.3 Related Work . .6 2.3.1 General Concept . .6 2.3.2 Implemented Systems and Frameworks . .7 2.3.3 Technologies . .9 3 Analysis 11 3.1 Multi-agent Simulation Overview . 11 3.1.1 AgentC and AgentPolis . 12 3.2 Evaluation of Available Technologies and Tools . 13 3.2.1 Data Formats Comparison . 13 3.2.2 Databases Comparison . 14 3.2.3 Javascript Frameworks Comparison . 14 3.2.4 Map Frameworks Comparison . 16 3.2.5 Web Technologies . 16 i 4 Solution 19 4.1 Selected Technologies and Libraries . 19 4.2 System Architecture . 20 4.2.1 Database Schema . 20 4.2.2 Input Format . 23 4.2.3 Server Architecture . 24 4.2.4 Data Processing . 27 4.2.5 Client Structure . 31 4.3 Implemented Algorithms . 34 4.3.1 Map Projection System . 34 4.3.2 K-means . 37 4.3.3 Line Clustering Algorithm . 38 4.3.4 Agglomerative Hierarchical Clustering . 39 4.3.5 Heat Map Generation . 41 5 Evaluation 43 5.1 Performance . 43 5.1.1 Random Data . 44 5.1.2 Real Data . 47 5.2 Line Clustering Algorithm Evaluation . 48 5.3 Technical Problems and Limitations . 50 6 Conclusion 51 Appendix A Code Listing 59 Appendix B User Guide 65 Appendix C Contents of the CD 83 ii List of Tables 3.1 Fields of Every Event . 12 3.2 Fields of Some Events . 12 3.3 Comparison of Data Formats . 13 3.4 Comparison of Databases . 14 3.5 Comparison of Javascript Frameworks . 15 3.6 Comparison of Map Frameworks . 16 4.1 Specified Input Format Fields . 23 4.2 Web Server Structure . 27 5.1 Size Comparison of Testing Data Files . 45 5.2 Speed Comparison of Points of Testing Data Files . 45 5.3 Speed Comparison of Traces of Testing Data Files . 46 5.4 Size Comparison of Real Data Files . 47 5.5 Speed Comparison of Points of Real Data Files . 47 5.6 Speed Comparison of Traces of Real Data Files . 48 5.7 Comparison of Line Clustering Algorithm { Points . 48 5.8 Comparison of Line Clustering Algorithm { Lines . 49 5.9 Comparison of Line Clustering Algorithm { Distance . 49 C.1 CD Path Structure . 83 iii List of Figures 2.1 Google Fusion Tables { Map Visualization of Imported Data . 10 4.1 System Architecture Diagram . 21 4.2 Database Schema { E-R Diagram . 22 4.3 Filter Architecture { Object Model and Data Flow . 25 4.4 Screenshot { Static Data Visualization { Map Layer with traces, the heat map and events (left) and Chart of events by days of the week (right) . 28 4.5 Screenshot { Dynamic Data Visualization { Vessel traces and Events 30 4.6 Diagram of Page Rendering in a Web Browser . 32 4.7 Diagram of Data Flow After User's Action . 33 4.8 Mercator Projection { Illustration of cylindric projection of Earth (based on [49]) . 34 4.9 Screenshot { Mercator Projection Example . 35 4.10 Trace Interpolation Diagram . 37 4.11 Centroids Initialization Points { Map Tile in Mercator Projection . 37 4.12 Screenshot { Generated Heat Map Visualization . 42 5.1 Visualization of Speed Comparison of Points Data . 45 5.2 Visualization of Speed Comparison of Traces Data . 46 iv List of Algorithms 2.1 Syslog Example (Apache HTTP Server) [46] . 10 4.1 Web Page Initialization Pseudocode . 31 4.2 Line Clustering Algorithm Pseudocode . 38 4.3 Agglomerative Hierarchical Clustering Pseudocode . 40 A.1 Input Data Format demonstration code in JSON formatting . 59 A.2 SQL Database Tables Generation Script . 60 v List of Abbreviations Abbreviation Description AJAX Asynchronous JavaScript and XML CSS Cascading Style Sheets CSV Comma-Separated Values GIS Geographical Information System GML Geography Markup Language GUI Graphical User Interface HTML HyperText Markup Language HTTP HyperText Transfer Protocol ISO International Organization for Standardization JDBC Java Database Connectivity JSON JavaScript Object Notation KML Keyhole Markup Language PHP PHP: Hypertext Preprocessor (recursive acronym) RDF Resource Description Framework REST Representational State Transfer RSS RDF Site Summary SDK Software Development Kit SVG Scalable Vector Graphics URL Uniform Resource Locator WKT Well-Known Text WMS Web Map Service XML Extensible Markup Language vi Chapter 1 Introduction Multi-agent simulations produce large amount of data which mostly contain a ge- ographical position or a trace. Data are usually visualized directly from a simulation, but there do not exist reasonable general visualization platforms which can compare simulation runs against each other or even reconstruct simulation flow. Web technologies have experienced huge technological shift last years. Now it is possible to play 3D games, render PDF documents or run offline applications without any extensions directly in a web browser. Application market is also being changed rapidly in these days which is noticeable on the growing segment of online applica- tions that were being used on a desktop before. An example of such applications are e-mail clients, office suites or games. Huge performance improvement of web browsers and multi-platform availability of web technologies are main motivators to build a web-based application which can process and visualize recorded multi-agent data. The application is then easily usable, because of well known web interface, and it can be run on any Java-compatible operating system. Specific problem of the work is to process and visualize large amount of data which contain geo-spatial information. Data need to be filtered and aggregated before visualization to produce relevant results and to reduce amount of input data. Web browsers need to be investigated on technical limits of visualization of these data too. It is also needed to find a way to export visualized data in a web browser for the future use because currently web browsers do not support export of vector graphics. 1 The result of the work is technology overview, algorithms for intelligent data pro- cessing and implementation of a web-based application that does static and dynamic visualization of entities on a map and chart visualization of aggregated entities. The application also supports export of visualized map or chart. The main challenge was to visualize huge amount of line segments. Therefore a line clustering algorithm is proposed, which significantly compacts vessel traces into a limited amount of lines that can be reasonably visualized in a web browser. Chapter 2 describes work specification and assigns use cases. Chapter 3 analyzes the problem, related work and available technologies. Chapter 4 describes used tech- nologies, system architecture and implemented algorithms. Chapter 5 evaluates the work and measures performance of the application. Chapter 6 concludes the work. 2 Chapter 2 Problem This chapter describes work specification (Section 2.1) and requirements related to multi-agent systems which generates input data for the application. The work must support importing of data from multi-agent systems, static and dynamic visualization of large amount of geo-spatial data, and chart visualization of filtered and aggregated data. It must also support exporting of visualizations and generation of a heat map. Static visualization of geo-spatial data should support displaying of clustered el- ements such as points or lines. Dynamic visualization should be able to reconstruct recorded simulation including occurred events and vessels movements. Both visual- izations should be able to restrict visualized data by a time range or by an exclusion pattern.