DOCSLIB.ORG

System G Distributed Graph Database

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
System G Distributed Graph Database System G Distributed Graph Database Gabriel Tanase Toyotaro Suzumura Jinho Lee IBM T.J. Watson Research IBM T.J. Watson Research IBM T.J. Watson Research Center Center Center 1101 Kitchawan Rd 1101 Kitchawan Rd 11501 Burnet Rd Yorktown Heights, NY 10598 Yorktown Heights, NY 10598 Austin, TX 78759 [email protected] [email protected] [email protected] Chun­Fu (Richard) Chen Jason Crawford Hiroki Kanezashi IBM T.J. Watson Research IBM T.J. Watson Research IBM T.J. Watson Research Center Center Center 1101 Kitchawan Rd 1101 Kitchawan Rd 1101 Kitchawan Rd Yorktown Heights, NY 10598 Yorktown Heights, NY 10598 Yorktown Heights, NY 10598 [email protected] [email protected] [email protected] ABSTRACT lems that need to be addressed when working with big data Motivated by the need to extract knowledge and value from is how to efficiently store and query them. In this paper we interconnected data, graph analytics on big data is a very focus on these challenges in the context of large connected active area of research in both industry and academia. To data sets that are modeled as graphs. Graphs are often used support graph analytics efficiently a large number of in mem- to represent linked concepts like communities of people and ory graph libraries, graph processing systems and graph things they like, bank accounts and transactions, or cities databases have emerged. Projects in each of these cate- and connecting roads. For all these examples vertices are gories focus on particular aspects such as static versus dy- in order of hundred of millions and edges or relations are in namic graphs, off line versus on line processing, small versus the order of tens of billions. For a property graph model large graphs, etc. [37] which is the focus of our work, there can be an ad- While there has been much advance in graph processing ditional arbitrary number of properties for each vertex and in the past decades, there is still a need for a fast graph pro- edge thus largely increasing the size of data to be maintained cessing, using a cluster of machines with distributed storage. and queried. In this paper, we discuss a novel distributed graph database The interest in graph databases has rapidly grown in the called System G designed for efficient graph data storage and past decade with many discussions on multiple aspects in- processing on modern computing architectures. In particu- cluding internal data structures and storage, query languages, lar we describe a single node graph database and a runtime programming models and analytic algorithms. As a re- and communication layer that allows us to compose a dis- sult, a few open-source based graph database solutions have tributed graph database from multiple single node instances. arisen [37, 6] and there are attempts to standardize pub- From various industry requirements, we find that fast inser- lic APIs and frameworks to query the graph data [10, 38]. tions and large volume concurrent queries are critical parts Also, a number of programming models for fast processing of of the graph databases and we optimize our database for OLAP (On-Line Analytical Processing work [33, 17, 28, 29] such features. We experimentally show the efficiency of have been proposed. However, there is not enough discus- System G for storing data and processing graph queries on sion on optimizing for performance in the distributed OLTP arXiv:1802.03057v1 [cs.DB] 8 Feb 2018 state-of-the-art platforms. (On-Line Transaction Processing) targeted graph databases and ways to design such databases are still under debate. Compared to OLAP-oriented approaches, OLTP systems 1. INTRODUCTION need to support masive changes to the data usually with Data is generated at an increasing rate and researchers continuous, large volume insertions and numerous small- in industry and academia are faced with novel challenges scale neighborhood searches. These performance require- on how to extract knowledge from it that subsequently can ments forces certain design choices and performance versus bring value to a community or company. A first set of prob- functionality trade-offs that need to be considered. In this work, we introduce the main components of the IBM System G graph database ecosystem: a fast, ACID- compliant, single node graph database and a distributed graph database that relaxes some of the ACID properties in order to achieve high performance. The scenarios we are currently targeting with our database are from financial world where the graph is constantly updated with hundreds of thousand of edges per second corresponding to transac- tions between various accounts while at the same time per- forming thousands of queries per second to look for spe- 1 cial patterns in small-scale neighborhoods. Additionally, we with support for persistent storage and a flexible runtime found out that many users do not keep the graph database for better work scheduling. as their primary database engine. Instead, they often choose OLAP Graph processing frameworks: Pregel and to migrate data from another type of database, often SQL, Giraph [33, 9, 42] employs a parallel programming model and run analytics on the graphs. In those cases, building called Bulk Synchronous Parallel (BSP) where the com- the graph may take several days to even weeks, dominating putation consists of a sequence of iterations. In each it- the time for the actual analytics phase. Therefore, the re- eration, the framework invokes a user-defined function for quirements for our graph database are: 1. Fast insertions each vertex in parallel. This function usually reads mes- of vertices and edges on the order of 100,000 per second. sages sent to this vertex from last iteration, sends messages 2. Support hundred of thousands parallel queries, which are to other vertices that will be processed at the next itera- often small scale searches around a given vertex or edge. We tion, and modifies the state of this vertex and its outgoing built SystemG because as far as we know, existing solutions edges. GraphLab [31] is a parallel programming and com- could not provide enough performance for the requirements putation framework targeted for sparse data and iterative above, for some of our industry collaborators. graph algorithms. PowerGraph [17] extends the GraphLab The contributions of our paper are a very fast single node with gather-apply-scatter model to optimize the distributed graph database implemented on top of a key value store graph processing. Pregel/Giraph and GraphLab are good and a distributed graph database designed as a composition at processing sparse data with local dependencies using it- of a set of single node databases referred to as shards. To erative algorithms. PGX.D [23] boosts performance of par- support high throughput insertions and low latency queries allel graph processing by communication optimization and we introduce a novel runtime supporting thousands of con- workload balancing. GraphX [43] is another Pregel based current requests per second and a Remote Procedure Call distributed framework that runs on Spark [3]. (RPC) based communication model to allow the database Another type of approach to reduce their network com- to move data efficiently between various machines hosting munication overhead is out-of-memory graph processing [28, different shards. On top of our distributed design we in- 39, 45, 32, 20]. They try to utilize the large storages (e.g., troduce two novel techniques for adding edges, techniques HDD, SSD). These frameworks are often focused on min- that reduces the number of exchanged messages compared imizing the number of storage accesses [28, 45], utilizing to a trivial implementation. We evaluate quantitatively our sequential accesses [39], or maintain multiple outstanding design and compare the performance of our database with I/O transactions to draw maximum bandwidth out of flash existing solutions such as Neo4J [37] and JanusGraph [6]. devices [44, 20]. There are also attempts to use mmap for ef- The remainder of the paper is organized as follows. In Sec- ficient storage management [32, 30]. However, none of these tion 2 we present related work, Section 3 briefly introduces are designed to answer ad hoc, small scale queries or process our single node graph database, Section 4 delves into our dis- graph with rich properties. tributed graph database design and organization. Section 5 OLTP Graph processing frameworks: Tinker- describes a general approach for writing graph queries on Pop [10] is an open-source graph ecosystem consisting of the distributed graph, Section 6 introduces two novel tech- key interfaces/tools needed in the graph processing space niques for adding edges, Section 7 describes the consistency including the property graph model (Blueprints), data flow model supported by our current API, Section 8 include our (Pipes), graph traversal and manipulation (Gremlin), graph- evaluation and comparison with other graph databases and object mapping (Frames), graph algorithms (Furnace) and Section 9 conclude our paper. graph server (Rexster and Gremlin Server). Interfaces de- fined by TinkerPop are becoming popular in the graph com- 2. RELATED WORK munity. As an example, Titan [11] and JanusGraph [6] ad- heres to a lot of APIs defined by TinkerPop and uses data In this section we discuss related projects and we clas- stores such as HBase and Cassandra as its scale-out per- sify them into four broad categories: graph datastructure sistent layer. TinkerPop focuses on defining data exchange libraries, graph processing frameworks, where the emphasis formats, protocols and APIs, rather than offering a software is on the programming models, graph databases, where the with good performance. focus is on storage, and hardware based approaches. Graph stores: Neo4J [37] provides a disk-based, pointer- Graph libraries: Graph libraries for in-memory-only chasing graph storage model.
Load more

Recommended publications
  • An Evaluation of the Design Space for Scalable Data Loading Into Graph Databases
    Otto-von-Guericke-Universit¨at Magdeburg Faculty of Computer Science Databases D B and Software S E Engineering Master's Thesis An Evaluation of the Design Space for Scalable Data Loading into Graph Databases Author: Jingyi Ma February 23, 2018 Advisors: M.Sc. Gabriel Campero Durand Data and Knowledge Engineering Group Prof. Dr. rer. nat. habil. Gunter Saake Data and Knowledge Engineering Group Ma, Jingyi: An Evaluation of the Design Space for Scalable Data Loading into Graph Databases Master's Thesis, Otto-von-Guericke-Universit¨at Magdeburg, 2018. Abstract In recent years, computational network science has become an active area. It offers a wealth of tools to help us gain insight into the interconnected systems around us. Graph databases are non-relational database systems which have been developed to support such network-oriented workloads. Graph databases build a data model based on graph abstractions (i.e. nodes/vertexes and edges) and can use different optimizations to speed up the basic graph processing tasks, such as traversals. In spite of such benefits, some tasks remain challenging in graph databases, such as the task of loading the complete dataset. The loading process has been considered to be a performance bottleneck, specifically a scalability bottleneck, and application developers need to conduct performance tuning to improve it. In this study, we study some optimization alternatives that developers have for load data into a graph databases. With this goal, we propose simple microbenchmarks of application-level load optimizations and evaluate these optimizations experimentally for loading real world graph datasets. We run our tests using JanusGraphLab, a JanusGraph prototype.
    [Show full text]
  • Data-Centric Graphical User Interface of the ATLAS Event Index Service
    EPJ Web of Conferences 245, 04036 (2020) https://doi.org/10.1051/epjconf/202024504036 CHEP 2019 Data-centric Graphical User Interface of the ATLAS Event Index Service Julius Hrivnᡠcˇ1,∗, Evgeny Alexandrov2, Igor Alexandrov2, Zbigniew Baranowski3, Dario Barberis4, Gancho Dimitrov3, Alvaro Fernandez Casani5, Elizabeth Gallas6, Carlos Gar- cía Montoro5, Santiago Gonzalez de la Hoz5, Andrei Kazymov2, Mikhail Mineev2, Fedor Prokoshin2, Grigori Rybkin1, Javier Sanchez5, Jose Salt5, Miguel Villaplana Perez7 1Université Paris-Saclay, CNRS/IN2P3, IJCLab, 91405 Orsay, France 2Joint Institute for Nuclear Research, 6 Joliot-Curie St., Dubna, Moscow Region, 141980, Russia 3CERN, 1211 Geneva 23, Switzerland 4Physics Department of the University of Genoa and INFN Sezione di Genova, Via Dodecaneso 33, I-16146 Genova, Italy 5Instituto de Fisica Corpuscular (IFIC), Centro Mixto Universidad de Valencia - CSIC, Valencia, Spain 6Department of Physics, Oxford University, Oxford, United Kingdom 7Department of Physics, University of Alberta, Edmonton AB, Canada Abstract. The Event Index service of the ATLAS experiment at the LHC keeps references to all real and simulated events. Hadoop Map files and HBase tables are used to store the Event Index data, a subset of data is also stored in the Oracle database. Several user interfaces are currently used to access and search the data, from a simple command line interface, through a programmable API, to sophisticated graphical web services. It provides a dynamic graph-like overview of all available data (and data collections). Data are shown together with their relations, like paternity or overlaps. Each data entity then gives users a set of actions available for the referenced data. Some actions are provided directly by the Event Index system, others are just interfaces to different ATLAS services.
    [Show full text]
  • Release Notes Date Published: 2021-03-25 Date Modified
    Cloudera Runtime 7.2.8 Release Notes Date published: 2021-03-25 Date modified: https://docs.cloudera.com/ Legal Notice © Cloudera Inc. 2021. All rights reserved. The documentation is and contains Cloudera proprietary information protected by copyright and other intellectual property rights. No license under copyright or any other intellectual property right is granted herein. Copyright information for Cloudera software may be found within the documentation accompanying each component in a particular release. Cloudera software includes software from various open source or other third party projects, and may be released under the Apache Software License 2.0 (“ASLv2”), the Affero General Public License version 3 (AGPLv3), or other license terms. Other software included may be released under the terms of alternative open source licenses. Please review the license and notice files accompanying the software for additional licensing information. Please visit the Cloudera software product page for more information on Cloudera software. For more information on Cloudera support services, please visit either the Support or Sales page. Feel free to contact us directly to discuss your specific needs. Cloudera reserves the right to change any products at any time, and without notice. Cloudera assumes no responsibility nor liability arising from the use of products, except as expressly agreed to in writing by Cloudera. Cloudera, Cloudera Altus, HUE, Impala, Cloudera Impala, and other Cloudera marks are registered or unregistered trademarks in the United States and other countries. All other trademarks are the property of their respective owners. Disclaimer: EXCEPT AS EXPRESSLY PROVIDED IN A WRITTEN AGREEMENT WITH CLOUDERA, CLOUDERA DOES NOT MAKE NOR GIVE ANY REPRESENTATION, WARRANTY, NOR COVENANT OF ANY KIND, WHETHER EXPRESS OR IMPLIED, IN CONNECTION WITH CLOUDERA TECHNOLOGY OR RELATED SUPPORT PROVIDED IN CONNECTION THEREWITH.
    [Show full text]
  • The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing: Extended Survey
    The VLDB Journal https://doi.org/10.1007/s00778-019-00548-x SPECIAL ISSUE PAPER The ubiquity of large graphs and surprising challenges of graph processing: extended survey Siddhartha Sahu1 · Amine Mhedhbi1 · Semih Salihoglu1 · Jimmy Lin1 · M. Tamer Özsu1 Received: 21 January 2019 / Revised: 9 May 2019 / Accepted: 13 June 2019 © Springer-Verlag GmbH Germany, part of Springer Nature 2019 Abstract Graph processing is becoming increasingly prevalent across many application domains. In spite of this prevalence, there is little research about how graphs are actually used in practice. We performed an extensive study that consisted of an online survey of 89 users, a review of the mailing lists, source repositories, and white papers of a large suite of graph software products, and in-person interviews with 6 users and 2 developers of these products. Our online survey aimed at understanding: (i) the types of graphs users have; (ii) the graph computations users run; (iii) the types of graph software users use; and (iv) the major challenges users face when processing their graphs. We describe the participants’ responses to our questions highlighting common patterns and challenges. Based on our interviews and survey of the rest of our sources, we were able to answer some new questions that were raised by participants’ responses to our online survey and understand the specific applications that use graph data and software. Our study revealed surprising facts about graph processing in practice. In particular, real-world graphs represent a very diverse range of entities and are often very large, scalability and visualization are undeniably the most pressing challenges faced by participants, and data integration, recommendations, and fraud detection are very popular applications supported by existing graph software.
    [Show full text]
  • Towards Scalable Loading Into Graph Database Systems
    Piecing together large puzzles, efficiently: Towards scalable loading into graph database systems Work in progress paper Gabriel Campero Durand Jingy Ma Marcus Pinnecke University of Magdeburg University of Magdeburg University of Magdeburg [email protected] [email protected] [email protected] Gunter Saake University of Magdeburg [email protected] Transaction checks ABSTRACT Id assignation 5 Many applications rely on network analysis to extract busi- Physical storage ness intelligence from large datasets, requiring specialized graph tools such as processing frameworks (e.g. Apache Gi- 4 Process organization raph, Gradoop), database systems (e.g. Neo4j, JanusGraph) Batching or applications/libraries (e.g. NetworkX, nvGraph). A recent Partitioning survey reports scalability, particularly for loading, as the fo- Pre-processing 3 remost practical challenge faced by users. In this paper we consider the design space of tools for efficient and scalable 1 Parsing for different input characteristics File Interpretation 2 graph bulk loading. For this we implement a prototypical CSV, JSON, Adjacency lists, Implicit graphs... loader for a property graph DBMS, using a distributed mes- sage bus. With our implementation we evaluate the impact Figure 1: Bulk Loading into a Graph Storage and limits of basic optimizations. Our results confirm the expectation that bulk loading can be best supported as a server-side process. We also find, for our specific case, gains 1. INTRODUCTION from batching writes (up to 64x speedups in our evaluati- Network analysis is one of many methods used by scien- on), uniform behavior across partitioning strategies, and the tists to study large data collections. For this, data has to be need for careful tuning to find the optimal configuration of represented with models based on graph theory.
    [Show full text]
  • Using Graph Databases, We Are Moving Essential Structure from Code to Data, Together with a Migration from an Imperative to a Declarative Coding Semantics
    Using Graph Databases Julius Hrivnᡠcˇ1,∗ on behalf of the ATLAS Computing Group 1Université Paris-Saclay, CNRS/IN2P3, IJCLab, 91405 Orsay, France Abstract. Data in High Energy Physics (HEP) are usually stored in tuples (ta- bles), trees, nested tuples (trees of tuples) or relational (SQL-like) databases, with or without defined schema. But many of our data are graph-like and schema-less. They consist of entities with relations, some of which are known in advance, but many are created ad-hoc, later. Such structures are not well covered by relational (SQL) databases. We don’t only need the possibility of adding new data with the existing defined relations, we also need to add new relations. Graph Databases have existed for a long time, but have only matured recently thanks to Big Data and AI (adaptive NN). There are now very good implemen- tations and de-facto standards available. The difference between SQL and Graph Database is similar to the difference between Fortran and C++. On one side, a rigid system, which can be very opti- mized. On the other side, a flexible dynamical system, which allows expression of complex structures. GraphDB is a synthesis of OODB and SQLDB. They al- low expression of a web of objects without fragility of OO world. They capture only essential relations, without keeping a complete object dump. Migrating to Graph Database means moving structure from code to data, together with migration from imperative to declarative semantics (things don’t happen, but exist). The paper describes basic principles of the Graph Database together with an overview of existing standards and implementations.
    [Show full text]
  • Suitability of Graph Database Technology for the Analysis of Spatio-Temporal Data
    future internet Article Suitability of Graph Database Technology for the Analysis of Spatio-Temporal Data Sedick Baker Effendi 1,* , Brink van der Merwe 1 and Wolf-Tilo Balke 2 1 Department of Computer Science, Stellenbosch University, Stellenbosch 7600, South Africa; [email protected] 2 Institute for Information Systems, TU Braunschweig, D-38106 Braunschweig, Germany; balke@ifis.cs.tu-bs.de * Correspondence: [email protected]; Tel.: +27-834-324-666 Received: 30 March 2020; Accepted: 23 April 2020; Published: 26 April 2020 Abstract: Every day large quantities of spatio-temporal data are captured, whether by Web-based companies for social data mining or by other industries for a variety of applications ranging from disaster relief to marine data analysis. Making sense of all this data dramatically increases the need for intelligent backend systems to provide realtime query response times while scaling well (in terms of storage and performance) with increasing quantities of structured or semi-structured, multi-dimensional data. Currently, relational database solutions with spatial extensions such as PostGIS, seem to come to their limits. However, the use of graph database technology has been rising in popularity and has been found to handle graph-like spatio-temporal data much more effectively. Motivated by the need to effectively store multi-dimensional, interconnected data, this paper investigates whether or not graph database technology is better suited when compared to the extended relational approach. Three database technologies will be investigated using real world datasets namely: PostgreSQL, JanusGraph, and TigerGraph. The datasets used are the Yelp challenge dataset and an ambulance response simulation dataset, thus combining real world spatial data with realistic simulations offering more control over the dataset.
    [Show full text]
  • Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries
    1 Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries MACIEJ BESTA, Department of Computer Science, ETH Zurich ROBERT GERSTENBERGER, Department of Computer Science, ETH Zurich EMANUEL PETER, Department of Computer Science, ETH Zurich MARC FISCHER, PRODYNA (Schweiz) AG MICHAŁ PODSTAWSKI, Future Processing CLAUDE BARTHELS, Department of Computer Science, ETH Zurich GUSTAVO ALONSO, Department of Computer Science, ETH Zurich TORSTEN HOEFLER, Department of Computer Science, ETH Zurich Graph processing has become an important part of multiple areas of computer science, such as machine learning, computational sciences, medical applications, social network analysis, and many others. Numerous graphs such as web or social networks may contain up to trillions of edges. Often, these graphs are also dynamic (their structure changes over time) and have domain-specific rich data associated with vertices and edges. Graph database systems such as Neo4j enable storing, processing, and analyzing such large, evolving, and rich datasets. Due to the sheer size of such datasets, combined with the irregular nature of graph processing, these systems face unique design challenges. To facilitate the understanding of this emerging domain, we present the first survey and taxonomy of graph database systems. We focus on identifying and analyzing fundamental categories of these systems (e.g., triple stores, tuple stores, native graph database systems, or object-oriented systems), the associated graph models (e.g., RDF or Labeled Property Graph), data organization techniques (e.g., storing graph data in indexing structures or dividing data into records), and different aspects of data distribution and query execution (e.g., support for sharding and ACID).
    [Show full text]
  • Query Analysis on a Distributed Graph Database Student: Bc
    ASSIGNMENT OF MASTER’S THESIS Title: Query Analysis on a Distributed Graph Database Student: Bc. Lucie Svitáková Supervisor: Ing. Michal Valenta, Ph.D. Study Programme: Informatics Study Branch: Web and Software Engineering Department: Department of Software Engineering Validity: Until the end of summer semester 2019/20 Instructions 1. Research current practicies of data storage in distributed graph databases. 2. Choose an existing database system for your implementation. 3. Analyse how to extract necessary information for further query analysis and implement that extraction. 4. Create a suitable method to store the data extracted in step 3. 5. Propose a method for a more efficient redistribution of the data. References Will be provided by the supervisor. Ing. Michal Valenta, Ph.D. doc. RNDr. Ing. Marcel Jiřina, Ph.D. Head of Department Dean Prague October 5, 2018 Master’s thesis Query Analysis on a Distributed Graph Database Bc. et Bc. Lucie Svit´akov´a Department of Software Engineering Supervisor: Ing. Michal Valenta, Ph.D. January 8, 2019 Acknowledgements I would like to thank my supervisor Michal Valenta for his valuable suggestions and observations, my sister S´arkaSvit´akov´aforˇ spelling corrections, my friends Martin Horejˇsand Tom´aˇsNov´aˇcekfor their valuable comments and my family for their continual motivation and support. Declaration I hereby declare that the presented thesis is my own work and that I have cited all sources of information in accordance with the Guideline for adhering to ethical principles when elaborating an academic final thesis. I acknowledge that my thesis is subject to the rights and obligations stip- ulated by the Act No.
    [Show full text]
  • Graph Computing with Janusgraph Jason Plurad Open Source Developer & Advocate
    Graph Computing with JanusGraph Jason Plurad Open Source Developer & Advocate LDBC 11th Technical User Community (TUC) Meeting / University of Texas at Austin / June 8, 2018 / © 2018 IBM Corporation JanusGraph • Established in January 2017 • Scalable graph database distributed on multi-machine clusters with pluggable storage and indexing • Vendor-neutral, open community with open governance JanusGraph™ • Founders: Expero, Google, Grakn, Maintainer The Linux Hortonworks, IBM Foundation License Apache Releases 0.3.0 planned 2Q 2018 https://janusgraph.org LDBC 11th Technical User Community (TUC) Meeting / University of Texas at Austin / June 8, 2018 / © 2018 IBM Corporation 2 JanusGraph Community • Contributors • Member companies • 49 total • Amazon • Huawei • Committers • Linkurious • 14 initial, 6 added • Netflix • Newforma • Technical Steering Committee • Orchestral Developments • 6 initial, 2 added • Uber • Issues • In production • 287 open, 352 closed • Celum • Finc • Open source projects • G-Data • Apache Atlas • IBM Cloud • Open Network Automation Platform (ONAP) • Times Internet LDBC 11th Technical User Community (TUC) Meeting / University of Texas at Austin / June 8, 2018 / © 2018 IBM Corporation 3 Apache TinkerPop • Established in 2009 • Apache incubator in 2015 • Top-level project in 2016 • Open source, vendor-agnostic, graph computing framework • Gremlin graph traversal language Apache TinkerPop™ Maintainer The Apache Software Foundation License Apache Releases 3.3.3 May 2018 https://tinkerpop.apache.org LDBC 11th Technical User
    [Show full text]
  • Practicalgremlin.Pdf
    PRACTICAL GREMLIN An Apache TinkerPop Tutorial Kelvin R. Lawrence Version 283-preview, August 2nd 2021 Table of Contents 1. INTRODUCTION . 1 1.1. How this book came to be. 1 1.2. Providing feedback. 2 1.3. Some words of thanks . 2 1.4. What is this book about?. 2 1.5. Introducing the book sources, sample programs and data . 4 1.6. A word about TinkerPop 3.4 . 5 1.7. Introducing TinkerPop 3.5 . 6 1.8. So what is a graph database and why should I care? . 6 1.9. A word about terminology . 8 2. GETTING STARTED . 10 2.1. What is Apache TinkerPop? . 10 2.2. The Gremlin console . 11 2.2.1. Downloading, installing and launching the console . 11 2.2.2. Saving output from the console to a file . 15 2.3. Introducing TinkerGraph . 16 2.4. Introducing the air-routes graph . 18 2.4.1. Updated versions of the air-route data . 22 2.5. TinkerPop 3 migration notes . 22 2.5.1. Creating a TinkerGraph TP2 vs TP3 . 22 2.5.2. Loading a graphML file TP2 vs TP3 . 22 2.5.3. A word about the TinkerPop.sugar plugin . 23 2.6. Loading the air-routes graph using the Gremlin console . 24 2.7. Turning off some of the Gremlin console’s output. 25 2.8. A word about indexes and schemas. 25 3. WRITING GREMLIN QUERIES . 27 3.1. Introducing Gremlin . 27 3.1.1. A quick look at Gremlin and SQL . 27 3.2. Some fairly basic Gremlin queries .
    [Show full text]
  • Demystifying Graph Databases: Analysis and Taxonomyof Data Organization, System Designs, and Graph Queries
    1 Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries Towards Understanding Modern Graph Processing, Storage, and Analytics MACIEJ BESTA, EMANUEL PETER, Department of Computer Science, ETH Zurich ROBERT GERSTENBERGER, Department of Computer Science, ETH Zurich MARC FISCHER, PRODYNA (Schweiz) AG MICHAŁ PODSTAWSKI, Future Processing CLAUDE BARTHELS, GUSTAVO ALONSO, Department of Computer Science, ETH Zurich TORSTEN HOEFLER, Department of Computer Science, ETH Zurich Graph processing has become an important part of multiple areas of computer science, such as machine learning, computational sciences, medical applications, social network analysis, and many others. Numerous graphs such as web or social networks may contain up to trillions of edges. Often, these graphs are also dynamic (their structure changes over time) and have domain-specific rich data associated with vertices and edges. Graph database systems such as Neo4j enable storing, processing, and analyzing such large, evolving, and rich datasets. Due to the sheer size of such datasets, combined with the irregular nature of graph processing, these systems face unique design challenges. To facilitate the understanding of this emerging domain, we present the first survey and taxonomy of graph database systems. We focus on identifying and analyzing fundamental categories of these systems (e.g., triple stores, tuple stores, native graph database systems, or object-oriented systems), the associated graph models (e.g., RDF or Labeled Property Graph), data organization techniques (e.g., storing graph data in indexing structures or dividing data into records), and different aspects of data distribution and query execution (e.g., support for sharding and ACID). 45 graph database systems are presented and compared, including Neo4j, OrientDB, or Virtuoso.
    [Show full text]