2nd Edition Compliments of Ian Robinson, Graph Jim Webber & Emil Eifrem Databases NEW OPPORTUNITIES FOR CONNECTED DATA Download now at: bit.ly/dl-neo4j neo4j.com SECOND EDITION Graph Databases Ian Robinson, Jim Webber & Emil Eifrem Graph Databases by Ian Robinson, Jim Webber, and Emil Eifrem Copyright © 2015 Neo Technology, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected]. Editor: Marie Beaugureau Interior Designer: David Futato Production Editor: Kristen Brown Cover Designer: Ellie Volckhausen Proofreader: Christina Edwards Illustrator: Rebecca Demarest Indexer: WordCo Indexing Services June 2013: First Edition June 2015: Second Edition Revision History for the Second Edition 2015-05-04: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491930892 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Graph Databases, the cover image of an European octopus, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-93200-1 [LSI] Table of Contents Foreword. vii Preface. xi 1. Introduction. 1 What Is a Graph? 1 A High-Level View of the Graph Space 4 Graph Databases 5 Graph Compute Engines 7 The Power of Graph Databases 8 Performance 8 Flexibility 9 Agility 9 Summary 10 2. Options for Storing Connected Data. 11 Relational Databases Lack Relationships 11 NOSQL Databases Also Lack Relationships 15 Graph Databases Embrace Relationships 18 Summary 24 3. Data Modeling with Graphs. 25 Models and Goals 25 The Labeled Property Graph Model 26 Querying Graphs: An Introduction to Cypher 27 Cypher Philosophy 28 MATCH 30 RETURN 30 iii Other Cypher Clauses 31 A Comparison of Relational and Graph Modeling 32 Relational Modeling in a Systems Management Domain 33 Graph Modeling in a Systems Management Domain 38 Testing the Model 39 Cross-Domain Models 41 Creating the Shakespeare Graph 45 Beginning a Query 46 Declaring Information Patterns to Find 48 Constraining Matches 49 Processing Results 50 Query Chaining 51 Common Modeling Pitfalls 52 Email Provenance Problem Domain 52 A Sensible First Iteration? 52 Second Time’s the Charm 55 Evolving the Domain 58 Identifying Nodes and Relationships 63 Avoiding Anti-Patterns 63 Summary 64 4. Building a Graph Database Application. 65 Data Modeling 65 Describe the Model in Terms of the Application’s Needs 66 Nodes for Things, Relationships for Structure 67 Fine-Grained versus Generic Relationships 67 Model Facts as Nodes 68 Represent Complex Value Types as Nodes 71 Time 72 Iterative and Incremental Development 74 Application Architecture 76 Embedded versus Server 76 Clustering 81 Load Balancing 82 Testing 85 Test-Driven Data Model Development 85 Performance Testing 91 Capacity Planning 95 Optimization Criteria 95 Performance 96 Redundancy 98 Load 98 iv | Table of Contents Importing and Bulk Loading Data 99 Initial Import 99 Batch Import 100 Summary 104 5. Graphs in the Real World. 105 Why Organizations Choose Graph Databases 105 Common Use Cases 106 Social 106 Recommendations 107 Geo 108 Master Data Management 109 Network and Data Center Management 109 Authorization and Access Control (Communications) 110 Real-World Examples 111 Social Recommendations (Professional Social Network) 111 Authorization and Access Control 123 Geospatial and Logistics 132 Summary 147 6. Graph Database Internals. 149 Native Graph Processing 149 Native Graph Storage 152 Programmatic APIs 158 Kernel API 158 Core API 159 Traversal Framework 160 Nonfunctional Characteristics 162 Transactions 162 Recoverability 163 Availability 164 Scale 166 Summary 170 7. Predictive Analysis with Graph Theory. 171 Depth- and Breadth-First Search 171 Path-Finding with Dijkstra’s Algorithm 173 The A* Algorithm 181 Graph Theory and Predictive Modeling 182 Triadic Closures 182 Structural Balance 184 Local Bridges 188 Table of Contents | v Summary 190 A. NOSQL Overview. 193 Index. 211 vi | Table of Contents Foreword Graphs Are Everywhere, or the Birth of Graph Databases as We Know Them It was 1999 and everyone worked 23-hour days. At least it felt that way. It seemed like each day brought another story about a crazy idea that just got millions of dollars in funding. All our competitors had hundreds of engineers, and we were a 20-ish person development team. As if that was not enough, 10 of our engineers spent the majority of their time just fighting the relational database. It took us a while to figure out why. As we drilled deeper into the persistence layer of our enterprise content management application, we realized that our software was managing not just a lot of individual, isolated, and discrete data items, but also the connections between them. And while we could easily fit the discrete data in relational tables, the connected data was more challenging to store and tremendously slow to query. Out of pure desperation, my two Neo cofounders, Johan and Peter, and I started experimenting with other models for working with data, particularly those that were centered around graphs. We were blown away by the idea that it might be possible to replace the tabular SQL semantic with a graph-centric model that would be much easier for developers to work with when navigating connected data. We sensed that, armed with a graph data model, our development team might not waste half its time fighting the database. Surely, we said to ourselves, we can’t be unique here. Graph theory has been around for nearly 300 years and is well known for its wide applicability across a number of diverse mathematical problems. Surely, there must be databases out there that embrace graphs! vii Well, we AltaVistad1 around the young Web and couldn’t find any. After a few months of surveying, we (naively) set out to build, from scratch, a database that worked natively with graphs. Our vision was to keep all the proven features from the relational database (transactions, ACID, triggers, etc.) but use a data model for the 21st century. Project Neo was born, and with it graph databases as we know them today. The first decade of the new millennium has seen several world-changing new busi‐ nesses spring to life, including Google, Facebook, and Twitter. And there is a com‐ mon thread among them: they put connected data—graphs—at the center of their business. It’s 15 years later and graphs are everywhere. Facebook, for example, was founded on the idea that while there’s value in discrete information about people—their names, what they do, etc.—there’s even more value in the relationships between them. Facebook founder Mark Zuckerberg built an empire on the insight to capture these relationships in the social graph. Similarly, Google’s Larry Page and Sergey Brin figured out how to store and process not just discrete web documents, but how those web documents are connected. Goo‐ gle captured the web graph, and it made them arguably the most impactful company of the previous decade. Today, graphs have been successfully adopted outside the web giants. One of the big‐ gest logistics companies in the world uses a graph database in real time to route phys‐ ical parcels; a major airline is leveraging graphs for its media content metadata; and a top-tier financial services firm has rewritten its entire entitlements infrastructure on Neo4j. Virtually unknown a few years ago, graph databases are now used in industries as diverse as healthcare, retail, oil and gas, media, gaming, and beyond, with every indication of accelerating their already explosive pace. These ideas deserve a new breed of tools: general-purpose database management technologies that embrace connected data and enable graph thinking, which are the kind of tools I wish had been available off the shelf when we were fighting the rela‐ tional database back in 1999. 1 For the younger readers, it may come as a shock that there was a time in the history of mankind when Google didn’t exist. Back then, dinosaurs ruled the earth and search engines with names like AltaVista, Lycos, and Excite were used, primarily to find ecommerce portals for pet food on the Internet. viii | Foreword I hope this book will serve as a great introduction to this wonderful emerging world of graph technologies, and I hope it will inspire you to start using a graph database in your next project so that you too can unlock the extraordinary power of graphs. Good luck! —Emil Eifrem Cofounder of Neo4j and CEO of Neo Technology Menlo Park, California May 2013 Foreword | ix Preface Graph databases address one of the great macroscopic business trends of today: lever‐ aging complex and dynamic relationships in highly connected data to generate insight and competitive advantage. Whether we want to understand relationships between customers, elements in a telephone or data center network, entertainment producers and consumers, or genes and proteins, the ability to understand and ana‐ lyze vast graphs of highly connected data will be key in determining which companies outperform their competitors over the coming decade.