An analysis of the graph processing landscape

Miguel E. Coimbra∗, Alexandre P. Francisco and Luís Veiga [email protected], [email protected], [email protected] INESC-ID/IST, Universidade de Lisboa, Portugal *Corresponding author

ABSTRACT definitions related to the potential for a graph to be up- The value of graph-based big data can be unlocked by dated. This survey is aimed at both the experienced soft- exploring the topology and metrics of the networks they ware engineer or researcher as well as the newcomer represent, and the computational approaches to this ex- looking for an understanding of the landscape of solu- ploration take on many forms. For the use-case of per- tions (and their limitations) for graph processing. forming global computations over a graph, it is first in- gested into a graph processing system from one of many 1. INTRODUCTION digital representations. Extracting information from gra- Graph-based data is found almost everywhere, phs involves processing all their elements globally, and with examples such as analyzing the structure of the can be done with single-machine systems (with vary- World Wide Web [34, 33, 36], bio-informatics data ing approaches to hardware usage), distributed systems representation via de Bruijn graphs [72] in metage- (either homogeneous or heterogeneous groups of ma- nomics [164, 304], atoms and covalent relationships chines) and systems dedicated to high-performance com- in chemistry [20], the structure of distributed com- puting (HPC). For these systems focused on processing putation itself [182], massive parallel learning of the bulk of graph elements, common use-cases consist in tree ensembles [219] and parallel topic models [265]. executing for example algorithms such as PageRank or Academic research centers in collaboration with in- community detection, which produce insights on graph dustry players like Facebook, Microsoft and Google structure and relevance of their elements. have rolled out their own graph processing systems, Considering another type of use-case, graph-specific contributing to the development of several open- databases may be used to efficiently store and repre- source frameworks [198, 59, 300, 51]. They need sent graphs to answer requests like queries about spe- to deal with huge graphs, such as the case of the cific relationships and graph traversals. While tabular- Facebook graph with billions of vertices and hun- type databases may be used to store relations between dreds of billions of edges [110]. elements, it is highly inefficient to use for this purpose these databases in terms of both storage space require- 1.1 Domains ments and processing time. Relational database man- We list some of the domains of human activity agement systems (RDBMS) and NoSQL databases, to that are best described by relations between ele- ments - graphs: arXiv:1911.11624v3 [cs.DC] 16 Feb 2021 achieve this purpose, need complex nested queries to represent the multi-level relations between data elements. Graph-specific databases employ efficient graph repre- • Social networks. They make up a large por- sentations or may make use of underlying storage sys- tion of social interactions in the Internet. We tems. name some of the best-known ones: Facebook In this survey we firstly familiarize the reader with (2.50 billion monthly active users as of Decem- common graph datasets and applications in the world ber 2019 [89]), Twitter (330 million monthly of today. We provide an overview of different aspects active users in Q1’19 [282]) and LinkedIn (330 of the graph processing landscape and describe classes million monthly active users as of December of systems based on a set of dimensions we describe. 2019 [170]). In these networks, the vertices The dimensions we detail encompass paradigms to ex- represent users and edges are used to represent press graph processing, different types of systems to use, friendship or follower relationships. Further- coordination and communication models in distributed more, they allow the users to send messages graph processing, partitioning techniques and different to each other. This messaging functionality can be represented with graphs with associ- ated time properties. Other examples of social • Epidemiology. The analysis of disease prop- networks are WhatsApp (1.00 billion monthly agation and models of transition between states active users as of early 2016 [295]) and Tele- of health, infection, recovery and death are gram (300 million monthly active users [250]). very important for public health and for en- suring standards of practices between coun- • World Wide Web. Estimates point to the tries to protect travelers and countries’ popu- existence of over 1.7 billion websites as of Oc- lations [63, 19, 40, 58]. These are represented tober 2019 [138], with the first one becoming as graphs, which can also be applied to lo- live in 1991, hosted at CERN. Commercial, calized health-related topics like reproductive educational and recreational activities are just health, sexual networks and the transmission some of the many facets of daily life that gave of infections [168, 24]. They have even been shape to the Internet we know today. With used to model epidemics in massively multi- the advent of business models built over the player online games such as World of War- reachability and reputation of websites (e.g. craft [173]. Real-life epidemics are perhaps at Google, Yahoo and Bing as search engines), the forefront of examples of this application of the application of graph theory as a tool to graph theory for health preservation, with the study the web structure has matured during most recent example as COVID-19 [274]. the last two decades with techniques to enable the analysis of these massive networks [34, 33]. Other types of data represented as graphs can be found [251]. To illustrate the growing magnitude • Telecommunications. These networks have of graphs, we focus on web graph sizes of different been used for decades to enable distant com- web domains in Fig 1, where we show the number munication between people and their struc- of edges for web crawl graph datasets made avail- tural properties have been studied using graph- able by the Laboratory of Web Algorithmics [162] based approaches [23, 21]. Though some of and by Web Data Commons [192]. If one were to its activity may have transferred to the ap- retrieve insights on the structure of these larger plications identified above as social networks, graphs (above a hundred million edges), it would they are still relevant. The vertices in these become immediately clear that a combination of networks represent user phones, whose study computer resources and specific software are nec- is relevant for telecommunications companies essary in order to process them. wishing to assess closeness relationships be- tween subscribers, calculate churn rates, enact 1.2 Motivation more efficient marketing strategies [4] and also We include this section in this survey to high- to support foreign signals intelligence (SIG- light three reasons. Firstly, the recent years have INT) activities [228]. seen a positive tendency in the field of all things re- • Recommendation systems. Graph-based lated to graph processing. As its aspects are further approaches to recommendation systems have explored and optimized, with new paradigms pro- been heavily explored in the last decades [115, posed, there has been a proliferation of multiple sur- 116, 261]. Companies such as Amazon and veys [183, 123, 152, 153, 128, 243, 266]. They have eBay provide suggestions to users based on made great contributions in systematizing the field user profile similarity in order to increase con- of graph processing, by working towards a consen- version rates from targeted advertising. The sus of terminology and offering discussion on how structures underlying this analysis are graph- to present or establish hierarchies of concepts inher- based [308, 302, 29]. ent to the field. Effectively, we have seen vast con- tributions capturing the maturity of different chal- • Transports, smart cities and IoT. Graphs lenges of graph processing and the corresponding have been used to represent the layout and responses developed by academia and industry. flow of information in transport networks com- The value-proposition of this document is there- prised of people circulating in roads, trains fore, on a first level, the identification of the di- and other means of transport [88, 284, 231]. mensions we observe to be relevant with respect to The Internet-of-Things (IoT) will continue to graph processing. This is more complex than, for grow as more devices come into play and 5G example, merely listing the types of graph process- proliferates. The way IoT devices engage for ing system architectures or the types of communi- collaborative purposes and implement security cation and types of coordination within the class of frameworks can be represented as graphs [105]. distributed systems for graph processing. Many of Web Crawl Big Graphs

100 G

10 G

1 G

100 M

10 M

1 M

100 k

10 k

1 k Number of edges |E| scale)(in log Number 100 uk-2002 indochina-2004it-2004 arabic-2005uk-2005 sk-2005 clueweb12ccrawl-aug-2012uk-2014-tpduk-2014-hostuk-2014 ccrawl-spr-2014eu-2015-tpdeu-2015-hostgsh-2015-tpdgsh-2015-hostgsh-2015eu-2015

Figure 1: Web graph edge counts for domain crawls since the year 2000 (in log scale). these dimensions, if not all, are interconnected in pious amounts of memory, or instead employ many ways. As the study of each one is deepened, compression techniques for graph processing. its individual overlap with the others is eventually • Multi-machine: distributed systems which can noted. For example, using distributed systems, it is be a cluster of machines (either homogeneous necessary to distribute the graph across several ma- or heterogeneous) or special-purpose high-per- chines. This necessity raises the question of how to formance computing systems (HPC). partition the graph to distribute it. Afterward, as a natural consequence, it is necessary to define the co- On the third level, and also considering a scope ordination (e.g. synchronous versus asynchronous) of meta-analysis, we discuss the structuring of the between nodes of the system. Orthogonally to this, field that is presented in existing surveys. We com- the relation between data and computation in graph plement it with our own highlight of important re- processing must be defined. The vertices of a graph lations between graph processing concepts and a may represent posts in a social network, while the chronological analysis of the field of graph process- edges dictate follower relationships. But to the sys- ing. tems that process them, one could specify the units of computation to be solely the vertices, exclusively 1.3 Document Roadmap the edges or components of the graph (and the def- This following survey sections are as follows: inition of component in this case would be required • Background on Graph Notation and Terminol- too). Herein, we first note the cornerstone aspects ogy covers the most essential notation used in of the graph processing field from individual works graph theory. and from existing surveys: dimensions and defini- tions. • Property Model and Query Languages describes Secondly, we provide an exhaustive list of systems the systematizations developed to structure the for graph processing, sharing their year of debut and execution of queries over graph data using com- distinctive features. We explore the different types putational resources. that exist: • Graph Algorithms: Natures and Types high- • Single-machine: they may simply require co- lights relevant aspects of graph-processing tasks and different types of graph algorithms. as G = (V,E) – it is usually defined by a set of vertices V of size n = |V | and a set of edges E of • Computational Representations details impor- size m = |E|. Vertices are sometimes referred to as tant computational representations of graphs nodes and edges as links. An undirected graph G is which typically use compression techniques. a pair (V,E) of sets such that E ⊆ V × V is a set • Profile of Developers and Researchers delves of unordered pairs. If E is a set of ordered pairs, into results of a study on users of graph pro- then we say that G is a directed graph. Between the cessing applications to identify important use- same two vertices there is usually at most one edge; case trends. if there are more, then the graph is called a multi- graph (note: an ordered graph in which a pair of • Graph Processing: Computational Units and vertices share two edges in opposite direction is not Models analyzes how graphs are conceptually necessarily a multigraph). Multigraphs are more manipulated in many contributions of the field’s common when looking at the applications and use- state of the art, and the different levels of gran- cases for graph databases such as Neo4j [199], where ularity. one may model more than one relation type between • Dimension: Partitioning presents the most- the same vertices. known approaches to decomposing graph-based Additionally, given a graph G = (V,E), the set of data into computational units for parallelism vertices of G is written as V (G) and the set of edges and distribution, showcasing models with dif- as E(G). More commonly, we write V (G) = V and ferent levels of granularity. E(G) = E. The concept of a vertex’s surrounding is important for specifying traversals (relevant when • Dimension: Dynamism enumerates scenarios considering graph query languages [131]) and also with different definitions of dynamism in graph defining scopes and units of computation in graph processing, from the graph representing tem- processing [182, 240, 279]. Two vertices u, v ∈ V poral data to the manipulated graph represen- are adjacent or neighbours if (u, v) is an edge - if tation being constantly updated with new in- (u, v) ∈ E. Given a vertex v ∈ V (G), its set of formation. neighbours is denoted by NG(v), or succinctly by N(G) when clearing G from the context. The set • Dimension: Workload discusses the nature of of edges with v as a source or target is written as graph processing workloads and different scopes E(v) = {(u, u0) ∈ E(G) | v = u or v = u0}. More such as analytics and storage. generally, for two sets X,Y ⊆ V , we denote E(X) as • Single-Machine and Shared-Memory Parallel the set of edges with an end in X: E(X) = {(u, v) ∈ Approaches presents these types of architec- E(G) | u ∈ X or v ∈ X} and denote as E(X,Y ) ture and describes important state-of-the-art the set of edges with an end in X and an end in Y : marks. E(X,Y ) = {(u, v) ∈ G | either u ∈ X and v ∈ Y , or v ∈ X and u ∈ Y }. Furthermore, the degree of a • High-Performance Computing is focused on multi- vertex is used to indicate its number of neighbours. core and multi-processor architectures. For a graph G = (V,E) and a vertex v ∈ V , the • Distributed Graph Processing Systems enumer- degree:- of v may simply be commonly written as ates systems which focus on distributing the dG(v) or d(v), with d(v) = |N(v)|. If G is a directed graph across machines to enable processing. graph, it is found in the literature to be written as din(v) and dout(v) the number of incoming and • Graph Databases enumerates the graph database outgoing neighbours of v, respectively. solutions we found as part of our research. These notations are the basic building blocks for • Conclusion and final remarks. graph theory and a mandatory learning topic for those who come from backgrounds such as Math- 2. BACKGROUND ON GRAPH ematics, Computer Science and many other fields. NOTATION AND TERMINOLOGY We note that across decades, the field of graph the- ory has matured and refined the notations and sys- Here we detail terms and concepts which are known tematization of how we research and approach prob- in graph theory. We include preliminary notions lems represented as graphs. These advancements that serve as a basis to familiarize the reader with have been accompanied with an even faster rhythm the language used in scientific documents on graph of technological progress. In tune, so has evolved applications, processing systems and novel techni- how these notations and representations are trans- ques. In the literature [211], a graph G is written lated to actual actionable computational tasks and we list some of the most relevant graph query lan- data. The last decades have seen the introduction guages (and proposals in the scope of graph query of query languages and models to represent graphs languages), projects that structure the access to from the perspective of managing computational re- graph-based data as well as open-source constructs sources and solving graph-oriented problems. that form their basis. • Cypher. An evolving graph query language[101] 3. PROPERTY MODEL AND QUERY which debuted with Neo4j’s entrance in the LANGUAGES field of graph databases. There have been ef- In this section we detail languages, models and forts to adopt and develop the language in an subsequent standards for querying graph-based data. open-source approach [111]. Cypher has been The property graph model defines the organization an open and evolving language as part of the of interrelated data as nodes (vertices), relation- openCypher project [132]. The project has ships (edges) and the properties of both types of members involved in aspects such as synergy elements [208]. Instances of this model are usually of engineering efforts with Apache Spark [303], found coupled to the design of graph databases and a language group and even interoperability fea- at the level of graph query languages [236, 286, 10, tures for systems that use Gremlin. This graph 111]. Figure ?? shows a depiction of the properties query language heavily influenced the ISO pro- that may be associated to vertices and edges in the ject for creating a standard graph query lan- context of the property graph model. guage[205] and has a syntax familiar to devel- The ISO SQL Committee has accepted on Septem- opers with knowledge of SQL. Cypher queries ber 2019 the Graph Query Language project pro- may be run on the following databases: Neo4j, posal [144], to enable SQL users to use property Graphflow[155], RedisGraph[49] and SAP Hana graph style queries on top of SQL tables. This will Graph[242] and all databases where Gremlin promote the interoperability between SQL databases is supported [206]. This language is also used and graph databases and constitutes an important to express computation in GRADOOP [148] and mark in the approximation between graph-structu- Python Ruruki [215]. red data and the databases that support it, an area • Gremlin. A graph traversal machine and lan- that has been studied for decades [253, 127]. The guage, developed in scope of the Apache Tin- language standards engineering team of Neo4j iden- kerPop project[98]. This project’s develop- tify the following cornerstones regarding the grow- ment and growth was promoted by the now- ing popularity of the property graph model [229]: defunct Titan graph database [16] which was 1. An intuitive model geared for application de- forked into the open-source JanusGraph data- velopers. base[143] and the commercial DataStax Enter- prise Graph solution. The traversal machine 2. Ability of rapid prototyping with schema-optio- of Gremlin is defined as a set of three compo- nality. nents [236]: the data represented by a graph G, a traversal Ψ (instructions) which consists of a 3. Availability of native graph databases. tree of functions called steps; a set of traversers 4. Declarative query languages focusing on ease T (read/write heads). With this composition, of use with graph pattern matching and visual Gremlin and its traversal machine enable the syntax, top-down linear statement order and exploration of multi-dimensional structures that language composability. model a heterogeneous set of “things” related to each other in a heterogeneous set of ways 3.1 Graph Query Languages as detailed in [236, 237]. A traversal eval- The access to elements offered by graph databases uated against a graph may generate billions is performed by means of specific query languages. of traversers, even on small graphs, due to There are even projects focused on analytics which the exponential growth of the number of paths offer the ability to explore datasets using graph query that exist with each step the traversers take. languages without actually running a graph database, Databases supporting Gremlin include Orient- such as GRADOOP [149] and GraphFrames [200]. Lan- DB[277], Neo4j[294], DataStax Enterprise[67], guages such as Cypher even support queries on gra- InfiniteGraph[212], JanusGraph[143], Azure phs which produce new graphs (for example to rep- Cosmos DB[221] and Amazon Neptune[25], while resent specific entities and relationships of the orig- the graph processing systems that allow its use inal graph) upon which to run further queries. Here are Apache Giraph [59] and Apache Spark [15]. • SPARQL. The standard query language for RDF changed independently of the other. For exam- data (triplets), also known as the query lan- ple, to serve queries in the graph, the data in guage for the semantic web. We mention it due the back-end could come from databases (e.g. to its graph pattern matching capability [224] Neo4j as the back-end serving GraphQL queries and scalability potential of querying large RDF received at a web endpoint [207]), in-memory graphs [133]. As RDF is a direct labeled graph representations or other APIs. data format, SPARQL becomes a language for graph-matching. Its queries have three com- • PGQL. The Property Graph Query Language, ponents: a pattern matching part allowing for based on the paradigm of graph pattern match- pattern unions, nesting, filtering values of ma- ing [286]. It closely follows the syntactic struc- tchings and choosing the data source to match tures of SQL, providing regular path queries by a pattern; a solution modifier to allow mod- with conditions on labels and properties to en- ifications to the computed output of the pat- able reachability and path finding queries. The tern, such as applying operators as projections, data types it defines are the intrinsic vertex, orderings, limits and distinct; the output, which edge, path and also an intrinsic graph type, al- can be binary answers, selections of values for lowing for graph construction and query com- variables that matched the patterns, construc- position. It was motivated by the fact that tion of new RDF data from the values or de- SPARQL is the RDF standard query language, scriptions of resources. While graph databases thus imposing that graphs be represented as a may not necessarily be triplet stores, the graph set of triples (or edges), and by Cypher’s lack query languages they support may allow for of support for regular path queries and graph example that the RDF-specific semantics of a construction as fundamental graph querying SPARQL query may be translated to Cypher, functionalities. PGQL also provides tabular out- Gremlin or another language. SPARQL is also put, allowing its queries to be naturally nested supported (analytics) over GraphX [249] and inside SQL queries, allowing for easy integra- the higher-level graph analytics tool Graph- tion into existing database technology. It is Frames[18]. Among the graph databases that used in Oracle’s products. support this language we have Amazon Neptu- ne[25] and AllegroGraph[136] (the later two • G-Log. A declarative query language on graphs more oriented to the purpose of RDF). which was designed to combine the expressive- ness of logic, the modeling of complex objects • GraphQL. A framework developed and inter- with identity and the representations enabled nally used at Facebook for years before its ref- by graphs [220]. The authors describe it as erence implementation was released as open- a deductive language for complex objects with source [91]. It introduced a new type of web- identity, with a data model that captures the based interface for data access. As a frame- modeling capabilities of object-oriented lan- work, one of its core components is a query guages, although lacking their typical data ab- language for expressing data retrieval requests straction features which are related to system sent to web servers that are GraphQL-aware. dynamism. They claim G-Log may be seen as The queries are syntactically similar to Java- a graph-based symbolism for first-order logic Script Object Notation (JSON). The Gra- and they prove that all sentences of first-order phQL specification implicitly assumes a logical logic may be written in G-Log. Secondly, they data model implemented as a virtual, graph- define the semantics of the language for data- based view over an underlying database man- base query evaluation. Lastly, the authors sta- agement system [125]. It has been studied te that due to being a very powerful language, with the semantics of its queries formalized its computation could be unnecessarily ineffi- as a labeled-graph data model and the total cient in the most general case. We mention size of a GraphQL response shown to be com- G-Log as it is historically relevant due to high- putable in polynomial time [126]. GraphQL is lighting the importance and expressiveness of more than a query language - it defines a con- graph-based data models in manipulating the tract between the back-end and front-end over relationships between data. an agreed-upon type system, forming an ap- plication data model as a graph. It is useful as A study on these aspects inherent to graph-stru- it proposes a decoupling between the back-end ctured data has been performed using different data and front-end, allowing each component to be models (RDF, property graph and relational model) and a sample of systems (RDF: Virtuoso [87], Tri- dustrial partners whose individual experiences are pleRush[271]; graph databases: Neo4j, Sparksee- helping to shape the future of the way graphs are [188]; relational database: Virtuoso) that offer the queried. models [117]. These systems were compared using The high-level solutions and languages we de- LUBM, a well-known and used synthetic RDF bench- scribe are able to translate into efficient computa- mark[119]. The authors make interesting obser- tions. They bring the benefit of standardizing and vations: the verbosity of implementing queries in approaching theoretical representations to query spe- Sparksee due to its design of delegating the ex- cification. However, such translation is not direct, ecution plan implementation to the developer, as as the nature of the computation to be performed opposed to using declarative languages like Cypher; on the graph and the type of algorithm to execute the importance of cost-based query optimization wi- have a profound impact on how computational re- thin query engines. Query patterns used in the lit- sources should be optimized. erature fall in the following categories [286, 117]: single triple patterns satisfying a given condition; 4. GRAPH ALGORITHMS: NATURES matches on the nodes that are adjacent and edges AND TYPES that are incident to a given node; triangles, which There are several aspects inherent to graph-pro- look for three nodes adjacent to each other; paths cessing tasks. Graphs have properties which may be of fixed or variable length. extrapolated using specific algorithms, from com- We see in Table ?? that Cypher and Gremlin puting the most important vertices (e.g. using an have seen broad use in both storage and analytics arbitrary function like PageRank [218]), finding the tasks, with the latter also used in RDF databases. biggest communities (for which there is a choice of SPARQL is mostly supported by RDF databases and many algorithms) or the most relevant paths (for some frameworks and libraries have interoperabil- a definition of relevancy). An algorithm that pro- ity with it. The GraphQL compatibility row in Ta- cesses the whole graph (as opposed to localized in- ble ?? signals that the systems with check marks formation queries expressed with graph query lan- have clients for its structure. GraphQL’s decoupling guages seen previously) is typically executed in par- of front-end and back-end has support by GraphDB allel fashion when the resources for parallelism are and also Neo4j Cypher [210] and OrientDB Grem- available. When implementing these algorithms, lin. We did not present an exhaustive list of the whether the developer manually implements the par- RDF-specific graph databases; only the most relevant allelism or merely uses such a functionality offered with relationships to most graph query languages. by an underlying framework (e.g. Apache Spark[108] There have been studies comparing the perfor- or Apache Flink[51]), some challenges must be con- mance of graph query languages between themselves sidered. This means that while the field of graph and native access. Comparisons have been made be- processing is developed with the goal of improving tween Cypher, Gremlin and native access regarding how we manipulate and extract value from graph- the ease in expressing queries and the performance based data, as the techniques to achieve this end gains in Neo4j [131]. The authors subjectively note become more refined, other aspects of graph struc- that Cypher will feel more natural for developers tures gain prominence as challenges to them. with existing experience with SQL languages and We list and comment inline here the major types could be considered more high-level than Gremlin. of challenges of parallel graph processing identified They further observe that Gremlin queries achieved in a previous study [178]: better performance than with Cypher in some query types. There have been other studies on the proper- 1. Data-driven computations: a graph has ver- ties of graph query languages across the decades [7, tices and edges which establish how compu- 120, 298, 12, 11], which emphasizes the relevance tations are performed by algorithms, making of the theoretical and pragmatic aspects of graph graph applications data-driven. We see this query language development, whose renewed inter- observation shift the focus to data - what shou- est by industries is proof of it being an important ld the elementary unit of computation be? In part of extracting value from data. We end this this survey we go over multiple solutions in the section on graph query languages by again drawing literature, considering the computation from attention to the recent global coordination to cre- the perspective of vertices [182, 244, 51, 108], ate a standard graph query language - it draws from edges [240] and sub-graphs [279]. the lessons of different graph query languages and involves the collaboration of many academic and in- 2. Irregular problems: the distribution of edges and vertices usually does not constitute uni- form graphs that form embarrassingly paral- Instances of this are for example calculating lel problems, whose benefits from simple par- single-source shortest-paths (SSSP), k-hop nei- allelism are easier to achieve [296]. We note ghborhood or breadth-first searches (BFS). that after defining the unit of computation in a graph, care needs to be taken when assign- • Graph analysis. Algorithms falling into this ing parts of the graph to different processing scope aim at understanding the structural prop- units. Skewed data will negatively impact load erties and topology of the graph. They may balance [152] unless tailored approaches are be executed to grasp properties like the diam- undertaken, which take into account different eter (greatest distance between any pair of ver- types of graph properties such as scale-free [96] tices), density (ratio of the number of edges |E| in their designs [107, 54]. with respect to the maximum possible edges) or degree distribution. 3. Poor locality: locality-based optimizations of- fered by many processors are hard to apply • Components. Concept: a connected compo- to the inherently irregular characteristics of nent is a subset of graph vertices for which graphs due to poor locality during computa- there is a path between any pair of vertices. tion. We believe it is important to mention Finding connected components is relevant to how this manifests when using distributed sys- detect frailties in the networks that graphs rep- tems (clusters) to process the graphs. To mit- resent. The connections between these compo- igate this, techniques may be used for exam- nents are called bridges, which if removed, will ple to replicate specific vertices based on prop- separate connected components. erties such as their degree, or to use specific • Communities. The groups called communities graph partitioning strategies when working wi- consist of sets of vertices that are close to each th vertex-centric approaches [266]. other within a community than to those out- 4. High data-access-to-computation ratio: the au- side it. There are different techniques to com- thors note that a large portion of graph pro- pute them such as minimal-cut sets and label cessing is usually dedicated to data access in propagation, among others. graph algorithms and so waiting for memory or • Centrality measures. These represent the im- disk fetches is the time-most consuming phase portance of a vertex with respect to the rest relative to the actual computation on a ver- of the network. Many definitions exist such as tex or edge itself. We note one approach [238] PageRank and betweenness centrality, for ex- to this problem that focused on balancing net- ample. There also heuristics to measure the work and storage latencies with computation relevance of edges such as spanning edge be- time to minimize the impact of underlying data tweenness [276]. accesses in a cloud computing setting. • Pattern matching. Related to algorithms aimed 4.1 Algorithms at recognizing or describing patterns, known Graph algorithms that execute globally over all as graph matching in this context. elements of a graph have a distinct nature from • Graph anonymization. To produce a graph those solved with graph query languages - the scope with similar structure but making it so the en- of computation is drastically different with respect tities represented by vertices and edges are not to the computational resources needed to satisfy identifiable. Two examples of anonymization it. We note, however, some graph databases such procedures are k-degree and k-neighbourhood as Neo4j have extensions like the Neo4j APOC Li- anonymity. brary for languages like Cypher to start algorithms with global computation from the graph query lan- Underlying the computations that take place to guage[137]. A previous survey on the scalability of solve graph-specific tasks lies the granularity. How graph processing frameworks [81, Sec. 3.3] defines are the vertices, edges and properties of graphs pro- a categorization of graph algorithms, which we re- cessed or stored? This required building a bridge produce here: from these mathematically-defined elements to the • Traversals. Starting from a single node, they bits and bytes of computers. employ recursive exploration of the neighbour- hood until a termination criteria is met, like reaching a desired node or a certain depth. 5. COMPUTATIONAL REPRESENTA- three arrays: one containing nonzero values; the sec- TIONS ond containing the extents of rows; the third storing column indices. Figure 2(d) shows a representation Underlying all these ways to extract information in this format (we omit the array the array contain- from graphs is their digital representation. It is im- ing the nonzero values as they are all one in this portant to understand the set of operations to be case). performed over the graph and its size in order to Let us consider that indices are zero-based. The guide the choice of representation. To represent the array on the left side is the column index array, edges, perhaps the two most well-known approaches where the pipe character | separates groups which are the adjacency list and adjacency matrix. The belong to each row of A. The second row (index choice of using an adjacency list or a matrix usually 1) of A has elements at position indexes 0 and 3 in depends on the amount of edges in the graph. A. Therefore, the second group of the column index Consider a given graph G = (V,E). If |E| is close array has elements [0, 3]. The array on the right is to the maximum number of edges that a graph can the row index array which has one element per row sustain (|E| ' |V |2), then it is a dense graph and in matrix A and an element which is the count of it makes more sense to choose the adjacency matrix nonzero elements of A at the end of the array (there (performance-wise). However, if the graph is sparse are variations without this count). For a given row (|E|  |V |2), where most nodes are not connected, i, it encodes the start index of the row group in it can be efficiently represented (storage-wise) with the column index array (on the left in Fig. 2(d)). an adjacency list. While the matrix consumes more This way, for example, the second row of matrix A space than the adjacency list, it allow for constant- (Fig. 2(b)) has row index 1 in A. Then, looking at time access. We show in Fig. 2 an example of dif- the row index array (the one on the right), as the ferent representations for the same graph. second row of matrix A has row index 1, we access Figure 2(a) shows a sample graph G, for which the elements with indices [1, 2] in the row index ar- the adjacency matrix is shown in Fig. 2(b) and the ray, which returns the pair (1, 3), indicating that corresponding adjacency list in Fig. 2(c). The first the second row (index one) of A is represented in row of the adjacency matrix A represents the outgo- the column index array starting (inclusive) at index ing edges of vertex 1, which is connected to vertex 1 and ending at index 3 (exclusive). If we look at 2. It is common in the literature [64, Ch. 22] to use the column index array and check the elements from the subscript notation A to refer to the presence i,j index 1 (inclusive) to 3 (exclusive), we get the set of of a specific edge in matrix A (the notation is rele- values {0, 3}. And if we look at the second row in vant for theoretical purposes even if using another A, column index 0 and column index 3 are exactly type of representation) starting from vertex i and the positions of the edges in A for that row. Gener- targeting vertex j: ally, for a matrix M’s row index i, we access indices ( 1 if there is an edge from i to j, [i, i+1] in the row index array, and the returned pair Ai,j = (1) 0 otherwise. dictates the starting (inclusive) and ending (exclu- sive) index interval in the column index array. The Matrix A also takes on a particular configuration set of elements in that interval in the column index depending on the graph being directed or undirected. array contains the indices of the columns with value In the later case, there is no explicit sense of source 1 for row index i in M. We point the reader to [43] or target of an edge, leading to symmetry in ma- for details on its representation and construction. trix A. Implementations of graph processing sys- There is also the compressed sparse column (CSC), tems often represent undirected graphs as directed which is similar but focused on the columns, as the graphs such that the undirected edge between a pair name suggest. of vertices is represented by two directed edges in Other approaches take advantage of domain-spe- opposite directions between the pair. cific properties of graphs. Such is the case of Web- There are more space-efficient ways to represent Graph[34], which exploits certain properties of web a graph, and they become a necessity when explor- graphs to represent them with increased compres- ing the realm of big graphs. The choice between sion. An important property they exploit is locality, an adjacency list or matrix is bound to the density as many links stay within the same domain, that is, of the graph. But to justify other representation if the web graph is lexicographically ordered, most types, factors such as graph size, storage limitations links point close by. Another property is similarity: and performance requirements need special focus. pages that are close by in the lexicographical order The compressed sparse row (CSR), also known as are likely to have sets of neighbours that are simi- the Yale format, is able to represent a matrix using   1 2 0 1 0 0 0 0   2 1 4  1 0 0 1 0 0    3 1 2 2 1 6  1 1 0 0 0 0     0 0 1 0 0 1  4 3 6    0 0 0 0 0 1  5 6   4 3 5 1 0 0 0 1 0 6 1 5

(a) Sample graph G (b) Adjacency matrix A (c) Adjacency list

1 0 3 0 1 2 5 5 0 4 0 1 3 5 7 8 10 (d) Compressed sparse row

Figure 2: Simple computational representations of sample directed graph G shown in (a). lar. The study performed with WebGraph also high- cessing performance while making use of compres- lighted, among other facts, the following: similarity sion. It achieved compression ratios similar to Web- was found to be much more concentrated than pre- Graph while reaching speedups of more than 2x. viously thought; consecutivity is common regarding The authors achieve results by applying logarithm- web graphs. The properties of ordering (and differ- based approaches to different graph elements. They ent techniques to produce them) have also been ex- describe its application on fine elements of the ad- ploited by the same authors to obtain compression jacency array (the basis of Log(Graph): vertex IDs, with social networks. WebGraph was used in an ex- offsets and edge weights. From information theory, tensive analysis of many different data sets, which the authors note that a simple storage lower bound were made available online by the Laboratory for can be the number of possible instances of an entity, Web Algorithmics [162, 34, 32, 30, 31]. meaning the number of bits required to distinguish The k2-tree is another data structure employed to them. Using this type of awareness on the different represent and efficiently store graphs [39]. It may be elements that represent an adjacency array and by used to represent static graphs and binary relations incorporating bit vectors, the authors present a C++ in general. It has been used to represent binary library for the development, analysis and compari- relations like web graphs, social networks and RDF son of graph representations composed of the many data sets by internally using compressed bit vectors. schemes described in their work. Conceptually, we recursively subdivide each block There were relevant techniques for graph com- of a graph’s adjacency matrix until we reach the pression the literature on graph compression [94, 14, level of individual cells of the matrix. The idea is 41, 154, 92, 169] with the WebGraph framework [34, to divide (following an MX-Quadtree strategy [247, 33] as one of the most well-known, and more re- Sec. 1.4.2.1]) the matrix in blocks and then assign cently the k2-tree structure [39, 130, 38, 104], only 0 to the block if it only contains zeros (no edges) later was the focus cast on being able to represent or 1 if it contains at least an edge. We show in big graphs with compression while allowing for up- Fig. 3 a sample adjacency matrix on the left and dates. Furthermore, if we add the possibility of dy- the corresponding k2-tree representation of the de- namism of the data (the graph is no longer a static composition. This representation of the adjacency object that one wishes to analyze) to the factors 2 matrix is actually a k -tree of height h = dlogk ne, guiding representation choice, then it makes sense where (n = |V | and) each node contains a single to think about how to represent a big graph in com- bit of data. It is 1 for internal nodes and 0 for mon hardware not only for storage purposes but leaves, except for the last level, in which all nodes also for efficient access with mutability. Works such are leaves representing values from the adjacency as VeilGraph [61] approach the concepts of efficient matrix. It is a data structure that also efficiently representations by for example incorporating sum- matches the properties of sparseness and clustering mary graph structures to reduce the total performed of web graphs. computations in the context of graph updates. Another proposal, Log(Graph) [28] is a graph A dynamic version of the k2-tree structure was representation that combines high compression ra- proposed for this purpose [37]. Using compact rep- tios with low overhead to enable competitive pro- resentations of dynamic bit vectors to implement   1 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0     0 0 1 0 0 0 0 0       0 0 0 0 0 0 0 0  1 0 0 1    0 0 0 0 1 0 0 0     0 0 0 0 0 0 0 1  1 0 0 1 1 1 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1

2 (a) Decomposition (k = 2) (b) k -tree (h = dlogk2 ne = dlog4 64e = 3)

Figure 3: A sample adjacency matrix (n = |V | = 64) and corresponding k2-tree representation. this data structure, the k2-tree was used to pro- resented as graphs, when they initially would vide a compact representation for dynamic graphs. seem to be the type of data that is perfect for However, this representation with dynamic compact relational database systems. bit vectors suffers from a known bottleneck in com- • Very large graphs are present from small to pressed dynamic indexing [204]. It suffers a loga- large enterprises, which contradicts the no- rithmic slowdown from adopting dynamic bit vec- tion that very large graphs are only present in tors. A recent comparative study on the graph op- the challenges of large corporations like Ama- erations supported by different k2-tree implementa- zon, Google. We can therefore claim that there tions has also been performed [62]. This work also is still immense value regarding innovative so- presented an innovative take on implementing dy- lutions that allow the processing of big graphs namic graphs by employing the k2-tree data struc- under the potentially-scarce resources held by ture with a document collection dynamization tech- the smaller companies and startups. nique [202], avoiding the bottleneck in compressed dynamic indexing. • As a consequence, the ability to process ve- To better understand the dynamics of these rep- ry large graphs efficiently is still the biggest resentations, we next analyze recent work on profil- limitation of existing software. We see in this ing the needs of practitioners of graph processing. the relevance of novel techniques underlying the design of graph processing systems (both 6. PROFILE OF DEVELOPERS AND RE- distributed and not) to reduce overhead fac- SEARCHERS tors such as storage and network latency, co- ordination (between threads in CPUs and be- There has been a recent study on the profile of tween nodes in a cluster) and redundancy (for practitioners in both academia and industry [243]. example consider the optimization of replica- To our knowledge, it has been the first one to iden- tion factor in vertex-centric systems). tify the profile of users and the types of compu- tational tasks they need to process over graphs. • The second most desired ability is visualiza- Specifically, this is the result of an online survey tion, related to challenges in graph query lan- aiming to establish: 1) the types of graph data used; guages. Visualizing all vertices in a graph qui- 2) the computations users run on their graphs; 3) ckly becomes a prohibitive endeavor as graph the softwares used to perform computations; 4) the size increases. However, being able to effi- major challenges faced by users when processing the ciently visualize structural aspects of the graph graph data. The authors share five main findings as (e.g. communities) or relevant nodes under a result of their online survey: some criteria (e.g PageRank [218]) is impor- tant. These same capabilities are also rele- • There is variety in the data represented with vant for graph query languages. When cou- graphs. From participants, they found that pled with graph databases (e.g. Neo4j), there enterprise data such as orders, transactions and may be a visualization layer to give direct feed- products are a very common form of data rep- back of the queries that are being written. We note this potential in visualization gains an- of these steps will vary depending on the type of other dimension if the graph is changing - it vertex-centric model used (scatter-gather, gather- may be more efficient to update existing visu- apply-scatter). alizations with approximations based on graph update deltas as opposed to recalculating the 7.2 Model: Superstep Paradigm visualizations from the beginning on the up- In a superstep S, a user-supplied function is exe- dated graph. cuted for each vertex v (this can be done in parallel) that has a status of active. When S terminates, all • Relational database management systems (R- vertices may send messages which can be processed DBMSes) are still relevant in managing by user-defined functions at step S + 1. and processing graphs. 7.3 Model: Scatter-Gather This survey [243] was important as it highlighted that graphs falling in the realm of big data are still Scatter-gather shares the same idea behind vertex- the top priority for innovative solutions, thus fur- centric but separates message sending from message ther motivating the importance of concisely defin- collecting and update application [270]. In the scat- ing the landscape of graph processing and different ter phase, vertices execute a user-defined function ways in which the need to process big graphs man- that sends messages along outgoing edges. In the ifests. The processing of a graph can take place gather phase, each vertex collects received messages from different perspectives such as vertices, edges and applies a user-defined function to update vertex or parts of the graph. These perspectives can more state. naturally represent one type of graph computation 7.4 Model: Gather-Apply-Scatter than other - herein we analyze them. Gather-Sum-Apply-Scatter (GAS) was introduced 7. GRAPH PROCESSING: COMPUTA- by PowerGraph [107] and was aimed at solving the limitations encountered by vertex-centric or scatter- TIONAL UNITS AND MODELS gather when operating on power-law graphs. The Here we detail the most relevant paradigms and discrepancy between the ratios of high-degree and computational units used to express computation low-degree vertices leads to imbalanced computa- in graph processing systems. Programming models tional loads during a superstep, with high-degree for graph processing have been studied and docu- vertices being more computationally-heavy and be- mented in the literature [153, 128]. They define coming stragglers. GAS consists of decomposing properties such as the granularity of the unit of the vertex program in several phases, such that com- computation, how to distribute it across the cluster putation is more evenly distributed across the clus- and how communication is performed to synchro- ter. This is achieved by parallelizing the compu- nize computational state across machines. tation over the edges of the graph. In the gather phase, a user-defined function is applied to each of 7.1 Unit: Vertex-Centric (TLAV) the adjacent edges of each vertex in parallel. The vertex-centric paradigm, also known as think- like-a-vertex (TLAV), debuted with Google’s Pre- 7.5 Unit: Edge-Centric (TLEV) gel system [182]. An open-source implementation The edge-centric approach, also referred as think- of this model known as Apache Giraph [59] was then like-an-edge (TLEV), was popularized by systems offered to the public. Other example systems that like X-Stream [240] and Chaos [238] which specify were created using that model are GraphLab [176], the computation from the point-of-view of edges. PowerGraph [107], PowerLyra [54]. As the unit of These systems made of use of this paradigm to op- computation is the vertex itself, the user algorithm timize the usage of secondary storage and network logic is expressed from the perspective of vertices. communication with cloud-based machines to pro- The idea is that a vertex-local function will receive cess large graphs. information from the vertex’s incoming neighbours, perform some computation, potentially update the 7.6 Unit: Sub-graph-Centric (TLAG) vertex state and then send messages through the The previous models are subjected to higher com- outgoing edges of the vertex. A vertex is the unit munication overheads due to being fine-grained. It of parallelization and a vertex program receives a di- is possible to use sub-graph structure to reduce these rected graph and a vertex function as input. It was overheads (also known as component-centric [128]). then extended to the concept of vertex scope, which In this category, the work of [153] denotes two sub- includes the adjacent edges of the vertex. The order graph-centric approaches: partition-centric and nei- ghbourhood-centric. Partition-centric instead of fo- ers. The communication between them then de- cusing on a collection of unassociated vertices, con- pends on the number of units that are replicated. siders sub-graphs of the original graph. Information We observe that partitioning has a cyclical nature from any vertex can be freely propagated within its to itself in the scope of distributed processing: one physical partition, as opposed to the vertex-centric may wish to execute graph partitioning over a dis- approach where a vertex only accesses the informa- tributed system as part of a domain-specific prob- tion of its most immediate neighbours. This allows lem; however, before that graph algorithm can ex- for reduction in communication overheads. Ulti- ecute, the graph data also incurs partitioning fol- mately, the partition becomes the unit of parallel lowed by distribution in the underlying (distributed) execution, with each sub-graph being exposed to computational infrastructure. While the study of a user function. This sub-graph-centric approach graph partitioning is not recent, it gained additional is also known as think-like-a-graph [279] (TLAG). depth in the last decade as the number of factors Neighborhood-centric, on the other hand, allows for guiding optimization of partitioning increased with a physical partition to contain more than one sub- the complexity of graph processing systems. We ex- graph. Shared state updates exchange information plore partitioning as a relevant dimension to classify between sub-graphs of the same partition, with repli- graph processing systems as they must approach it cas and messages for sharing between sub-graphs in order to enable parallel computation over graphs. that aren’t in the same partition. For completion, The way it is approached becomes a distinctive fea- we refer the reader to an analysis of distributed al- ture between the systems. gorithms on sub-graph centric graph platforms [150]. Graph partitioning aims to divide the nodes of the graph into mutually-exclusive groups and to 7.7 Model: MEGA minimize the edges between partitions. This is ef- The MEGA model was introduced by Tux2 [299], fectively a grouping of the node set of the graph, a system designed for graph computations in ma- which can represent a minimization of communica- chine learning. The model is composed of four func- tion between partitions, with each partition for ex- tions defined by the user: an exchange function ample assigned to a specific worker in a distributed which is applied to each edge and can change the system. Partitioning is a task that produces groups value of the edge and adjacent vertices; an apply of nodes, but grouping nodes is not only achieved function to synchronize the value of vertices with with partitioning. We note that other terms ex- their replicas; a global sync function to perform ist in the literature such as clustering and com- shared computations and update values shared among munity detection. They are not interchangeable, partitions; a mini-batch function to indicate the ex- for if a clustering algorithm breaks down the graph ecution sequence of other functions in each round. into three clusters, it does not necessarily hold true There are graph processing systems that offer more that each cluster represents its own community. As than one type of model. To achieve parallelism and an example, executing a clustering algorithm over harness multiple machines in clusters, it is neces- a social network graph will result in a number of sary to define how to break down the graph - we clusters. If each cluster represents for example a provide a high-level overview of methods employed different continent, that does not necessarily mean in most well-known graph processing solutions each cluster represents one single community. Com- munity detection algorithms, on the other hand, 8. DIMENSION: PARTITIONING consider properties such as the density and inter- connections within communities. While clustering Graph partitioning is an important problem in and community detection aim to identify similari- graph processing, and this importance manifests in ties between nodes, their underlying assumptions of two formats. The first, is out of a user’s domain the graph are not equal, even though proposals have application with the goal of splitting the graph in been made to map between these two tasks [118]. parts which provide a relevant view of the data. Graph clustering shares similarities with graph par- The second, is when partitioning may be consid- titioning in the sense that both produce groups of ered as a hyper-algorithm, that is, it is employed to nodes. However, the objective functions they use divide the parts of the graph across a computational are defined differently and subject to different con- infrastructure, typically within the distributed sys- straints. Graph partitioning, on which we focus, for tems’ coordination layer, or across processing units example, requires that the number of groups (parti- or cores within machines. Machine loads in dis- tions) is known beforehand and is typically subject tributed graph processing systems depend on the to more constraints. way computational units are distributed across work- An earlier work on balanced graph partitioning [9] tions which have their adjacent edges. Care must be defines the problem as (k, v)-balanced partitioning: taken to balance the number of edges per partition to divide the vertices of the graph into k compo- (its measure of size) and to minimize the number of nents of almost equal size, with each of size less than vertex copies. This objective may be formulated as n c · k for a given constant c > 1. It is a balanced such [266]: k-way partitioning problem which has been studied in the literature [46]. To consider the partition- 1 X min |P (v)| (4) ing problem as a challenge to enabling distributed P |V | v∈V processing, it is necessary to ask if the goal is to distribute the vertices (edge cut model - EC) or |E| s.t. max |{e ∈ E|P (e) = pi}| ≤  (5) the edges (vertex cut model - VC) of the graph pi k across machines in order perform it. We provide Vertex-cut achieves better performance than edge- detail into these problem formulations with an ex- cut for natural graphs such as those representing ample of vertex-cut and edge-cut in Fig. 4. Fur- web structure and social networks [266]. thermore, different combinations between compu- tational unit and cut model are possible: vertex- 8.3 Hybrid-Cut (HC) cut can be used to process in a vertex-centric [201] Hybrid strategies can be employed to perform the or edge-centric [107] way, and the same is possible cuts. They can for example be guided with heuris- using edge-cut used to partition a graph where com- tics such as vertex degree in order to decide what to putation is vertex-specific [22, 187] or edge-specific. do with them. The PowerLyra [54] system for ex- ample allocates the incoming edges of vertices with 8.1 Edge-Cut (EC) low degree in a worker. It uses edge-cut for ver- Balanced k-way partitioning may be defined for tices of low-degree and vertex-cut for high-degree edge-cut partitioning, which is associated to vertex- vertices. centric (TLAV) systems, the most common com- 8.4 Stream-based Partitioning putational model in graph processing systems [182, 108, 244]. We reproduce the definition of [266, Sec. In these methods of partitioning, vertices or edges 2] for this case, where for a given graph G = (V,E), in the graph are analyzed in succession in a stream. we wish to find a set of partitions P = {P1,P2,...,Pk}. Placement decisions are made online, that is, when These partitions must be pairwise disjoint and their the vertices or edges appear in the stream, and union is equal to V while following these condi- the decisions are based on the location of previ- tions [266]: ous elements. This is done under the assumption that there will be no information on the edges or vertices that will arrive in the flow of the stream. min |{e|e = (vi, vj) ∈ E, vi ∈ Px, vj ∈ Py, x 6= y}| P This type of method can rely on edge-cut parti- (2) tioning (e.g. Random heuristic and the Linear Deterministic Greedy [268], Gemini which uses max |P | s.t. i i ≤  (3) chunk-based assuming adjacency list model [311], 1 Pk [281]), vertex-cut partitioning (e.g. i=1 |Pi| Fennel Grid k heuristic [142], PowerGraph greedy heuristic [107], Depending on the application objective for which Graphbuilder [142] placing the edge in the smallest this partitioning type will be performed, Eq. 3 should partition, HDRF [227] method which takes into con- be adapted. For example, in the case of machines sideration vertex degrees) and there are aspects of having different characteristics, it should be consid- these methods that will have different approaches ered that the load of any machine will be less than regarding how this is achieved with parallel and the maximum computing power. Or if the graph distributed execution. Stream-based partitioning is structure is stored in secondary memory, the inter- also used as a good choice for loading the graph as est is on having balanced size partitions with high it does not have to be fully loaded in memory for speed sequential storage access and decreasing the partitioning. number of cut edges is no longer a focus. 8.5 Distributed Partitioning 8.2 Vertex-Cut (VC) Many distributed partitioning algorithms are based In the vertex-cut model, the goal is to distribute on label propagation [32, 114, 171, 312], with vari- edges across partitions. They are placed in differ- ations such as how the specific labeling of a ver- ent partitions, with vertices being copied in parti- tex should be influenced by its neighbours, if it 2 1 6 2 1 6 1 6

4 3 5 4 3 4 5

(a) Edge-cut on G (b) Edge-cut replication

2 1 6 2 1 6

4 3 5 4 3 3 5

(c) Vertex-cut on G (d) Vertex-cut replication

Figure 4: Depiction of vertex-cut and edge-cut over the sample graph G. should also be influenced by the label’s global rep- techniques having been proposed for that purpose resentation in the graph and also constraints on (among others, xDGP [287] to repartition massive the minimum and maximum sizes required for par- graphs to adapt to structural changes, GPS [246] titions. For example, Revolver, which performs which reassigns vertices based on communication vertex-centric graph partitioning with reinforcement patterns, X-Pregel [22] with reduction of message learning, assigns an agent to each vertex, with agents exchanges and dynamic repartitioning). Dynamic assigning vertices to partitions based on their prob- partitioning methods have the advantage of out- ability distribution (these are then refined based on putting very good load balancing and communica- feedbacks [201]). The authors of [266] note that tion cost reductions due to considering heteroge- other approaches consider the partitioning problem neous hardware and runtime characteristics. as a multi-objective and multi-constraint problem, achieving better results compared to one-phase meth- 8.7 Partitioning: Summary ods [264]. Distributed partitioning systems are good Employed graph partitioning strategies vary, with for when partitioning is performed once and then different systems offering different solutions. Among calculations are repeatedly performed. performance-impacting factors [266], we have the number of active vertices and edges influencing ma- 8.6 Dynamic Graph Partitioning chine load. At the same time, communication will When the graph is no longer static, vertex and be more expensive depending on how replication of edges may be added or removed as time passes - edges and vertices is performed. Partitioning must this is especially true in social networks. This im- balance communication and machine loads. The plies that for graphs from which we need to per- partitioning challenge in vertex-centric systems is form computations as they evolve, the original par- relevant due to how widespread this model is. The titioning may become inefficient. With predictable authors of [266] note three major approaches for algorithm runtime characteristics, it becomes fea- big graph partitioning: a) partitioning the graph sible to keep close the vertices which will be used serially in a single pass and permanently assigning together in the same supersteps, using for example the partition on the first time an edge or vertex graph traversal algorithms. But when this is not the is assigned (stream-based); b); methods that parti- case, systems can be designed for example to mon- tion in a distributed way; c) dynamic methods that itor the load and communication of the machines adapt the partitions based on monitoring the load and migrate vertices as appropriate, with different and communication of machines during algorithm execution. The way the distribution is achieved and or event, only the elements with valid times- data is represented will be a factor in going beyond tamps under required constraints are consid- the read-eval-write loop. In this scope, a dynamic ered for computation. The work of [161] intro- method would be necessary as a basis to develop the duces the temporal graph as a representation properties we described. For an in-depth analysis of encoding temporal data into the graph while partitioning methods, vertex cut models and their retaining the temporal information of the orig- relation to the dynamic nature of data, we invite inal data. They present metrics that can be the reader to read [266]. used to study temporal graphs and use the rep- Being able to decompose the graph is a corner- resentation to explore dynamic temporal prop- stone for efficient and distributed computation of erties of data using graph algorithms without graphs. An equally-important aspect that deter- requiring data-driven simulations. Immortal- mines how we must approach the computation is Graph [196] is a storage and execution engine the possible dynamism of the graph. A static graph designed with temporal graphs in mind, hav- over which we want to perform analytics is a sce- ing achieved greater efficiency than database nario different from maintaining a large graph avail- solutions for graph queries. ImmortalGraph able for separate queries and susceptible to updates. schedules common bulk operations in a way to maximize the benefit of in-memory data 9. DIMENSION: DYNAMISM locality. It explores the relation between lo- We include and consider dynamism a relevant di- cality, parallelism and incremental computa- mension of graph processing due to there existing tion while enabling mining tasks on temporal different meanings associated to it in the literature graphs. For more information and reach on the and for which different systems can be attributed. topic of temporal graphs, we direct the reader While one may consider static graphs to be com- to [197]. pletely unrelated to dynamism, there is in fact a • Streaming graph algorithms [95]. With these, relation to it due to what is known as stream pro- the common scenario starts from an empty cessing. For example, a graph processing system graph without edges (and a fixed set of ver- may ingest an unbounded stream of edges and up- tices). For each algorithm step, a new edge date statistics over the stream (e.g., keeping a trian- is inserted into the graph or an edge is re- gle count updated [2]), but stream processing may moved. It is desired that these algorithms also take place in static graph processing. This are developed to minimize parameters such as is the case with approaches that process a static graph data structure storage, the time to pro- graph but process its elements from a stream per- cess an edge or the time to recover the final so- spective (e.g., Chaos [238] and X-Stream [240] with lution. There exist several systems which pro- their edge-centric approach). Considering if a sys- cess streaming graph computations - we note tem targets graphs that change or are immutable also for the reader a recent framework for com- (static) is an obvious way to separate graph pro- paring the systems aimed at this type of dy- cessing systems when classifying them. However, namism [86]. The STINGER data structure has this dimension is actually a spectrum between the been used for streaming graphs as well [84]. immutable (e.g. stream-based perspectives to pro- cess static graphs) and the changing - for example, • Sketching and dynamic graph streams. Sketch- is the whole graph structure kept in memory (or ing techniques [3] may be applied to the edge secondary storage) in a single machine (or across incidence matrix of the input graph to approxi- cluster nodes), or is it discarded by proxy of some mate cut structure and connectivity. The idea criteria (and thus one simply updates mathemati- is to consume a stream of events in order to cal properties of the graph using only recent infor- generate a probabilistic data structure repre- mation from the stream)? For this spectrum, the senting properties of the graph. authors of [27] cover definitions found in the litera- • Multi-pass streaming graph algorithms. In this ture: type of algorithm, all updates are streamed • Temporal graphs. These are, in essence, static more than once in order to approximate the graphs which have annotated temporal infor- computation quality of the solution. Addi- mation which allows for recreating the domain tional complexity can emerge on how the strea- represented by the graph at any given point ming model behaves - it can for example al- in time. It is not structurally-changed while low for the stream to be manipulated across doing so; it means that for a given time range passes [1] or to stream sorting passes [74]. • Dynamic graph algorithms. For these types, dra [163]), others in a centralized server (but allow- the focus is cast on being able to approximate ing cluster nodes for the specific purpose of redun- combinatorial properties of the graph [27] (e.g., dancy). They employ schemes to store the graph connectivity, shortest path distance, cuts, spec- efficiently while offering transaction mechanisms to tral properties) while processing insertions and operate over the graph and to perform queries. deletions. The objective with this type of al- The latter type (b) is seen in big (graph) data pro- gorithm is to quickly integrate graph updates. cessing systems like Spark (GraphX library) [300] Ringo [225] is a single-machine analytics sys- and Flink (Gelly library) [51]. The mentioned tem that supports dynamic graphs. names are all distributed processing frameworks that can take advantage of multi-core machines and clus- While partitioning and dynamism are relevant as- ters. These systems and their libraries allow for pects, the scope of graph processing solutions in expressive computation over graphs in few lines of both industry and academia was shaped by the type code. Many of the systems come with their sets of executed workloads. of graph algorithms, allowing for the composition of workflows while abstracting away many details 10. DIMENSION: WORKLOAD from the programmer (regarding distributed com- The type of workload performed by a graph pro- putation orchestration and the internal implemen- cessing system also plays an important role in clas- tation of the graph algorithms). sifying them. The type of task performed by graph It is important to consider two definitions regard- databases is different from the systems that run ing the nature of computational tasks: online ana- global algorithms over them. The concept of ana- lytical processing (OLAP) and online transaction pro- lyzing a graph takes on different contexts depending cessing (OLTP). OLAP is an approach to enable an- on user needs. We note that when a graph is to be swering multi-dimensional analytical queries quickly. processed, the scope of its data analysis usually falls Among its instances we may find tasks such as busi- in these two categories: ness reporting for sales, management reporting, busi- ness process management [26], financial reporting (a) To retrieve instances of domain-specific rela- and others. OLTP, on the other hand, refers to sys- tions in the graph (e.g. pattern matching, tems that enable and manage transaction-oriented multi-hop queries). These are usually found applications, with transaction meaning in a compu- in graph databases, with an emphasis on opti- tational context the atomic state changes that take mization of data query and storage for online place in database systems. OLTP examples include transaction processing scenarios. This is of- retails sales and financial transaction systems, and ten accompanied with the use of graph query applications of this type tend to be high-throughput languages (GQLs) to execute queries that re- and update/insertion-intensive in order to provide turn a view on the graph and also potentially availability, speed, recoverability and concurrency- producing effects on it. [66]. (b) To execute an algorithm over the whole graph The earlier type a) of graph processing task may (e.g. PageRank, connected components, de- be associated to OLTP systems as the goal is to tecting communities, finding shortest paths). store representations of graphs by quickly ingesting The solutions for this task, performance-wise, new information, efficiently representing it regard- aim to achieve high-performance computational ing space consumption and access speed, and being throughput, whether using distributed systems able to execute updates under ACID properties (or or a single-machine setup. It is a focus leaning a subset of those). For this type of task a), one on the data analytics aspect. may find numerous graph databases to match the description, such as those for designed for seman- The former (a) is a common scenario in graph tic (RDF) representations [213, 271, 277, 136], or for databases such as Neo4j [294] and JanusGraph [143], property graph models [198, 199, 155, 49, 242, 16, among others. These databases offer graph query 143, 67, 221, 188, 134, 83, 278], both [25] and also languages (usually even allowing interchangeabil- other specific purposes [109]. The latter type of task ity between languages) such as Cypher or Grem- b) may be associated to OLAP, where there is a focus lin [131]. They are built to store the graph, some on extracting value from the data and the nature of with sharding (horizontal scaling) to distribute the the task is typically read-only. We include graph graph across the storage/computational infrastruc- processing systems (not databases) in this group of ture (some outsource the storage medium to data- OLAP-type tasks, even the systems which support base technologies such as HBase [106] or Cassan- mutability in graphs due to supporting dynamism tices that receive messages (the vertex order in any form. in which computation will take place within an There is a considerable overlap between OLTP- iteration). GRACE’s design targets both shared- type tasks and graph databases, and there is also an memory and distributed system scenarios, but overlap between OLAP-type tasks and graph process- the initial prototype focuses on shared-memory. ing systems. While the distinction between OLAP We did not find the source code available. and OLTP task types is not a dimension that per- fectly divides systems in the graph processing land- • Ligra [259] is a C++ lightweight graph process- scape, we note that such a distinction holds value in ing framework targeting shared-memory para- guiding future taxonomies of the graph processing llel/multi-core machines, easing the writing of system landscape, and for that reason we include it graph traversal algorithms. This framework as a dimension. offers two map primitives to operate a given The way these three dimensions are accounted logic on vertices (VertexMap) and edges (Edge- for influence the design of graph processing sys- Map) and supports two data types: the tradi- tems. Many different architectures exist, for which tional graph G = (V,E) as we described in we share an exhaustive list of specific solutions, from an earlier section, and another one to repre- single-machine systems to parallel processing in clus- sent subsets of vertices. The interface is de- ters and storage in tailor-made graph databases. signed to enable the processing of edges in dif- ferent orders depending on the situation (as 11. SINGLE-MACHINE AND SHARED- opposed to Pregel or Giraph). The code of MEMORY PARALLEL APPROA- Ligra represents in-edges and out-edges as ar- CHES rays, with in-edges for all vertices being par- titioned by their target vertex and storing the • GraphLab [174] was published as a framework source vertices, and the out-edges are in an ar- (implemented in C++) for parallel machine learn- ray partitioned by source vertices and storing ing and later extended to support distributed the target vertices. While the authors claim to settings while retaining strong data consistency have achieved good performance results, they guarantees [176]. The authors evaluate it on mention Ligra does not support algorithms Amazon EC2, outperforming equivalent Map- based on modifying the input graph. It is Reduce implementations by over 20X and ma- available [260] under the MIT License. tch the performance of specifically-crafted MPI implementations. GraphLab requires the whole • Ringo [225] is an approach for multi-core single- graph and program state to reside in RAM. machine big-memory setups. It is a high-per- It uses a chromatic engine so that no adja- formance interactive analytics system using a cent vertices have the same color and to en- Python front-end on a scalable parallel C++ able the efficient use of network bandwith and back-end, representing the graph as a hash ta- processor time. The authors evaluate it for ap- ble of nodes. It supports fast execution times plications such as Netflix movie recommenda- with exploratory and interactive use, offering tion, video co-segmentation and named entity graph algorithms in a high-level language and recognition. It is open-source [175] under the rich support for transformations of input data Apache License 2.0. into graphs. Ringo is open-source and avail- able [226] under the BSD License. • GRACE [290] is a synchronous iterative graph programming model, with separation of appli- • Polymer [305] is a NUMA-aware graph ana- cation logic and execution policies. Its design lytics system on multi-core machines that is includes the implementation (C++) of a paral- open-source [306] under the Apache License lel execution engine for both synchronous and 2.0 and implemented in C++. It innovated user-specified asynchronous execution policies. by differentially allocating and placing topol- GRACE stores directed graphs, and in its model ogy data, application-defined data and muta- and the computation is expressed and perfor- ble run-time graph system states according to med in a way similar to Pregel. It provides access patterns to minimize remote accesses. additional flexibility, by allowing the user to Polymer also deals with random accesses by relax synchronization of computation. This is converting the random ones into sequential re- achieved with user-defined functions which al- mote accesses using lightweight vertex repli- low updating the scheduling priority of ver- cation across the NUMA nodes. It was built with a hierarchical barrier for increased par- we mention what are, to the best of our knowledge, allelism and locality. The design also includes the most relevant works: edge-oriented balanced partitioning for skewed • Parallel Boost Graph Library (PBGL) [112] graphs and adaptive data structures in func- . It is an extension (C++) of Boost’s graph li- tion of the fraction of active vertices. It was brary. It is a distributed graph computation compared to Ligra, X-Stream and Galois on library, also offering abstractions over the com- an 80-core Intel machine (no hyper-threading) munication medium (e.g. MPI). The graph is and on a 64-core AMD machine. For different represented as an adjacency list that is dis- algorithms across several data sets, Polymer tributed across multiple processors. In PBGL, consistently almost always achieved the lowest vertices are divided among the processors, and execution time. each vertex’s outgoing edges are stored on the processor storing that vertex. PBGL was eval- • GraphMat [272] is a framework written in C++ uated on a system composed of 128 compute aimed at bridging the user-friendly graph an- nodes connected via Infiniband. It is avail- alytics and native hand-optimized code. It able [113] under a custom Boost Software Li- presents itself as a vertex-centric framework cense 1.0. without sacrificing performance, as it takes ver- tex programs and maps them to exclusively • CombBLAS [45]. A parallel graph distributed- use sparse matrix high-performance back-end memory library in C++ offering linear algebra operations. GraphMat takes graph algorithms primitives based on sparse arrays for graph an- expressed as vertex programs and performs gen- alytics. This system considers the adjacency eralized sparse matrix vector multiplication on matrix of the graph as a sparse matrix data them. It achieved greater performance than structure. CombBLAS is edge-based in the sense other frameworks such as 5-7X faster than Gra- that each element of the matrix represents an phLab, Galois and ComBLAS. It also achieved edge and the computation is defined over it. It multi-core scalability, being over 10X faster decouples the parallel logic from the sequen- than single-threaded implementation on a 24- tial parts of the computation and makes use core machine. It is open-source and available- of MPI. However, its MPI implementation does [273] under specific conditions by Intel. not take advantage of flexible shared-memory operations. Its authors targeted hierarchical • Mosaic [181] is a system for single heteroge- parallelism of supercomputers for future work. neous machines with fast storage media (e.g., It is available [44] under a custom license. NVMe and SSDs) and massively-parallel co- • HavoqGT [222] is a C++ system with techniques processors (e.g., Xeon Phi) developed to en- for processing scale-free graphs using distribu- able the processing of trillion-edge graphs. The ted memory. To handle the scale-free proper- system is designed explicitly separating graph ties of the graph, it uses edge list partitioning processing engine components into scale-up and to deal with high-degree vertices (hubs) and scale-out goals. It is written in C++ uses a com- dummy vertices to represent them to reduce pact representation of the graph using Hilbert- communication hot spots. HavoqGT allows al- ordered tiles for locality, load balancing and gorithm designers to define vertex-centric pro- compression and uses a hybrid computation cedures in a distributed asynchronous visitor model that uses both vertex-centric operations queue. This queue is part of an asynchronous (on host processors) and edge-centric opera- visitor pattern designed to tackle load imbal- tions (on co-processors). Mosaic is open-sour- ance and memory latencies. HavoqGT targets ce[180] under the MIT License. supercomputers and clusters with local NV- RAM. It is available [223] under the GNU Les- 12. HIGH-PERFORMANCE COMPU- ser General Public License 2.1. TING 13. DISTRIBUTED GRAPH PROCES- These systems are hallmarks of high-performance computing solutions applied to graph processing. SING SYSTEMS Their merits encompass algebraic decomposition of While the previous systems we detailed performed the major graph operations, implementing them and analytics and enabled the execution of graph algo- translating them across different homogeneous lay- rithms, they did so with a focus on specific hard- ers of parallelism (across cores, across CPUs). Here ware and distributed memory. We list here some of the most relevant state-of-the-art systems used for systems could benefit from offering tiered pro- graph processing in the scope of analytics (OLAP). gramming abstraction levels as in Naiad. Its Their use of different architectures (from using local low-level primitives allow for the combination commodity clusters to cloud-based execution) and of dataflow primitives (similar to those Veil- greater flexibility of deployment scenarios differen- Graph uses from Flink) with finer-grained con- tiate them from those of the previous section. The trol on iterative computations. An extension following systems are relevant names in the litera- to Flink’s architecture to offer this detailed ture, with Giraph being the first open-source im- control would enrich the abilities that our frame- plementation of the Pregel approach to graph pro- work is able to offer to users. cessing, and Spark and Flink being open-source general distributed processing systems with graph • Apache Flink [51], formerly known as Strato- processing APIs: sphere [5], it is a framework which supports built-in iterations [51] (and delta iterations) to • Apache Giraph [59] is an open-source Java im- efficiently aid in graph processing and machine plementation of Pregel [182], tailor-made for learning algorithms. It is licensed [99] under graph algorithms, supporting the GAS model the Apache License 2.0 and has a graph pro- and licensed [97] under the Apache License cessing API called Gelly, which comes pack- 2.0. It was created as an efficient and scalable aged with algorithms such as PageRank, single- fault-tolerant implementation on clusters with source shortest paths and community detec- thousands of commodity hardware, hiding im- tion, among others. Flink supports Java, Py- plementation details underneath abstractions. thon and Scala. It explicitly supports three Work has been done to extend Giraph from vertex-based programming models: think-like- the think-like-a-vertex (TLAV) model to think- a-vertex (TLAV) described as the most generic like-a-graph (TLAG) [279]. It uses Hadoop’s model, supporting arbitrary computation and MapReduce implementation to process graphs. messaging for each vertex; Scatter-Gather, whi- Giraph [97] allows for master computation, sh- ch separates the logic of message production arded aggregators (relevant when computing from the logic of updating vertex values, which a final result comprised of intermediate data may typically make these programs have lower from nodes), has edge-oriented input, and also memory requirements (concurrent access to the uses out-of-core computation – limited parti- inbox and outbox of a vertex is not required) tions in memory. Partitions are stored in local while at the same time potentially leading to disks, and for cluster computing settings, the non-intuitive computation patterns; Gather- out-of-core partitions are spread out across all Sum-Apply-Scatter (GAS), which is similar to disks. Giraph attempts to keep vertices and Scatter-Gather but the Gather phase paral- edges in memory and uses only the network lelizes the computation over the edges, the mes- for the transfer of messages. Improving Gi- saging phase distributes the computation over raph’s performance by optimizing its messag- the vertices and vertices work exclusively on ing overhead has also been studied [172]. It is neighbourhood, where in the previous two mod- interesting to note that single-machine large- els a vertex can send a message to any vertex memory systems such as Ringo highlight the provided it knows its identification. It sup- message overhead as one of the major reasons ports all Hadoop file systems as well as Amazon to avoid a distributed processing scheme. S3 and Google Cloud storage, among others. Delta iterations are also possible with Flink, • Naiad is an open-source [232] (Apache Licen- which is quite relevant as they take advantage se 2.0) dataflow processing system [203] offer- of computational dependencies to improve per- ing different levels of complexity and abstrac- formance. It also has flexible windowing mech- tions to programmers. It allows programmers anisms to operate on incoming data (the win- to implement graph algorithms such as weakly dowing mechanism can also be based on user- connected components, approximate shortest specific logic). Researchers have also looked paths and others while achieving better per- into extending its DataStream constructs and formance than other systems. Naiad is imple- its streaming engine to deal with applications mented in C# and allows programmers to use where the incoming flow of data is graph-ba- common high-level APIs to express algorithm sed[151]. logic and also offers a low-level API for perfor- mance. Its concepts are important and other • Apache Spark [303] and its GraphX [300] graph processing library, licensed [100] under the A- dataflows grants flexibility to program implementa- pache License 2.0. It is a graph processing tion and execution by decoupling the program logic framework built on top of Spark (a framework from how it is translated to the workers of a cluster, supporting Java, Python and Scala), enabling the graph libraries of these systems do not allow in low-cost fault-tolerance. The authors target an efficient way for a graph to be updated using graph processing by expressing graph-specific stream-processing semantics while also maintaining optimizations as distributed join optimizations the graph structure during computation. It is possi- and graph views’ maintenance. In GraphX, the ble to update graphs using these systems, but they property graph is reduced to a pair of collec- make use of batch processing APIs for which the tions. This way, the authors are able to com- dataflow graphs must not become excessively big (or pose graphs with other collections in a dis- else dataflow plan optimizers may be locked in the tributed dataflow framework. Operations such phase of exploring the optimization space of the exe- as adding additional vertex properties are then cution plan) and graph must be periodically written naturally expressed as joins against the collec- to secondary storage (as a solution to avoid having tion of vertex properties. Graph computations progressively bigger execution plans). and comparisons are thus an exercise in ana- Flink’s Gelly library has been used in GRADO- lyzing and joining collections. OP, which is an open-source [167] (Apache Licen- se 2.0) distributed graph analytics research frame- • GraphTau [141] is a time-evolving graph pro- work [149] under active development and provid- cessing framework on top of Spark (Java, Sca- ing higher-level operations. GRADOOP extends Gelly la). It represents computations on time evolv- with additional specialized operators such as a graph ing graphs as a stream of consistent and re- pattern matching operator (which abstracts a cost- silient graph snapshots and a small set of oper- based query engine) and a graph grouping operator ators that manipulate such streams. GraphTau (implemented as a composition of map, filter, group builds fault-tolerant graph snapshots as each and join transformations on Flink’s DataSet). GRA- small batch of new data arrives. It is also able DOOP also adopts the Cypher query language (typ- to periodically load data from graph databases ically found in graph databases like Neo4j) to ex- and reuses many operators from GraphX and press logic that is translated to the relational alge- Spark Streaming. For algorithms (based on bra that underlies Flink’s DataSet [148]. label propagation) that are not resilient to gra- In a similar way, Spark has its graph process- ph changes, GraphTau introduced an online ing library Graphx which was built over the sys- rectification model, where errors caused by un- tem’s batch processing API, like the case of Flink’s derlying graph modifications are corrected in Gelly and also suffering from the same previously online fashion with minimal state. Its API mentioned limitations. A higher-level API was de- frees programmers from having to implement signed to extend the functionalities of GraphX while graph snapshot generation, windowing opera- harnessing Spark’s DataFrame API. For this, the tors and differential computation mechanisms. GraphFrames open-source [283] (Apache License We did not find its source code available. 2.0) library was created [71]. A look at its im- plementation reveals that it has less high-level op- • Tink [165] is a library for distributed tempo- erations than Gelly. Effectively, without simulat- ral graph analytics. It is built on Flink (Java, ing some of Gelly’s API, equivalent programs in Scala) and focuses on interval graphs, where GraphX lend themselves to more conceptual ver- each edge has an associated starting time and bosity due to the lack of syntactic sugar. ending time. The author created different gra- We display in Figure 5 parallels between Flink, phs with information provided by Facebook Spark and the graph processing ecosystems built and Wikipedia in order to evaluate the frame- on top of them. Gelly’s equivalent in Spark is work. Tink defines a temporal property graph GraphX, implemented in Scala. Vertices and edges model. It is available online [166], although we are manipulated by using Spark’s Resilient Dis- did not find information pertaining licensing. tributed Datasets (RDDs), which can be viewed as a conceptual precursor to Flink’s DataSet. Spark To the best of our knowledge, both Flink and also offers the DataFrame API to enable tabular ma- Spark are currently the most widely-known distribu- nipulation of data. GraphFrames is another graph ted processing frameworks (we note GraphTau, al- processing library for Spark. While it has interop- though its code is not available, is built over Spark) erability and a certain overlap with the functional- based on dataflow programming. While the use of past, graph processing in Chaos achieves scala- bility with multiple machines in a cluster com- puting system. It is based on different func- tionalities: load balancing, randomized work stealing, sequential access to storage and an adaptation of X-Stream’s streaming partitions to enable parallel execution. Chaos is com- posed of a storage sub-system and a computa- tion sub-system. The former exists concretely as a storage engine in each machine. Its con- Figure 5: Contrast of the Flink and Spark dis- cern is that of providing edges, vertices and up- tributed dataflow ecosystems for graph pro- dates to the computation sub-system. Previ- cessing. ous work on X-Stream highlighted that the pri- mary resource bottleneck is the storage device bandwidth. In Chaos, the storage and compu- ity offered in GraphX, it integrates the tabular per- tation engines’ communication is designed in a spective supported by Spark’s DataFrame API and way that storage devices are busy all the time also supports performing traversal-like queries of – thus optimizing for the bandwidth bottle- the graph via SparkSQL. In this way, GraphFrames neck. It was released [239] under the Apache provides graph analytics capabilities in Spark much License 2.0. the same way GRADOOP does in Flink. The next two examples, X-Stream and Chaos are The following graph processing systems were grou- grouped together as they brought relevance to the ped together because each of the improvements they edge-centric (TLAE) model and employed it to ex- proposed are important concerns to be aware of in plore novel ways to balance network latencies and designing graph processing systems. use of SSDs to increase performance: • PowerLyra [54] is a graph computation engine • X-Stream [240]. A system that provided an al- written in C++ which adopts different parti- ternative view to the traditional vertex-centric tioning and computing strategies depending on approach. It is based on considering compu- vertex types. The authors note that most sys- tation from the perspective of edges instead of tems use a one-size-fits-all approach. They vertices and experiments optimized the use of note that Pregel and GraphLab focus in hid- storage I/O both locally and on the cloud. X- ing latency by evenly distributing vertices to -Stream is an open-source system written in machines, making resources locally accessible. C++ which introduced the concept of edge-cen- This may result in imbalanced computation tric graph processing via streaming partitions. and communication for vertices with higher X-Stream exposes an edge-centric scatter-ga- degrees (frequent in scale-free graphs). An- ther programming model that was motivated other provided design example is that of Po- by the lack of access locality when traversing werGraph and GraphX which focus on evenly edges, which makes it difficult to obtain good parallelizing the computation by partitioning performance. State is maintained in vertices. edges among machines, incurring communica- This tool uses the streaming partition, which tion costs on vertices, even those with low de- works well with RAM and secondary (SSD and grees. PowerLyra was released under the Apa- Magnetic Disk) storage types. It does not pro- che License 2.0 [55]. vide any way by which to iterate over the edges • Kineograph [57] is a system which combines or updates of a vertex. A sequential access to snapshots allowing full processing in the back- vertices leads to random access of edges which ground and explicit alternative/custom func- decreases performance. X-Stream is innova- tions that, besides assessing updates’ impact, tive in the sense that it enforces sequential pro- also apply them incrementally, propagating th- cessing of edges (edge-centric) in order to im- eir outcome across the graph. It is a distributed prove performance. It is available [241] under system to capture the relations in incoming the Apache License 2.0. data feeds, built to maintain timely updates • Chaos [238]. A system written in C++ which against a continuous flow of new data. Its ar- had its foundations on XStream. On top of chitecture uses ingest nodes to register graph the secondary storage studies performed in the update operations as identifiable transactions, which are then distributed to graph nodes. No- was able to execute a custom (Pixie Random des of the latter type form a distributed in- Walk) algorithm on an object graph of 3 bil- memory key/value store. Kineograph performs lion vertices and 17 billion edges. On a single computation on static snapshots, simplifying server, they were able to serve around 1200 algorithm design. We did not find its source recommendation requests per seconds with 60 code online. millisecond latency. The authors note that the deployment of Pixie benefited from large • Tornado [254] is a system for real-time itera- RAM machines, using a cluster of Amazon tive analysis over evolving data. It was imple- AWS r3.8xlarge machines with 244GB RAM. mented over Apache Storm and provides an They fitted the pruned Pinterest graph (3 bil- asynchronous bounded iteration model, offer- lion vertices, 17 billion edges) in around 120GB ing fine-grained updates while ensuring cor- of RAM, in a setup that yielded the following rectness. It is based on the observations that: advantages: random walk not forced to cross 1) loops starting from good enough guesses machines, which increases performance; mul- usually converge quickly; 2) for many itera- tiple walks can be executed on the graph in tive methods, the running time is closely re- parallel; the system can be parallelized and lated to the approximation error. From this, scaled by adding more machines to the clus- an execution model was built where a main ter. This system is a relevant case study (of loop continuously gathers incoming data and applying graph theory to recommendation sys- instantly approximates the results. Whenever tems at scale) as a scalable system for process- a result request is received, the model creates a ing on large graphs a biased random walk al- branch loop from the main loop. This branch gorithm (with user-specific preferences) while loop uses the most recent approximations as a using graph pruning techniques to disregard guess for the algorithm. We did not find its large boards that are too diverse and diffuse source code online. the random walk (the non-pruned graph has 7 • KickStarter [289] is a system that debuted billion vertices and 100 bilion edges). We did a runtime technique for trimming approxima- not find the source code available online. tion values for subsets of vertices impacted by edge deletions. The removal of edges may in- • FlowGraph [52] is a system that proposes a validate the convergence of approximate val- syntax for a language to detect temporal pat- ues pertaining monotonic algorithms. Kick- terns in large-scale graphs and introduces a Starter deals with this by identifying values novel data structure to efficiently store results impacted by edge deletions and adapting the of graph computations. This system is a uni- network impacts before the following compu- fication of graph data with stream process- tation, achieving good results on real-world ing considering the changes of the graph as a use-cases. Despite this, by focusing on mono- stream to be processed and offering an API to tonic graph algorithms, its scope is narrowed satisfy temporal patterns. We did not find its to selection-based algorithms. For this class, source code available. updating a vertex value implies choosing a nei- ghbour under some criteria. KickStarter is • GPS [246] is an open-source (BSD License) scal- now known as GraphBolt, a recent work [186, able graph processing system written in Ja- 185] licensed under the MIT License [186] whi- va and allowing fault-tolerant and easy-to-pro- ch offers a generalized incremental program- gram algorithm execution on very large graphs. ming model enabling development of incremen- It adopts Pregel’s vertex-centric API and ex- tal versions of complex aggregations. Both tends it with: features to make global com- were written in C++. putations easier to express and more efficient; dynamic repartitioning scheme to reassign ver- • Pixie [85] is a graph-based scalable real-time tices to different workers during computation recommendation system used at Pinterest. Us- based on messaging patterns; distribution of ing a set of user-specific pins (in Pinterest, high-degree vertex adjacency lists across all users have boards in which they store pins, computer nodes to improve performance (some- where each pin is a combination of image and thing that PowerGraph and PowerLyra later text), Pixie chooses in real-time the pins that adopted). It was designed to run on a cluster are most related to the query, out of billions of machines such as Amazon EC2, over which of candidates. With this system, Pinterest the authors tested their work. GPS’s initial ver- sion was run on up to 100 Amazon EC2 large cused on vertex-cut as it has better partition- instances and on graphs of up to 250 million ing properties for real-world graphs that have vertices and 10 billion edges. It is open-source power-law degree distributions. GrapH has two and available under the BSD License [245]. partitioning techniques, H-load which is used for the initial partitioning of the graph so that • GoFFish [262] is a sub-graph centric program- each cluster worker can load it into local mem- ming abstraction and framework co-designed ory, and H-adapt, a distributed edge migration with a distributed persistent graph storage for algorithm to address the dynamic heterogeneity- large scale graph analytics on commodity clus- aware partitioning problem. In evaluation, Gra- ters, aiming to combine the scalability of the pH outperformed PowerGraph’s vertex-cut par- vertex-centric (TLAV) approach with flexibil- titioning algorithm with respect to communi- ity of shared-memory sub-graph computation cation costs. While its source code is avail- (TLAG). It is written in Java. GoFFish states able [190], we found no information on licens- that two sub-graphs many not share the same ing. vertex, but they can have remote edges that connect their vertices, provided that the sub- • Julienne [78] is built over Ligra (C++) and graphs are on different partitions. If two sub- provides an interface to maintain a collection graphs in the same partition share an edge, by of buckets under vertex insertions and bucket definition they are merged into a single-sub- deletions. They evaluated under bucketing al- graph. It was evaluated with a cluster of 12 gorithms such as weighted breadth-first search, nodes each with 8-core Intel Xeon CPUs, 16 k-core and approximate set cover. The au- GB RAM and 1 TB SATA HDD. The authors thors describe as bucketing-based algorithms compare the execution of GoFFish (one worker those that maintain vertices in a set of un- per node) with Giraph (default two workers ordered buckets - and in each round, the al- per node), achieving faster execution times for gorithm extracts the vertices contained in the algorithms such as PageRank, connected com- lowest (or highest) bucket to perform compu- ponents and single-source shortest-paths. Its tation on them. Then, it can update the buck- source code is available though we did not find ets containing the extracted vertices or their any information pertaining licensing. While neighbours. For example, weighted breadth- its source code is available [263], we did not first search processes vertices level by level, find information regarding licensing. where level i contains all vertices at distance i • FBSGraph [307] presents a forward and back- from the source vertex. The system was tested ward sweeping execution method to acceler- in a multi-core machine with 72 cores (4 CPUs ate state propagation for asynchronous graph at 2.4GHz) and 1TB of main memory, achiev- processing. In asynchronous graph processing, ing performance improvements on several data each vertex maintains a state which can be sets when compared to systems such as Ga- asynchronously updated in an iterative fash- lois, base Ligra and Galois. We did not ion. The method presented in FBSGraph relies find its source code available. on the observation that state can be propa- gated faster by processing vertices sequentially • GraphD [301] is an out-of-core system inspired along the graph path in each round. They by Pregel and targeting efficient big graph achieve greater execution speed when analyz- processing using a small cluster of commod- ing several graph algorithms across a set of ity machines connected by Gigabit Ethernet, datasets, comparing against systems such as contrasting with other out-of-core works that PowerGraph and GraphLab. We did not find focus on specialized hardware. The authors its source available. focus on a setting that sees vertex-centric pro- grams being data-intensive, as the CPU cost • GrapH [191] is a graph processing system writ- of computing a message is small when com- ten in Java that uses vertex-cut graph parti- pared to the network transmission cost. Gra- tioning that takes into consideration the di- phD masks disk I/O overhead with message versity of vertex traffic and the heterogeneous transmission though parallelism of computa- costs of the network. It relies on a strategy tion and communication. It eliminates the need of adaptive edge migration to reduce the fre- for (expensive) external-memory join or group- quency of communication across expensive net- by operations, which are required in other sys- work links. For this work, the authors fo- tems such as Chaos. It was evaluated on PageR- ank, single-source shortest-paths and connected base computation made GraphIn faster than components. GraphD was evaluated against systems using fixed strategies. We did not find distributed out-of-core systems Pregelix, Ha- its source code available. Loop and Chaos, against single-machine sys- tems GraphChi and X-Stream and representa- The following works focus on specific techniques tive in-memory systems Pregel and Giraph, such as using specific hardware such as SSDs or achieving better performance in some scenar- GPUs. We first list frameworks and systems that ios. We did not find its source available. were proposed in the last years to use the single- instruction multiple-data (SIMD) capabilities of G- • TurboGraph++ [159] is a graph analytics sys- PUs for graph processing: tem that exploits external memory for scale- up without compromising efficiency. It intro- • MapGraph [102] is a high-performance parallel duced an abstraction called nested windowed graph programming framework, able to achieve streaming to achieve scalability and increase up to 3 billion traversed edges per second us- efficiency in processing neighbourhood-centric ing a GPU. It represents the graph with a analytics (in which the total size of neighbour- compressed sparse row (CSR) data structure hoods around vertices can exceed the available and chooses different scheduling strategies de- memory budget). This streaming model re- pending on the size of the frontier (the set of gards a sequence of vertex values and an adja- vertices that are active in a given iteration). cency list stream. The goal is to efficiently It encapsulates the complexity of the GPU ar- support the k-walk neighbourhood query (a chitecture while enabling dynamic runtime de- class of graph queries defined by the authors, cisions among several optimization strategies. where walks are enumerated and then compu- Users need only to write sequential C++ code to tation is done for each one) with fixed size use the framework. The early MapGraph work memory. In the model, during user compu- was first available as an open-source project [103] tation, they define an update stream as the licensed under the Apache License 2.0, but it sequence of updates generated to the ending has been integrated in the former line of prod- vertex of each walk, with each update repre- ucts of Blazegraph, also available online[275]. sented as a pair of target vertex ID and update • CuSha [157] is a CUDA-based graph processing value. TurboGraph++ has the goal of balanc- framework written in C++ which was motivated ing the workloads across machines, which re- by the negative impact that irregular mem- quires balancing the number of edges and the ory accesses have on the compressed sparse number of high-degree and low-degree vertices row graph (CSR) representation. CuSha over- among machines. It also focuses on balanc- comes this by: 1) organizing the graph into au- ing the number of vertices on each machine so tonomous sets of ordered edges called shards (a that each one requires the same memory bud- representation they call G-Shards) unto which get. We did not find its source code available GPU hardware resources are mapped for fully online. coalesced memory accesses; 2) accounting for input graph properties such as sparsity (the • GraphIn [252] is a dynamic graph analytics sparser the graph, the smaller the computa- framework proposed to handle the scale and tion windows) to avoid GPU under-utilization evolution of real-world graphs. It aimed to im- (Concatenated Windows, or CW ). This frame- prove over approaches to processing dynamic work allows users to define vertex-centric algo- graphs which repeatedly run static graph ana- rithms to process large graphs on GPU. It is lytics on stored snapshots. GraphIn proposes open-source [158] and available under the MIT an adaptation of gather-apply-scatter (GAS) License. called I-GAS which enables the implementa- tion of incremental graph processing algorithms • Gunrock [291, 293] is an open-source [292] (A- across multiple CPU cores. It also introduces pache License 2.0) CUDA library for graph an optimization heuristic to choose between processing targeting the GPU and written in static or dynamic execution based on built-in C. It implements a data-centric abstraction fo- and user-defined graph properties. Native and cused on operations on a vertex or edge fron- benchmarking code were implemented in C++ tier. For different graph algorithms, it achieved and for experimental evaluation it was com- at least an order of magnitude speedup over pared to GraphMat and STINGER. The heuristic- PowerGraph and better performance than any other high-level GPU graph library at the time. kernel function executions, minimizing PCIe Its operations are bulk-synchronous and ma- data transfer overhead. It is open-source [256] nipulate a frontier, which is a subset of the and licensed under the GNU General Public edges or vertices within the graph that is rel- License 2.0. evant at a given moment in the computation. Gunrock couples high-performance GPU com- • Aspen [79] is a graph-streaming extension of puting primitives and optimization strategies the Ligra interface, supporting graph updates. with a high-level programming model to quickly To support this, the authors developed and develop new graph primitives. It was evalu- presented the C-tree data structure which achie- ated using breadth-first search, depth-first sear- ves good cache locality, lowers space use and ch, single-source shortest paths, connected com- has operations which are efficient from a theo- ponents and PageRank. retical perspective. It applies a chunking sche- me over the tree, storing multiple elements in a • Lux [145] is a distributed multi-GPU system tree-node. The scheme takes the ordered set of written in C++ for fast graph processing by ex- elements that are represented. More relevant ploiting aggregate memory bandwidth of mul- elements are stored in tree nodes, while the tiple GPUs and the locality of the memory hi- remaining ones are associated in tails of the erarchy of multi-GPU clusters. It proposes a tree nodes. It employs compression and sup- dynamic graph repartitioning strategy to en- ports parallelism. The authors evaluate it with able well-balanced distribution of workload with the largest publicly-available graph, which has minimal overhead (improving performance by more than two hundred billion edges on a multi- up to 50%), as well as a performance model core server with 1 TB memory. Source code is providing insight on how to choose the optimal available online [80] albeit no license informa- number of nodes and GPUs to optimize perfor- tion was provided. mance. Lux is aimed at graph programs that • Gluon [68] was introduced as a new approach can be written with iterative computations. to create distributed-memory graph analytics Vertex properties are read-only in each itera- systems able to use heterogeneity in partition- tion, with updates becoming visible at the end ing policies, processor types (GPU and CPU) of an iteration. It offers two execution models: and programming models. To use Gluon, pro- pull which optimizes run-time performance of grammers implement applications in shared- GPUs (enables optimizations like caching and memory programming systems of their choice locally aggregating updates in GPU shared me- and then interface the applications with Glu- mory); and push, which optimizes algorithmic on to enable execution on heterogeneous clus- efficiency (maintains a frontier queue and only ters. Gluon optimizes communication by tak- performs computation over the out-edges of ing advantage of temporal and structural in- vertices in the frontier). Its source code is variants of graph partitioning policies. It runs available [146] under Apache License 2.0. on shared-memory NUMA platforms and NVI- DIA GPUs. Its programming model offers a • Frog [255] is a light-weight asynchronous pro- small number of programming patterns imple- cessing framework written in C. The authors mented in C++, its library offers concurrent note that common coloring algorithms may suf- data structures, schedulers and memory allo- fer from low parallelism due to a large num- cators and the runtime executes programs in ber of colors being needed to process large parallel, using parallelization strategies as op- graphs with billions of vertices. Frog sepa- timistic and round-based execution. Gluon is rates vertex processing based on color distribu- available [70] under the 3-Clause BSD Licen- tion. They propose an efficient hybrid graph se. coloring algorithm, relying on a relaxed pre- partition method to solve vertex classification • Hornet [47] is a data structure for efficient with a lower number of colors, without forcing computation of dynamic sparse graphs and ma- all adjacent vertices to be assigned different trices using GPUs. It is platform-independent colors. The execution engine of Frog scans the and implements its own memory allocation op- graph by color, and all vertices under the same eration instead of standard function calls. The color are updated in parallel in the GPU. For implementation uses an internal data manager large graphs, when processing each partition, which makes use of block arrays to store adja- the data transfers are overlapped with GPU cency lists, a bit tree for finding and reclaiming empty memory blocks and B+ trees to man- with I/O, a concept similar to X-Stream and age them. It was evaluated using an NVIDIA Chaos, and edges lists are only accessed if re- Tesla GPU and experiments targeted the up- quested by applications from SSDs. Flash- date rates it supports, algorithms such as brea- Graph has a vertex-centric (TLAV) interface, dth-first search (BFS) and sparse matrix-vector its designed to reduce CPU overhead and in- multiplication. Hornet is available [48] under crease throughput by conservatively merging the 3-Clause BSD License. I/O requests, and the authors demonstrate that FlashGraph in semi-external memory executes • faimGraph [297] introduced a fully-dynamic many algorithms with a performance of up to graph data structure performing autonomous 80% of the in-memory implementation and It memory management on the GPU. It enables also outperformed PowerGraph. It is open- complete reuse of memory and reduces mem- source [310] under the Apache License 2.0. ory requirements and fragmentation. The im- plementation has a vertex-centric update scheme • GraphSSD [189] is a semantic-aware SSD frame- that allows for edge updating in a lock-free work and full system solution to store, access way. It reuses free vertex indices to achieve ef- and execute graph analytics. Instead of con- ficient vertex insertion and deletion, and does sidering storage as a set of blocks, it accounts not require restarting as a result of a large for graph structure while choosing graph lay- number of edge updates. faimGraph was bench- out, access and update mechanisms. Graph- marked against Hornet on an NVIDIA Geforce SSD innovates by considering a vertex-to-page GTX Titan Xp GPU using algorithms such as mapping scheme and uses advanced knowledge PageRank and triangle counting. Source code of flash properties to reduce page accesses. It is available online [69] without a specified li- offers a simple API to ease development of cense. applications accessing graphs as native data and its evaluation showcased average perfor- • GraphCage [56] is a cache-centric optimization mance gains for basic graph data fetch func- framework to enable highly efficient graph pro- tions on breadth-first search, connected com- cessing on GPUs. It was motivated by the ran- ponents, random-walk, maximal independent dom memory accesses which are generated by set and PageRank. We did not find its source sparse graph data structures, which increase available. memory access latency. The authors note that conventional cache-blocking suffers from repea- In Table 1 we summarize distinguishing features ted accesses when processing large graphs on and licenses for the graph processing systems de- GPUs, and propose a throughput-oriented ca- tailed in this section. The last reference in front of che blocking scheme (TOCAB). GraphCage ap- every system name is its open-source code reposi- plies the scheme to both push and pull di- tory, when available. The second group from the rections and coordinates with load balancing top (PBGL, CombBLAS and HavoqGT) contains sys- strategies by considering sparsity of sub-graphs. tems which use multiple machines for computation This technique is applied to traversal-based al- but not in the typical cluster scenario. Instead, gorithms by considering the benefit and over- they are characterized by using specific machines head in different iterations with working sets for high-performance computing. of different sizes. In its evaluation, GraphCa- ge achieved in average lower execution times 14. GRAPH DATABASES for one PageRank iteration compared to both Here we list and describe different graph databases Gunrock and CuSha. We did not find its source which we group together according to their type of code available. supported graph model (e.g., property graph model, For more information on GPU use cases for graph RDF, hybrid). Upon developing this list, we ob- processing approaches, we point the readers to [257]. serve regarding graph query languages that they either implement their own custom language (ne- • FlashGraph [309] is a graph processing engine glecting interoperability even though there may be implemented in C++ over a user-space SSD file automatic converters developed by third parties), or system designed for high IOS and very high they use those that have been improved, standard- levels of parallelism. Vertex state is stored ized and adopted by many projects (e.g., Cypher, in memory while edge lists are on SSDs. La- Gremlin, SPARQL). The list includes both open-source tency is hidden by overlapping computation and commercial products, as well as database sys- System Languages License Notes GPU Cluster Multi-core GraphLab [174, 175] C++ AL 2.0 N/A GRACE [290] C++ Unavailable N/A Ligra [259, 260] C++ MIT N/A Ringo [225, 226] C++, Python BSD N/A Polymer [305, 306] C++ AL 2.0 N/A GraphMat [272, 273] C++ Custom N/A Mosaic [181, 180] C++ MIT Fast storage PBGL [112, 113] C++ Custom Hardware CombBLAS [45, 44] C++ Custom Hardware HavoqGT [222, 223] C++ GNU LGPL 2.1 Hardware Apache Giraph [59, 97] Java AL 2.0 N/A Naiad [203, 232] C# AL 2.0 N/A Apache Flink [51, 99] Java, Python, Scala AL 2.0 N/A Apache Spark [303, 100] Java, Python, Scala AL 2.0 N/A GraphTau [141] Java, Scala Unavailable N/A Tink [165, 166] Java, Scala AL 2.0 N/A X-Stream [240, 241] C++ AL 2.0 N/A Chaos [238, 239] C++ AL 2.0 N/A PowerLyra [54, 55] C++ AL 2.0 N/A Kineograph [57, 186] Unknown Unavailable N/A Tornado [254] Unknown Unavailable N/A KickStarter [289] C++ MIT N/A Pixie [85] Unknown Unavailable N/A FlowGraph [52] Unknown Unavailable N/A GPS [246, 245] Java BSD N/A GoFFish [262, 263] Java Unknown Copyright FBSGraph [307] Unknown Unavailable N/A GrapH [191, 190] Java Unknown Copyright Julienne [78] C++ Unavailable N/A GraphD [301] Unknown Unavailable N/A TurboGraph++ [159] Unknown Unavailable N/A GraphIn [252] C++ Unavailable N/A MapGraph [102, 103] C++ AL 2.0 Discontinued CuSha [157, 158] C++ MIT N/A Gunrock [291, 293, 292] C AL 2.0 N/A Lux [145, 146] C++ AL 2.0 N/A Frog [255, 256] C GPL 2.0 N/A Gluon [68, 70] C++ 3C BSD N/A GraphCage [56] Unknown Unavailable N/A FlashGraph [309, 310] C++ AL 2.0 SSDs GraphSSD [189] Unknown Unavailable SSDs

Table 1: Summary of graph system distinctive features. Circle on the Multi-core, GPU and Cluster columns indicate that option is supported. Languages lists the programming languages the systems were written in. License lists the licenses of the open-source project or of the free edition of a commercial product: AL 2.0 is Apache License 2.0, CC 1.0 is Commons Clause 1.0, (GPL) v3 is GNU General Public License (GPL) v3. Notes covers additional information, with Copyright meaning that it may be illegal to reuse the source code. tems implemented to explore novel techniques as • Is it ACID compliant (ACID)? part of research activities. We showcase these dif- • Does the specific graph database ecosystem of- ferent graph database solutions in Table 2, in which fer analytics capabilities (e.g., first-party, third- we present relevant properties for assessing them. party, none) to complement the OLTP focus of We consider these properties to be outstanding fea- the graph database itself (Visual)? tures that are important for developers and users to assess their suitability for their use-cases. They • Does it offer scale-out/horizontal scalability (e.g., provide information on guarantees of graph data Neo4j is very well-known but it does not sup- storage, parallelism of computation and supported port sharding, only replication when using mul- graph sizes, visualization capabilities and the abil- tiple machines) to improve performance and ity to use them freely (regarding both open-source support bigger graphs (Scale)? as well as free versions of commercial products): • What graph models does it support, is it for example RDF, property graph or others (Mod- Apache License 2.0 and supporting Java. It els)? integrates with the Cassandra [163] distributed database (over which it provides graph data • What languages can be used to program func- models) and supports TinkerPop (property gra- tionality that uses the database product, for ph with Gremlin). It is complemented by the example Go, Java, .NET, Python or others (Lan- DataStax Studio, which allows for interactive guages)? querying and visualization of graph data sim- • What graph query languages (e.g., Cypher, Grem- ilarly to Neo4j and SAP Hana Graph. lin, SPARQL, custom) does it support (GQLs)? • Dgraph [76, 77] was written in Go and it is • What license (or types of licenses) is it sub- a distributed graph database offering horizon- jected to - for example, is it a common license tal scaling and ACID properties. It is built type or other (License)? to reduce disk seeks and minimize network us- age footprint in cluster scenarios. Dgraph is li- • For database products that have both free and censed under two licenses: the Apache Licen- commercial/proprietary editions, what are the se 2.0 and a Dgraph Community License. It features missing from the free versions - the automatically moves data to rebalance clus- focus here is not on marketing-speech such as ter shards. It uses a simplified version of the five-nines availability, but features that are easy GraphQL query language. Support for Gremlin to implement but are clearly missing from the or Cypher has been mentioned for the future free version to stimulate purchases (Limits)? but will depend on community efforts. Dgraph For further information on the evolution of graph has a scalability advantage over Neo4j as the databases, we point the reader to [13, 230], which latter may have multiple servers but they are covers, among other aspects, data models and query merely replicas, while the former can grow hor- languages. The following graph databases we list izontally (vertical scaling is expensive). There are focused on the property graph model. Their in- is a proprietary enterprise version (conditions formation describes the use of this model; we found specified under a custom Dgraph Community no graph query languages for RDF in their features, License) with advanced features for backups even if these databases are described as multi-model: and encryption. Official clients include Java, JavaScript and Python. To the best of our • Alibaba Graph Database (GD) [60] is a cloud- knowledge Dgraph does not offer visualization oriented (thus supporting horizontal scalabil- and global analytics functionalities. ity) graph database service supporting ACID • Graphflow [155, 195, 194] was released as a transactions and the TinkerPop stack. It sup- prototype active graph database. It is an in- ports the property graph model, the Gremlin memory graph store supporting the property query language and there are programming in- graph model and supports one-time as well as terfaces for Go, Java, .NET, Node.js and Py- continuous sub-graph queries. Graphflow sup- thon. We did not find references to visual- ports this using a one-time query processor ization capabilities (visual feedback to query called Generic Join and a Delta Join which construction and execution). enables the continuous sub-graph queries. It • ChronoGraph [121] is a TinkerPop-compliant extends the openCypher language with trig- (offering Gremlin to query the data in prop- gers to perform actions upon certain condi- erty graph model) graph database supporting tions. We did not find information regarding ACID transactions, system-time content ver- its direct use beyond academic purposes nor sioning and analysis. It is implemented as a about supporting ACID transactions. Its code key-value store enhanced with temporal infor- is available online [156] under the Apache Li- mation, using a B-tree data structure. The cense 2.0. project is available online [122] under the aGPL • JanusGraph [143] is an open-source project li- v3 (open-source and academic purposes) and censed [17] under the .A commercial licenses are available on demand. Apache License 2.0 database optimized for storing (in adjacency We did not find visualization functionalities list format) and querying large graphs with accompanying it. (billions of) edges and vertices distributed a- • DataStax Enterprise Graph (DSE) [67] is a cross a multi-machine cluster with ACID trans- proprietary fork of Titan, licensed under the actions. JanusGraph, which debuted in 2017, is based on the source Java code base of the the query language Cypher and data is stored Titan graph database project and is supported on disk as fixed-size records in linked lists. by the likes of Google, IBM and the Linux Neo4j has a library offering many different graph Foundation, to name a few. Like Titan, it algorithms. As far as we know, Neo4j’s scale- supports Cassandra, HBase and BerkeleyDB. out capabilities are only true for read oper- It was designed with a focus on scalability and ations. All writes are directed to the Neo4j it is in fact a transactional database aimed cluster master, an architecture which has its at handling many concurrent users, complex limitations. Among other uses, Neo4j has also traversals and analytic queries. JanusGraph been employed for building applications using can integrate platforms such as Spark, Gi- the GRANDstack framework [179]. Neo4j also raph and Hadoop. It also natively integrates has an interactive graph explorer to query and with the TinkerPop graph stack, supporting update specific elements of the graph. Gremlin applications, the query language and its graph server, with graphs in the property • RedisGraph [49] is a property graph database graph model. Due to supporting TinkerPop, which uses sparse matrices to represent a gra- one may use one of its drivers to use Grem- ph’s adjacency matrix and uses linear algebra lin from Elixir, Go, Java, .NET, PHP, Python, for graph queries. It uses custom memory- Ruby and Scala. It supports global analytics efficient data structures stored in RAM, hav- using Spark integration as well. ing on-disk persistence and tabular result sets. Queries may be written in a subset of Cy- • Nebula Graph is an open-source graph database pher and are internally translated into linear (available online [288]) licensed under Apache algebra expressions. It has a custom license License 2.0, provides a custom Nebula Gra- and client libraries for RedisGraph have been ph Query Language (nGQL) with syntax close developed in Elixir, Go, Java, JavaScript, to SQL and Cypher support is planned. It sup- PHP, Python, Ruby and Rust, complementing ports the property graph model, ACID trans- existing accesses that Redis already supports. actions and is implemented with a separation As far as we know, it only works in single- of storage and computation, being able to scale server mode (bounded by the machine’s RAM) horizontally. It supports multiple storage en- and it does not support ACID properties. It gines like HBase [106] (implementing the graph has a Community Edition under a custom li- logic over these key-value stores) and Rocks- cense. source code available online [177] under DB[90] and has clients in Go, Java and Python. a custom license. It also has the complementing Nebula Graph Studio for interactive visual querying and an- • SAP Hana Graph [242, 134, 248] is a column- alytics. oriented, in-memory relational database man- agement system. It performs different type of • Neo4j [294] is a graph database with multiple data analysis, among which graph data pro- editions [209]: a community edition licensed cessing with the property graph model and under the free GNU General Public License ACID transactions. This graph functionality (GPL) v3, a commercial one and also an ad- includes interpretation of Cypher and a visual vanced edition licensed under AGPLv3. It sup- graph manipulation tool. Its graph processing ports different programming languages C/C++, capabilities have served use cases like fraud de- Clojure, Go, Haskell, Java, JavaScript, .N- tection and route planning. We did not find ET, Perl, PHP, Python, R and Ruby. Neo4j is source code available online. optimized for highly-connected data. It relies on methods of data access for graphs without • Sparksee [188, 267] (formerly DEX) is a prop- considering data locality. Neo4j’s graph pro- erty graph database offering ACID transac- cessing consists of mostly random data access. tions and representing the graph using bitmap For large graphs which require out-of-memory data structures with high compression rates processing, the major performance bottleneck (with each bitmap partitioned into chunks that becomes the random access to secondary stor- fit disk pages). The graphs in Sparksee are la- age. The authors created a system which sup- beled multigraphs and it has multiple licenses ports ACID transactions, high availability, with depending on the purpose, with free licenses operations that modify data occurring within for evaluation, research and development. It transactions to guarantee consistency. It uses offers APIs in C++, .NET, Java, Objective-C, Python and mobile devices. We found no ca- Public License v1 (EPL v1) which supports pabilities for data visualization in Sparksee, several programming languages (C#, C, Common though it is able to export data to formats Lisp, Clojure, Java, Perl, Python, Scala) supported by third-party software. that was purpose-built for RDF (triple-store). It supports an array of mechanisms to access • TigerGraph [75, 280] is a commercial graph the information it stores, namely reasoning wi- database (formerly GraphSQL) implemented in th ontology (RDFS++ Reasoning), materialized C++ and comes in three versions: developer reasoning (generating new triples based on in- edition (supporting only single-machine, no dis- ference rules - OWL2 RL Materialized Rea- tribution and is only for non-production, re- soner, SPARQL queries, Prolog and also low- search or educational purposes), cloud edition level APIs. (as a managed service) and enterprise edition (allowing for horizontal scalability - distributed • BlazeGraph [275] (formerly Bigdata) is an RDF graphs). It supports ACID consistency, access database able to support up to billions of edges through a REST API, has a custom SQL-like in a single machine and available under the G- query language (GSQL) and features a graphi- NU General Public License v2.0 supporting cal user interface named GraphStudio to per- SPARQL and the TinkerPop Blueprints AP- form interactive graph data analytics. The I. It has .NET and Python clients. One of TigerGraph model was designed to support its associated internal projects is blazegraph- graph vertices, edges and their attributes to gremlin, which allows the storage of prop- support an engine that performs massively- erty graphs internally in RDF format, which can parallel processing to compute queries and an- then be queried with SPARQL. It essentially has alytics. Each vertex and edge acts as both an alternative approach to RDF reification, giv- a unit of storage and computation, integrating ing labeled property graph capabilities to RDF and extending both TLAV and TLEV paradig- graphs, with the ability to query the graphs in ms. It supports the property graph model plus Gremlin as well. extensions to enable the massively-parallel pro- cessing. We did not find available source code. • BrightstarDB is an open-source [35] multi- • Weaver [83] is an open-source [82] graph data- threaded multi-platform (including mobile) .N- base (custom permissive license) for efficient, ET RDF store, supporting SPARQL and binding transactional graph analytics. It introduced of RDF resources to .NET dynamic objects (it the concept of refinable timestamps. It is a has tools to use .NET interfaces and generate mechanism to obtain a rough ordering of dis- concrete classes to persist their data in Bri- tributed operations if that is sufficient, but ghtstarDB). It is licensed under the MIT Li- also fine-grained orderings when they become cense and there is also an Enterprise version. necessary. It is capable of distributing a graph BrightstarDB supports single-threaded writes across multiple shards while supporting con- and multi-threaded reads, with ACID transac- currency. Refinable timestamps allow for the tions. It does not support horizontal scaling. existence of a multi-version graph: write op- erations use their timestamps as a mark for • Cray Graph Engine (CGE) [233] is an RDF tri- vertices and edges. This allows for the ex- ple database offering the SPARQL query lan- istence of consistent versions of the graph so guage. As a commercial product, it was de- that long-running analysis queries can operate signed while considering different architectures on a consistent version of the graph, as well of proprietary systems (containing the Cray as historical queries. Weaver is written in C++, Aries interconnect [6]) of the company be- offering binding options for Python. We did hind GCE. It offers APIs for Java, Python and not find any support for popular graph query Spark, having a number of pre-built graph al- languages. gorithms. It is not open-source and its back- end relies on internal queries written in C++ to The following graph databases we list are focused work with a global address space using mul- on the data model and variations (including RDF tiple processes on multiple compute nodes to support for the property graph model by represent- share data and synchronize operations. This ing them as [124]): RDF product being both proprietary and reliant on • AllegroGraph [136] is a proprietary commer- custom hardware has the consequence of not cial graph database with clients under Eclipse being so widespread. However, its results and special-purpose architecture make it a compet- and an enterprise edition which supports un- itive platform which harnessed innovation in limited cluster size, both supporting commer- design as a graph database. cial and non-commercial use. It supports third- party visualization tools. We did not find de- • Ontotext GraphDB [213] is a graph database tails on its internal data structures nor source focused on RDF data and offering ACID trans- code online. action properties. It comes in three editions: free which is used for smaller projects and for • ArangoDB is an open-source [278] multi-threa- testing and is only able to execute at most ded database with support for graph storage two concurrent queries; standard which can (as well as key/value pairs and documents) load and query statements at scale; enterprise that is available in both a proprietary license edition which offers horizontal scalability and and the Apache License 2.0. It is written in other features. It supports SPARQL and offers JavaScript from the browser to the back-end a Java programming API. We did not find its and all data is stored as JSON documents. A- source code available. rangoDB provides a storage engine for mostly in-memory operations and an alternative stor- The following support at least both the property age engine based on RocksDB, enabling datasets graph model and/or RDF explicitly plus other data that are much bigger than RAM. It guarantees models (e.g., at least any of: document collections, ACID transactions for multi-document and m- relational model, object model): ulti-collection queries in a single instance and for single-document operations in cluster mode. • Amazon Neptune [25] is a managed proprietary Replication and sharding are offered, allow- service (freeing the user from having to focus ing users to set up the database in a master- on management tasks, provisioning, patching, slave configuration or to spread bigger datasets etc.) that is ACID-compliant and focused on across multiple servers. It exposes a Pregel- highly-connected datasets and among its use like API to express graph algorithms (implying cases are recommendation engines, fraud de- access to the stored data in the database), has tection and drug discovery, among others. Its a custom SQL-like query language called AQL implementation language and internal graph (ArangoDB Query Language) and includes a representation have not been disclosed and it built-in graph explorer. supports both the property graph and RDF mod- els, offering Gremlin and SPARQL to query them. • IBM System G [135] more than a proprietary As a full-fledged commercial product, it also graph database, is a complete suite of func- has many features related to backup, replicas, tionalities, able to support the property graph security and management tasks, using product (with Gremlin) as well as RDF (though we did features such as Amazon’s S3, EC2 and Cloud- not find comments on RDF-specific query lan- Watch to offer scalability. Usage samples avail- guages). It is comprised of proprietary com- able online [8]. ponents as well as open-source and comes with visual query capabilities, providing visual feed- • AnzoGraph DB [50] (previously SPARQLVerse) back into query building and result analysis is a proprietary database built to enable RDF to ease the debugging process. ACID trans- with SPARQL and the property graph with Cy- actions are supported and the graph is repre- pher queries to analyze big graphs (trillions of sented in its native store with a data structure relationships) and it has Java and C++ APIs similar to compressed sparse vectors, using off- to create functions, aggregates and services. It sets to delimit, for each graph element, the supports ACID transactions and also supports latest and earliest temporal information of the an RFDF+ inference engine following W3C stan- element. dards and uses compressed in-memory and on disk storage of data. This database is described • OrientDB [277] is a distributed (property mo- as beyond a transaction-oriented database and del) graph database that supports TinkerPop as a Graph Online Analytics Processing- and functions both as a graph database and (GOLAP) database, enabling interactive view, NoSQL document database as well, with a Com- analysis and update of graph data, in a way munity Edition licensed [217] under Apache similar to the interactive capabilities of the License 2.0 and a commercial edition. There Neo4j database. It comes as a single-machine are drivers supporting OrientDB at least in (and memory usage limitations) free edition the following languages: Clojure, Go, Java, JavaScript, .NET, Node.js, PHP, Python, R, deployments, software upgrades and other op- Ruby and Scala. It supports sharding (hori- erations. An online repository of source code zontal scaling), has ACID support and offers information is available [65]. an adapted SQL for querying. • FaunaDB [93] is a distributed database plat- • Oracle Spatial and Graph (OSG) [216] is a form targeting the modern cloud and container- long-standing commercial product which has centric environments. It has a custom Fau- spatial and graph capabilities, among which na Query Language (FQL) which operates on the property graph model using PGQL and the schema types such as documents, collections, RDF model with SPARQL. It also supports a indices, sets and databases. This language can feature-rich studio with notebook interpreters, be accessed through drivers in languages such shell user-interface and graph visualization. The- as Android, C#, Go, Java, JavaScript, Py- re are different Java APIs, one for the Oracle thon, Ruby, Scala and Swift. FaunaDB sup- Spatial and Graph Property Graph, another ports concurrency, ACID transactions and of- for TinkerPop Blueprints and Database Pro- fers a RESTful HTTP API. perty Graph. • Google Cayley is an open-source [109] data- • Stardog [269] is a proprietary (with a limited- base behind Google’s Knowledge Graph, hav- time free trial version and a paid Enterprise li- ing been tested at least since 2014 (and it is cense) knowledge graph database with a graph the spiritual successor to graphd [193]). It is a model based on RDF and extensions to support community-driven database written in Go, in- the property graph model. It is horizontally- cluding a REPL, a RESTful API, a query ed- scalable, supports ACID operations, GraphQL, itor and visualizer. It supports Gizmo (query Gremlin and SPARQL for querying and intro- language inspired by Gremlin) and GraphQL. spection and may be programmed in Clojure, Cayley supports multiple storage back-end such Groovy, Java, JavaScript, .NET and Spring. as LevelDB, Bolt, PostgreSQL, MongoDB (dis- For exploration, it features Stardog Studio. tributed stores) and also an ephemeral in-me- mory storage. The ability to support ACID • Virtuoso [87] is a multi-model database man- transactions is delegated to the underlying stor- agement system that supports relational as well age back-end. Cayley being distributed de- as property graphs and has an open-source [214] pends on the underlying storage being distri- edition under the GPLv2 license as well as a buted as well. Also in active development as proprietary enterprise edition. At least the fol- of 2019. lowing programming languages are supported: C/C++, C#, Java, JavaScript, .NET, PHP, Py- • HyperGraphDB [139] is as an open-source gen- thon, Ruby and Visual Basic. It supports eral purpose data storage mechanism. It is horizontal scaling, has functionalities for inter- used to store hypergraphs (a graph generaliza- active data exploration and supports SPARQL. tion where an edge can join any number of ver- tices). The low-level storage is based on Ber- We lastly note the following databases that have keleyDB and is implemented in Java. Despite been used to represent graphs, though not having the description on the website mentioning dis- an explicit description of supporting the property tribution, the support for either distributed graph model or RDF: sharding or distributed replication is not sup- ported in its current implementation [140] and • Azure Cosmos DB [221] is a commercial data- we did not find mentions of ACID transaction base solution that is multi-model, globally-dis- support. tributed, schema-agnostic, horizontally-scala- ble and fully supports ACID. It is classified as • Objectivity/DB [212] is the database tech- a NoSQL database, but the multi-model API nology powering the massively scalable graph is a relevant offering, for it can expose stored software platform ThingSpan (formerly known data for example as table rows (Cassandra), as InfiniteGraph). It is a fully-distributed collections (MongoDB) and most importantly as database (able to scale horizontally) offering graphs (Gremlin). It has connectors for Java, APIs in C++, C#, Java and Python. Objecti- .NET, Python and Xamarin. It is also a fully- vity/DB is described as a distributed object managed service with scalability, freeing devel- database, supporting many data models (a- oper resources from topics such as data center mong which highly complex and inter-related data). We did not find any information about and graph-aware optimizations. it supporting graph-specific query languages, As part of our exhaustive analysis of existing con- and licensing is defined on a case-by-case ba- tributions of different domains in the state-of-the- sis. art of graph processing and storage, we provide di- rect links to source code repositories such as GitHub Other notable mentions include the [184] OQGRAPH whenever they were available. Should the reader graph storage engine of , developed to han- MariaDB wish to delve into the implementation of a given dle hierarchies and vertices with many connections contribution, a link to the contribution’s source code and intended for retrieving hierarchical information repository is to be found as part of the bibliogra- such as graphs, routes and social relationships in phy. We provide these so that other researchers and . We note the following resources for further SQL developers may look into them without need to en- deepening of the aspects involved in the evolution gage in error-prone searches looking for up-to-date of the study of graph databases [147, 42, 258, 234, documentation and source-code. 160, 285, 73, 129, 235], covering aspects such as the This systematic analysis fosters some additional internal graph representations, experimental com- comments regarding data processing. Data is abun- parisons, principles for querying and extracting in- dant, big and evolving, and paradigms such as edge formation from the graph and designing graph da- computing and the evolution of the Internet-of-Thin- tabases to make use of distributed infrastructures. gs come together to reshape our relationship with 15. CONCLUSION data. With an increase in smart devices and compu- tational capabilities becoming more ubiquitous for This survey explores different aspects of the graph example in daily objects such as vehicles and smart processing landscape and highlights vectors of re- homes, new graphs of data mapping interaction and search. We cover dimensions that enable the clas- purpose become available. This implies a continu- sification of graph processing systems according to ous trend in the increasing size of data. At the same the mutability of data (dynamism [27] and its modal- time, the dimension of dynamism (spread across ities), the nature of the tasks (workloads where the the types we enumerate in this document) gains focus may be efficient storage [66] or swift compu- renewed importance as we move to a faster and tation [53] over transient data) and how the data is ever-connected world. With the advent of 5G tech- associated to different computing agents (e.g., dis- nologies and the alternative possibilities of space tributed via partitioning [266] to threads in a CPU, internet (among the private initiatives we count CPUs in a machine, machines in a cluster). Each SpaceX’s Starlink, Jeff Bezos’ Blue Origin and the of these dimensions constitutes a different branch of late Steve Jobs’ vision for an always-connected smart- the study of graph processing, and herein we group phone) becoming a closer reality, the temporal as- their recent literature surveys and draw on their re- pect will become even more granular. lationships. On drawing a line between graph pro- One would not be wrong to speculate that we will cessing systems and those that also focus on the have more devices which will generate data more storage, the graph databases, we found most com- frequently. In such a world, the graph processing mercial graph solutions to fall on the category of dimensions we enumerate in this document will play graph database. Graph databases, along the last a relevant role in building systems to handle these decade, have continued to refine their efficiency in changing scenarios. executing traversals and global graph algorithms Funding. This work was partly supported by over the graph representation stored in the database. national funds through FCT – Fundação para a We consider that a novel approach to extracting Ciência e Tecnologia, under projects PTDC/CPO- value from graph-based data will include the use of CPO/28495/2017, PTDC/CCI-BIO/29676/2017, P- graph-aware data compression techniques on scal- TDC/EEI-COM/30644/2017 and UIDB/50021/2020. able distributed systems, potentially breaking the abstraction that these systems establish between 16. REFERENCES the high-level graph data representations and the [1] G. Aggarwal, M. Datar, S. Rajagopalan, and M. Ruhl. On the streaming model augmented with a sorting primitive. lower-level data distribution and transmission. We In 45th Annual IEEE Symposium on Foundations of observe that the architecture of systems targeting Computer Science, pages 540–549. IEEE, 2004. [2] N. K. Ahmed, N. Duffield, T. L. Willke, and R. A. Rossi. graphs depend on how generic is the graph pro- On Sampling from Massive Graph Streams. Proc. VLDB cessing desired to be. Generic dataflow processing Endow., 10(11):1430–1441, Aug. 2017. [3] K. J. Ahn, S. Guha, and A. McGregor. Graph Sketches: systems offer abstractions over their basic computa- Sparsification, Spanners, and Subgraphs. In Proceedings tional primitives in order to represent and process of the 31st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, PODS ’12, pages graphs, but in exchange abdicate from fine-tuning 5–14, New York, NY, USA, 2012. ACM. Database Models Languages GQLs License Limits Scale ACID Visual Alibaba GDB [60] PG Go, Java, .NET, Node.js, Python Gremlin Paid N/A ChronoGraph [121, 122] PG Java Gremlin aGPL v3 N/A DSE [67] PG Java Gremlin Paid N/A Dgraph [76, 77] PG Java, JavaScript, Python GraphQL AL 2.0 Admin Graphflow[155, 195, PG Java Cypher AL 2.0 N/A 194] Elixir, Go, Java, .NET, PHP, JanusGraph [143, 17] PG Gremlin AL 2.0 N/A Python, Ruby, Scala Nebula Graph [288] PG Go, Java, Python nGQL AL 2.0, CC 1.0 N/A C/C++, Clojure, Go, Haskell, Neo4j [294, 209] PG Java, JavaScript, .NET, Perl, Cypher GPL v3 Size PHP, Python, R, Ruby Elixir, Go, Java, JavaScript, RedisGraph [49, 177] PG Cypher Custom N/A PHP, Python, Ruby, Rust Hana [242, 134, 248] PG N/A Cypher Paid N/A C++, .NET, Java, Sparksee [188, 267] PG N/A Custom Size Objective-C, Python TigerGraph [75, 280] PG N/A GSQL Custom Scale Weaver [83, 82] PG C++, Python N/A Custom N/A C#, C, Common Lisp, Clojure, AllegroGraph [136] RDF SPARQL EPL v1 Size Java, Perl, Python Gremlin BlazeGraph PG RDF O .NET Python , GPL v2 N/A [275] , , , SPARQL BrightstarDB [35] RDF .NET SPARQL MIT N/A Cray Graph Engi- RDF Java, Python SPARQL Paid N/A ne [233] Ontotext GraphDB [213] RDF Java, JavaScript SPARQL Custom Scale Gremlin Neptune PG RDF Managed , Paid N/A [25, 8] , SPARQL Gremlin AnzoGraph DB PG RDF Java C++ , Custom Scale [50] , , SPARQL ArangoDB [278] PG Go, Java, JavaScript, PHP AQL AL 2.0 Admin IBM System G [135] PG, RDF Managed Gremlin Paid N/A Clojure, Go, Java, OrientDB [277, 217] PG, O JavaScript,.NET, Node.js, PHP, 6 AL 2.0 Visual Python, R, Ruby, Scala PGQL OSG PG RDF Java , Paid N/A [216] , SPARQL GraphQL Clojure Groovy Java , Stardog RDF , , , Gremlin Paid Trial [269] JavaScript .NET , , SPARQL C/C++, C#, Java, JavaScript, Virtuoso [87, 214] PG, O .NET, PHP, Python, SPARQL GPL v2 Admin Ruby, Visual Basic Cosmos DB [221, 65] MM Gremlin Managed Paid N/A Android, C#, Go, Java, FaunaDB [93] MM JavaScript, Python, Ruby, FQL Custom Quota Scala, Swift Gizmo Cayley RDF N/A , AL 2.0 N/A [109, 193] GraphQL HyperGraphDB [139] HG Java N/A AL 2.0 N/A Objectivity/DB [212] O C++, C#, Java, Python N/A Paid N/A

Table 2: Summary of graph database distinctive features. Circle on the ACID, Visual and Scale (updating a graph that is distributed) columns indicate they are present. Models lists the supported data models: property graph model (PG), RDF, multi-model (MM or others (O). Languages lists the programming languages that interface with the database, either directly or by being supported by the supported interfaces (e.g., TinkerPop drivers). GQLs lists only directly supported graph query languages (and shown in italic if they are not standard) - those supported using compatibility tools as seen previously are not included. License lists the licenses of the open-source project or of the free edition of a commercial product: AL 2.0 is Apache License 2.0, CC 1.0 is Commons Clause 1.0, aGPL v3 is the Affero General Public License 3, (GPL) v3 is GNU General Public License (GPL) v3. Limits covers free version limitations in commercial products (N/A if it does not apply). If a database has Custom license and limits are not N/A, it means it has a non-standard free license.

[4] N. R. Al-Molhem, Y. Rahal, and M. Dakkak. Social network analysis in Telecom data. Journal of Big Data, 6(1):99, 2019. Concepts, Models, Systems, and Parallelism. CoRR, [5] A. Alexandrov, R. Bergmann, S. Ewen, J.-C. Freytag, abs/1912.12740, 2019. F. Hueske, A. Heise, O. Kao, M. Leich, U. Leser, [28] M. Besta, D. Stanojevic, T. Zivic, J. Singh, M. Hoerold, V. Markl, F. Naumann, M. Peters, A. Rheinländer, M. J. and T. Hoefler. Log (graph) a near-optimal Sax, S. Schelter, M. Höger, K. Tzoumas, and D. Warneke. high-performance graph representation. In Proceedings of The stratosphere platform for big data analytics. The the 27th International Conference on Parallel VLDB Journal, 23(6):939–964, 2014. Architectures and Compilation Techniques, pages 1–13, [6] B. Alverson, E. Froese, L. Kaplan, and D. Roweth. Cray 2018. XC series network. Cray Inc., White Paper [29] Y. Beyene, M. Faloutsos, D. H. Chau, and C. Faloutsos. WP-Aries01-1112, 2012. The eBay Graph: How do online auction users interact? [7] B. Amann and M. Scholl. Gram: a graph data model and In IEEE INFOCOM Workshops 2008, pages 1–6. IEEE, query languages. In Proceedings of the ACM conference 2008. on Hypertext, pages 201–211, 1993. [30] P. Boldi, B. Codenotti, M. Santini, and S. Vigna. [8] I. Amazon. Amazon Neptune Samples - Source Code. Ubicrawler: A scalable fully distributed web crawler. [Online, GitHub; accessed 24-April-2020], 2020. Software: Practice & Experience, 34(8):711–726, 2004. [9] K. Andreev and H. Racke. Balanced graph partitioning. [31] P. Boldi, A. Marino, M. Santini, and S. Vigna. BUbiNG: Theory of Computing Systems, 39(6):929–939, 2006. Massive Crawling for the Masses. In Proceedings of the [10] R. Angles. The Property Graph Database Model. [Online; Companion Publication of the 23rd International accessed 24-April-2020], 2018. Conference on World Wide Web, pages 227–228. [11] R. Angles, M. Arenas, P. Barceló, P. Boncz, G. Fletcher, International World Wide Web Conferences Steering C. Gutierrez, T. Lindaaker, M. Paradies, S. Plantikow, Committee, 2014. J. Sequeda, et al. G-CORE: A core for future graph [32] P. Boldi, M. Rosa, M. Santini, and S. Vigna. Layered query languages. In Proceedings of the 2018 Label Propagation: A Multiresolution Coordinate-free International Conference on Management of Data, Ordering for Compressing Social Networks. In pages 1421–1432, 2018. S. Srinivasan, K. Ramamritham, A. Kumar, M. P. [12] R. Angles, M. Arenas, P. Barceló, A. Hogan, J. Reutter, Ravindra, E. Bertino, and R. Kumar, editors, and D. Vrgoč. Foundations of modern query languages Proceedings of the 20th International Conference on for graph databases. ACM Computing Surveys (CSUR), World Wide Web, WWW ’11, pages 587–596, New York, 50(5):1–40, 2017. NY, USA, 2011. ACM. [13] R. Angles and C. Gutierrez. Survey of graph database [33] P. Boldi and S. Vigna. The WebGraph Framework II: models. ACM Computing Surveys (CSUR), 40(1):1–39, Codes For The World-Wide Web. In 2004 Data 2008. Compression Conference (DCC 2004), 23-25 March [14] A. Apostolico and G. Drovandi. Graph compression by 2004, Snowbird, UT, USA, page 528. IEEE Computer BFS. Algorithms, 2(3):1031–1044, 2009. Society, 2004. [15] M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. [34] P. Boldi and S. Vigna. The WebGraph framework I: Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, Compression techniques. In S. I. Feldman, M. Uretsky, and M. Zaharia. Spark SQL: Relational Data Processing M. Najork, and C. E. Wills, editors, Proceedings of the in Spark. In Proceedings of the 2015 ACM SIGMOD 13th international conference on World Wide Web, International Conference on Management of Data, WWW 2004, New York, NY, USA, May 17-20, 2004, SIGMOD ’15, pages 1383–1394, New York, NY, USA, pages 595–602, New York, NY, USA, 2004. ACM. 2015. ACM. [35] BrightstarDB. BrightstarDB - Source Code. [Online; [16] Aurelius. Titan: Distributed Graph Database. [Online; accessed 24-April-2020], 2015. accessed 24-April-2020], 2015. [36] S. Brin and L. Page. The anatomy of a large-scale [17] J. Authors. JanusGraph - Source Code. [Online, GitHub; hypertextual web search engine. Computer networks and accessed 24-April-2020], 2020. ISDN systems, 30(1-7):107–117, 1998. [18] R. A. Bahrami, J. Gulati, and M. Abulaish. Efficient [37] N. R. Brisaboa, A. Cerdeira-Pena, G. de Bernardo, and processing of SPARQL queries over graphframes. In G. Navarro. Compressed representation of dynamic Proceedings of the International Conference on Web binary relations with applications. Information Systems, Intelligence, pages 678–685, 2017. 69:106–123, 2017. [19] P. Bajardi, C. Poletto, J. J. Ramasco, M. Tizzoni, [38] N. R. Brisaboa, G. de Bernardo, G. Gutiérrez, S. Ladra, V. Colizza, and A. Vespignani. Human mobility networks, M. R. Penabad, and B. A. Troncoso. Efficient Set travel restrictions, and the global spread of 2009 H1N1 Operations over k2-Trees. In Data Compression pandemic. PloS one, 6(1), 2011. Conference (DCC), pages 373–382, 2015. [20] A. T. Balaban. Applications of graph theory in [39] N. R. Brisaboa, S. Ladra, and G. Navarro. Compact chemistry. Journal of chemical information and representation of web graphs with extended functionality. computer sciences, 25(3):334–343, 1985. Information Systems, 39:152–174, 2014. [21] B. Balasundaram and S. Butenko. Graph domination, [40] D. Brockmann and D. Helbing. The hidden geometry of coloring and cliques in telecommunications. In Handbook complex, network-driven contagion phenomena. science, of Optimization in Telecommunications, pages 865–890. 342(6164):1337–1342, 2013. Springer, 2006. [41] G. Buehrer and K. Chellapilla. A scalable pattern mining [22] N. T. Bao and T. Suzumura. Towards highly scalable approach to web graph compression with communities. In pregel-based graph processing platform with x10. In Proceedings of the 2008 International Conference on Proceedings of the 22nd International Conference on Web Search and Data Mining, pages 95–106, 2008. World Wide Web, pages 501–508, 2013. [42] M. Buerli and C. Obispo. The current state of graph [23] A. Baritchi, D. J. Cook, and L. B. Holder. Discovering databases. Department of Computer Science, Cal Poly Structural Patterns in Telecommunications Data. In San Luis Obispo, mbuerli@ calpoly. edu, 32(3):67–83, FLAIRS Conference, pages 82–85, 2000. 2012. [24] P. S. Bearman, J. Moody, and K. Stovel. Chains of [43] A. Buluç, J. T. Fineman, M. Frigo, J. R. Gilbert, and affection: The structure of adolescent romantic and C. E. Leiserson. Parallel sparse matrix-vector and sexual networks. American journal of sociology, matrix-transpose-vector multiplication using compressed 110(1):44–91, 2004. sparse blocks. In Proceedings of the twenty-first annual [25] B. R. Bebee, D. Choi, A. Gupta, A. Gutmans, symposium on Parallelism in algorithms and A. Khandelwal, Y. Kiran, S. Mallidi, B. McGaughy, architectures, pages 233–244, 2009. M. Personick, K. Rajan, et al. Amazon Neptune: Graph [44] A. Buluç and J. R. Gilbert. Combinatorial BLAS - Source Data Management in the Cloud. [Online; accessed Code. [Online, Berkeley; accessed 24-April-2020], 2011. 24-April-2020], 2018. [45] A. Buluç and J. R. Gilbert. The Combinatorial BLAS: [26] A. Benisis. Business Process Management: A Data Design, implementation, and applications. The Cube to Analyze Business Process Simulation Data for International Journal of High Performance Computing Decision Making. VDM Publishing, 2010. Applications, 25(4):496–509, 2011. [27] M. Besta, M. Fischer, V. Kalavri, M. Kapralov, and [46] A. Buluç, H. Meyerhenke, I. Safro, P. Sanders, and T. Hoefler. Practice of Streaming and Dynamic Graphs: C. Schulz. Recent advances in graph partitioning. In Algorithm Engineering, pages 117–158. Springer, 2016. Design and Implementation, pages 752–768, 2018. [47] F. Busato, O. Green, N. Bombieri, and D. A. Bader. [69] R. Dathathri, G. Gill, L. Hoang, H.-V. Dang, A. Brooks, Hornet: An efficient data structure for dynamic sparse N. Dryden, M. Snir, and K. Pingali. faimGraph - Source graphs and matrices on GPUs. In 2018 IEEE High Code. [Online, GitHub; accessed 24-April-2020], 2020. Performance extreme Computing Conference (HPEC), [70] R. Dathathri, G. Gill, L. Hoang, H.-V. Dang, A. Brooks, pages 1–7. IEEE, 2018. N. Dryden, M. Snir, and K. Pingali. Gluon - Source [48] F. Busato, O. Green, N. Bombieri, and D. A. Bader. Code. [Online, GitHub; accessed 24-April-2020], 2020. Hornet - Source Code. [Online, GitHub; accessed [71] A. Dave, A. Jindal, L. E. Li, R. Xin, J. Gonzalez, and 24-April-2020], 2020. M. Zaharia. GraphFrames: an integrated API for mixing [49] P. Cailliau, T. Davis, V. Gadepally, J. Kepner, graph and relational queries. In Proceedings of the R. Lipman, J. Lovitz, and K. Ouaknine. RedisGraph Fourth International Workshop on Graph Data GraphBLAS Enabled Graph Database. In 2019 IEEE Management Experiences and Systems, pages 1–8, 2016. International Parallel and Distributed Processing [72] N. G. de Bruijn. A Combinatorial Problem. Koninklijke Symposium Workshops (IPDPSW), pages 285–286. Nederlandsche Akademie Van Wetenschappen, IEEE, 2019. 49(6):758–764, June 1946. [50] Cambridge Semantics. AnzoGraph® DB. [Online; [73] R. De Virgilio, A. Maccioni, and R. Torlone. accessed 24-April-2020], 2020. Model-driven design of graph databases. In International [51] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, Conference on Conceptual Modeling, pages 172–185. S. Haridi, and K. Tzoumas. Apache Flink: Stream and Springer, 2014. batch processing in a single engine. Bulletin of the IEEE [74] C. Demetrescu, I. Finocchi, and A. Ribichini. Trading off Computer Society Technical Committee on Data space for passes in graph streaming problems. ACM Engineering, 36(4):28–38, 2015. Transactions on Algorithms (TALG), 6(1):1–17, 2009. [52] H. N. Chaudhry. FlowGraph: Distributed temporal [75] A. Deutsch, Y. Xu, M. Wu, and V. Lee. TigerGraph: A pattern detection over dynamically evolving graphs. In Native MPP Graph Database. arXiv preprint Proceedings of the 13th ACM International Conference arXiv:1901.08248, 2019. on Distributed and Event-based Systems, pages 272–275, [76] Dgraph Labs, Inc. A Tour of Dgraph. [Online; accessed 2019. 24-April-2020], April 2020. [53] C. Chen, X. Yan, F. Zhu, J. Han, and P. S. Yu. Graph [77] Dgraph Labs, Inc. Dgraph - Source Code. [Online, OLAP: Towards Online Analytical Processing on Graphs. GitHub; accessed 24-April-2020], 2020. In 2008 Eighth IEEE International Conference on Data [78] L. Dhulipala, G. Blelloch, and J. Shun. Julienne: A Mining, pages 103–112, Dec 2008. framework for parallel graph algorithms using [54] R. Chen, J. Shi, Y. Chen, and H. Chen. PowerLyra: work-efficient bucketing. In Proceedings of the 29th Differentiated graph computation and partitioning on ACM Symposium on Parallelism in Algorithms and skewed graphs. In Proceedings of the Tenth European Architectures, pages 293–304, 2017. Conference on Computer Systems, page 1. ACM, 2015. [79] L. Dhulipala, G. E. Blelloch, and J. Shun. Low-latency [55] R. Chen, J. Shi, Y. Chen, and H. Chen. PowerLyra - graph streaming using compressed purely-functional Source Code. [Online, GitHub; accessed 24-April-2020], trees. In Proceedings of the 40th ACM SIGPLAN 2018. Conference on Programming Language Design and [56] X. Chen. GraphCage: Cache Aware Graph Processing on Implementation, pages 918–934, 2019. GPUs. arXiv preprint arXiv:1904.02241, 2019. [80] L. Dhulipala, G. E. Blelloch, and J. Shun. Aspen - Source [57] R. Cheng, J. Hong, A. Kyrola, Y. Miao, X. Weng, Code. [Online, GitHub; accessed 24-April-2020], 2020. M. Wu, F. Yang, L. Zhou, F. Zhao, and E. Chen. [81] D. Dominguez-Sal, N. Martinez-Bazan, Kineograph: Taking the Pulse of a Fast-changing and V. Muntes-Mulero, P. Baleta, and J. L. Larriba-Pey. A Connected World. In Proceedings of the 7th ACM discussion on the design of graph database benchmarks. European Conference on Computer Systems, EuroSys In Technology Conference on Performance Evaluation ’12, pages 85–98, New York, NY, USA, 2012. ACM. and Benchmarking, pages 25–40. Springer, 2010. [58] M. Chinazzi, J. T. Davis, M. Ajelli, C. Gioannini, [82] A. Dubey. Weaver - Source Code. [Online, GitHub; M. Litvinova, S. Merler, A. P. y Piontti, K. Mu, L. Rossi, accessed 24-April-2020], 2016. K. Sun, et al. The effect of travel restrictions on the [83] A. Dubey, G. D. Hill, R. Escriva, and E. G. Sirer. spread of the 2019 novel coronavirus (COVID-19) Weaver: A High-Performance, Transactional Graph outbreak. Science, 2020. Database Based on Refinable Timestamps. PVLDB, [59] A. Ching. Scaling Apache Giraph to a trillion edges. 9(11):852–863, 2016. Facebook Engineering Blog, page 25, 2013. [84] D. Ediger, R. McColl, J. Riedy, and D. A. Bader. Stinger: [60] A. Cloud. Alibaba Graph Database. [Online; accessed High performance data structure for streaming graphs. In 24-April-2020], 2020. 2012 IEEE Conference on High Performance Extreme [61] M. E. Coimbra, S. Esteves, A. P. Francisco, and L. Veiga. Computing, pages 1–5. IEEE, 2012. VeilGraph: Streaming Graph Approximations. arXiv, [85] C. Eksombatchai, P. Jindal, J. Z. Liu, Y. Liu, R. Sharma, pages arXiv–1810, 2019. C. Sugnet, M. Ulrich, and J. Leskovec. Pixie: A System [62] M. E. Coimbra, A. P. Francisco, L. M. S. Russo, for Recommending 3+ Billion Items to 200+ Million G. de Bernardo, S. Ladra, and G. Navarro. On dynamic Users in Real-Time. In Proceedings of the 2018 World succinct graph representations. In Data Compression Wide Web Conference, WWW ’18, pages 1775–1784, Conference (DCC), page 10. IEEE, January 2020. Republic and Canton of Geneva, Switzerland, 2018. [63] V. Colizza, A. Barrat, M. Barthélemy, and A. Vespignani. International World Wide Web Conferences Steering Predictability and epidemic pathways in global outbreaks Committee. of infectious diseases: the SARS case study. BMC [86] B. Erb, D. Meißner, F. Kargl, B. A. Steer, F. Cuadrado, medicine, 5(1):34, 2007. D. Margan, and P. Pietzuch. GraphTides: a framework [64] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and for evaluating stream-based graph processing platforms. C. Stein. Introduction to Algorithms, Third Edition. In Proceedings of the 1st ACM SIGMOD joint The MIT Press, 3rd edition, 2009. international workshop on graph data management [65] M. Corporation. Hub for Azure Cosmos DB - Source experiences & systems (GRADES) and network data Code. [Online, GitHub; accessed 24-April-2020], 2019. analytics (NDA), pages 1–10, 2018. [66] O. Corporation. Application and System Performance [87] O. Erling. Virtuoso, a Hybrid RDBMS/Graph Column Characteristics. [Online, GitHub; accessed 24-April-2020], Store. IEEE Data Eng. Bull., 35(1):3–8, 2012. Dec 1999. [88] L. Euler. The seven bridges of Konigsberg. Wm. Benton, [67] DataStax, Inc. DataStax Enterprise Graph. [Online; 1956. accessed 24-April-2020], 2016. [89] Facebook. Newsrooms. [Online; accessed 05-May-2020], [68] R. Dathathri, G. Gill, L. Hoang, H.-V. Dang, A. Brooks, 2020. N. Dryden, M. Snir, and K. Pingali. Gluon: A [90] Facebook Database Engineering Team. RocksDB - Source communication-optimizing substrate for distributed Code. [Online, GitHub; accessed 24-April-2020], 2012. heterogeneous graph analytics. In Proceedings of the 39th [91] Facebook, Inc. GraphQL. [Online, GitHub; accessed ACM SIGPLAN Conference on Programming Language 24-April-2020], October 2016. [92] W. Fan, J. Li, X. Wang, and Y. Wu. Query preserving [116] Q. Gu, J. Zhou, and C. Ding. Collaborative filtering: graph compression. In Proceedings of the 2012 ACM Weighted nonnegative matrix factorization incorporating SIGMOD International Conference on Management of user and item graphs. In Proceedings of the 2010 SIAM Data, pages 157–168, 2012. international conference on data mining, pages 199–210. [93] I. Fauna. FaunaDB | The database built for serverless, SIAM, 2010. featuring native GraphQL. [Online; accessed [117] A. Gubichev and M. Then. Graph Pattern Matching: Do 24-April-2020], 2020. We Have to Reinvent the Wheel? In Proceedings of [94] T. Feder and R. Motwani. Clique partitions, graph Workshop on GRAph Data management Experiences compression and speeding-up algorithms. Journal of and Systems, pages 1–7, 2014. Computer and System Sciences, 51(2):261–272, 1995. [118] R. Guidotti and M. Coscia. On the equivalence between [95] J. Feigenbaum, S. Kannan, A. McGregor, S. Suri, and community discovery and clustering. In International J. Zhang. On graph problems in a semi-streaming model. Conference on Smart Objects and Technologies for Departmental Papers (CIS), page 236, 2005. Social Good, pages 342–352. Springer, 2017. [96] S. Fortunato, A. Flammini, and F. Menczer. Scale-free [119] Y. Guo, Z. Pan, and J. Heflin. LUBM: A benchmark for network growth by ranking. Physical review letters, OWL knowledge base systems. Journal of Web 96(21):218701, 2006. Semantics, 3(2-3):158–182, 2005. [97] T. A. S. Foundation. Apache Giraph - Source Code. [120] P. Haase, J. Broekstra, A. Eberhart, and R. Volz. A [Online, GitHub; accessed 24-April-2020], 2019. comparison of RDF query languages. In International [98] T. A. S. Foundation. Apache TinkerPop. [Online; Semantic Web Conference, pages 502–517. Springer, accessed 24-April-2020], 2019. 2004. [99] T. A. S. Foundation. Apache Flink - Source Code. [121] M. Haeusler, T. Trojer, J. Kessler, M. Farwick, [Online, GitHub; accessed 24-April-2020], 2020. E. Nowakowski, and R. Breu. ChronoGraph: A Versioned [100] T. A. S. Foundation. Apache Spark - Source Code. TinkerPop Graph Database. In International Conference [Online, GitHub; accessed 24-April-2020], 2020. on Data Management Technologies and Applications, [101] N. Francis, A. Green, P. Guagliardo, L. Libkin, pages 237–260. Springer, 2017. T. Lindaaker, V. Marsault, S. Plantikow, M. Rydberg, [122] M. Haeusler, T. Trojer, J. Kessler, M. Farwick, P. Selmer, and A. Taylor. Cypher: An evolving query E. Nowakowski, and R. Breu. ChronoDB - Source Code. language for property graphs. In Proceedings of the 2018 [Online, GitHub; accessed 24-April-2020], 2020. International Conference on Management of Data, [123] M. Han, K. Daudjee, K. Ammar, M. T. Özsu, X. Wang, pages 1433–1445, 2018. and T. Jin. An Experimental Comparison of Pregel-like [102] Z. Fu, M. Personick, and B. Thompson. MapGraph: A Graph Processing Systems. Proc. VLDB Endow., high level API for fast development of high performance 7(12):1047–1058, Aug. 2014. graph analytics on GPUs. In Proceedings of Workshop [124] O. Hartig. Reconciliation of RDF* and property graphs. on GRAph Data management Experiences and Systems, arXiv preprint arXiv:1409.3288, 2014. pages 1–6, 2014. [125] O. Hartig and J. Pérez. An initial analysis of Facebook’s [103] Z. Fu, M. Personick, and B. Thompson. MapGraph - GraphQL language. [Online; accessed 24-April-2020], Source Code. [Online, GitHub; accessed 24-April-2020], 2017. 2016. [126] O. Hartig and J. Pérez. Semantics and complexity of [104] T. Gagie, J. I. González-Nova, S. Ladra, G. Navarro, and GraphQL. In Proceedings of the 2018 World Wide Web D. Seco. Faster compressed quadtrees. In Data Conference, pages 1155–1164, 2018. Compression Conference (DCC), 2015, pages 93–102. [127] H. He and A. K. Singh. Graphs-at-a-time: query language IEEE, 2015. and access methods for graph databases. In Proceedings [105] G. George and S. M. Thampi. A graph-based security of the 2008 ACM SIGMOD international conference on framework for securing industrial IoT networks from Management of data, pages 405–418, 2008. vulnerability exploitations. IEEE Access, 6:43586–43601, [128] S. Heidari, Y. Simmhan, R. N. Calheiros, and R. Buyya. 2018. Scalable Graph Processing Frameworks: A Taxonomy [106] L. George. HBase - The Definitive Guide: Random and Open Challenges. ACM Computing Surveys Access to Your Planet-Size Data. O’Reilly, 2011. (CSUR), 51(3):60, 2018. [107] J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and [129] C. E. Henderson. System and method for creating, C. Guestrin. PowerGraph: Distributed Graph-parallel deploying, integrating, and distributing nodes in a grid of Computation on Natural Graphs. In Proceedings of the distributed graph databases, 2014. US Patent 8,775,476. 10th USENIX Conference on Operating Systems Design [130] C. Hernández and G. Navarro. Compressed and Implementation, OSDI’12, pages 17–30, Berkeley, representations for web and social graphs. Knowl. Inf. CA, USA, 2012. USENIX Association. Syst., 40(2):279–313, 2014. [108] J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. [131] F. Holzschuher and R. Peinl. Performance of Graph Franklin, and I. Stoica. GraphX: Graph Processing in a Query Languages: Comparison of Cypher, Gremlin and Distributed Dataflow Framework. In Proceedings of the Native Access in Neo4J. In Proceedings of the Joint 11th USENIX Conference on Operating Systems Design EDBT/ICDT 2013 Workshops, EDBT ’13, pages and Implementation, OSDI’14, pages 599–613, Berkeley, 195–204, New York, NY, USA, 2013. ACM. CA, USA, 2014. USENIX Association. [132] P. Horn and M. Rydberg. openCypher - Source Code. [109] Google. Google Cayley - Source Code. [Online, GitHub; [Online, GitHub; accessed 24-April-2020], 2020. accessed 24-April-2020], 2017. [133] J. Huang, D. J. Abadi, and K. Ren. Scalable SPARQL [110] Gordon Donnelly. 75 Super-Useful Facebook Statistics for querying of large RDF graphs. Proceedings of the VLDB 2018. [Online; accessed 05-May-2020], March 2020. Endowment, 4(11):1123–1134, 2011. [111] A. Green, M. Junghanns, M. Kießling, T. Lindaaker, [134] M. Hwang. Graph Processing Using SAP HANA: A S. Plantikow, and P. Selmer. openCypher: New Teaching Case. e-Journal of Business Education and Directions in Property Graph Querying. In EDBT, pages Scholarship of Teaching, 12(2):155–165, 2018. 520–523, 2018. [135] IBM System G Team. The World is Big and Linked: [112] D. Gregor and A. Lumsdaine. The parallel BGL: A Whole Spectrum Industry Solutions towards Big Graphs generic library for distributed graph computations. - Graph Computing and Tutorial of IBM System G. Parallel Object-Oriented Scientific Computing [Online; accessed 24-April-2020], Oct 2015. (POOSC), 2:1–18, 2005. [136] F. Inc. AllegroGraph. [Online; accessed 24-April-2020], [113] D. Gregor and A. Lumsdaine. Parallel Boost Graph 1984. Library - Source Code. [Online, GitHub; accessed [137] N. Inc. Neo4j APOC Library - Source Code. [Online, 24-April-2020], 2018. GitHub; accessed 24-April-2020], 2020. [114] S. Gregory. Finding overlapping communities in networks [138] InternetLiveStats.com. Total number of Websites. by label propagation. New journal of Physics, [Online; accessed 24-April-2020], 2020. 12(10):103018, 2010. [139] B. Iordanov. HyperGraphDB: a generalized graph [115] J. Grujić. Movies recommendation networks as bipartite database. In International Conference on Web-age graphs. In International Conference on Computational Information Management, pages 25–36. Springer, 2010. Science, pages 576–583. Springer, 2008. [140] B. Iordanov. Distributed HypergraphDB: partial [161] V. Kostakos. Temporal graphs. Physica A: Statistical replication. [Online; accessed 24-April-2020], 2020. Mechanics and its Applications, 388(6):1007–1023, 2009. [141] A. P. Iyer, L. E. Li, T. Das, and I. Stoica. Time-evolving [162] Laboratory for Web Algorithmics. Datasets. [Online; graph processing at scale. In Proceedings of the Fourth accessed 05-May-2020], 2020. International Workshop on Graph Data Management [163] A. Lakshman and P. Malik. Cassandra: A Decentralized Experiences and Systems, pages 1–6, 2016. Structured Storage System. SIGOPS Oper. Syst. Rev., [142] N. Jain, G. Liao, and T. L. Willke. Graphbuilder: 44(2):35–40, Apr. 2010. scalable graph etl framework. In First International [164] R. Li, H. Zhu, J. Ruan, W. Qian, X. Fang, Z. Shi, Y. Li, Workshop on Graph Data Management Experiences and S. Li, G. Shan, K. Kristiansen, S. Li, H. Yang, J. Wang, Systems, pages 1–6, 2013. and J. Wang. De novo assembly of human genomes with [143] JanusGraph Authors. JanusGraph: Distributed Graph massively parallel short read sequencing. Genome Database. [Online; accessed 24-April-2020], 2017. Research, 20(2):265–272, 2010. [144] I. JCC Consulting. GQL Standard. [Online; accessed [165] W. Lightenberg, Y. Pei, G. Fletcher, and M. Pechenizkiy. 24-April-2020], 2020. Tink: A temporal graph analytics library for Apache [145] Z. Jia, Y. Kwon, G. Shipman, P. McCormick, M. Erez, Flink. In Companion Proceedings of the The Web and A. Aiken. A distributed multi-gpu system for fast Conference 2018, pages 71–72, 2018. graph processing. Proceedings of the VLDB Endowment, [166] W. Lightenberg, Y. Pei, G. Fletcher, and M. Pechenizkiy. 11(3):297–310, 2017. Tink - Source Code. [Online, GitHub; accessed [146] Z. Jia, Y. Kwon, G. Shipman, P. McCormick, M. Erez, 24-April-2020], 2019. and A. Aiken. Lux - Source Code. [Online, GitHub; [167] W. Lightenberg, Y. Pei, G. Fletcher, and M. Pechenizkiy. accessed 24-April-2020], 2018. GRADOOP - Source Code. [Online, GitHub; accessed [147] C. Jin, S. S. Bhowmick, X. Xiao, J. Cheng, and B. Choi. 24-April-2020], 2020. GBLENDER: towards blending visual query formulation [168] F. Liljeros, C. R. Edling, and L. A. N. Amaral. Sexual and query processing in graph databases. In Proceedings networks: implications for the transmission of sexually of the 2010 ACM SIGMOD International Conference transmitted infections. Microbes and infection, on Management of data, pages 111–122, 2010. 5(2):189–196, 2003. [148] M. Junghanns, M. Kießling, A. Averbuch, A. Petermann, [169] Y. Lim, U. Kang, and C. Faloutsos. Slashburn: Graph and E. Rahm. Cypher-based Graph Pattern Matching in compression and mining beyond caveman communities. Gradoop. In P. A. Boncz and J. Larriba-Pey, editors, IEEE Transactions on Knowledge and Data Proceedings of the Fifth International Workshop on Engineering, 26(12):3077–3089, 2014. Graph Data-management Experiences & Systems, [170] LinkedIn Corporation. Quarterly results. [Online; GRADES@SIGMOD/PODS 2017, Chicago, IL, USA, accessed 05-May-2020], 2020. May 14 - 19, 2017, pages 3:1–3:8. ACM, 2017. [171] X. Liu and T. Murata. Advanced modularity-specialized [149] M. Junghanns, M. Kießling, N. Teichmann, K. Gómez, label propagation algorithm for detecting communities in A. Petermann, and E. Rahm. Declarative and distributed networks. Physica A: Statistical Mechanics and its graph analytics with GRADOOP. PVLDB, Applications, 389(7):1493–1500, 2010. 11(12):2006–2009, 2018. [172] Y. Liu, C. Zhou, J. Gao, and Z. Fan. GiraphAsync: [150] D. Kakwani and Y. Simmhan. Distributed Algorithms for Supporting Online and Offline Graph Processing via Subgraph-Centric Graph Platforms. arXiv preprint Adaptive Asynchronous Message Processing. In arXiv:1905.08051, 2019. Proceedings of the 25th ACM International on [151] V. Kalavri, P. Carbone, D. Bali, and Z. Abbas. Gelly Conference on Information and Knowledge Streaming - Source Code. [Online, GitHub; accessed Management, CIKM ’16, pages 479–488, New York, NY, 24-April-2020], 2019. USA, 2016. ACM. [152] V. Kalavri, S. Ewen, K. Tzoumas, V. Vlassov, V. Markl, [173] E. T. Lofgren and N. H. Fefferman. The untapped and S. Haridi. Asymmetry in Large-Scale Graph Analysis, potential of virtual game worlds to shed light on real Explained. In Proceedings of Workshop on GRAph Data world epidemics. The Lancet infectious diseases, Management Experiences and Systems, GRADES’14, 7(9):625–629, 2007. pages 4:1–4:7, New York, NY, USA, 2014. ACM. [174] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, [153] V. Kalavri, V. Vlassov, and S. Haridi. High-level and J. Hellerstein. GraphLab: A New Framework for programming abstractions for distributed graph Parallel Machine Learning. In Proceedings of the processing. IEEE Transactions on Knowledge and Data Twenty-Sixth Conference on Uncertainty in Artificial Engineering, 30(2):305–324, 2017. Intelligence, UAI’10, page 340–349, Arlington, Virginia, [154] U. Kang and C. Faloutsos. Beyond’caveman USA, 2010. AUAI Press. communities’: Hubs and spokes for graph compression [175] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and mining. In 2011 IEEE 11th International and J. Hellerstein. GraphLab - Source Code. [Online, Conference on Data Mining, pages 300–309. IEEE, 2011. GitHub; accessed 24-April-2020], 2014. [155] C. Kankanamge, S. Sahu, A. Mhedbhi, J. Chen, and [176] Y. Low, J. E. Gonzalez, A. Kyrola, D. Bickson, S. Salihoglu. Graphflow: An active graph database. In C. Guestrin, and J. M. Hellerstein. GraphLab: A new Proceedings of the 2017 ACM International Conference framework for parallel machine learning. CoRR, on Management of Data, pages 1695–1698, 2017. abs/1408.2041, 2014. [156] Kankanamge, Chathura and Sahu, Siddhartha and [177] R. L. Ltd. RedisGraph - Source Code. [Online, GitHub; Mhedbhi, Amine and Chen, Jeremy and Salihoglu, accessed 24-April-2020], 2020. Semih. Graphflow - Source Code. [Online, GitHub; [178] A. Lumsdaine, D. Gregor, B. Hendrickson, and J. Berry. accessed 24-April-2020], 2017. Challenges in parallel graph processing. Parallel [157] F. Khorasani, K. Vora, R. Gupta, and L. N. Bhuyan. Processing Letters, 17(01):5–20, 2007. CuSha: vertex-centric graph processing on GPUs. In [179] W. Lyon. GRANDstack. [Online; accessed 24-April-2020], Proceedings of the 23rd international symposium on 2020. High-performance parallel and distributed computing, [180] S. Maass, C. Min, S. Kashyap, W. Kang, M. Kumar, and pages 239–252, 2014. T. Kim. Mosaic - Source Code. [Online, GitHub; accessed [158] F. Khorasani, K. Vora, R. Gupta, and L. N. Bhuyan. 24-April-2020], 2017. CuSha - Source Code. [Online, GitHub; accessed [181] S. Maass, C. Min, S. Kashyap, W. Kang, M. Kumar, and 24-April-2020], 2015. T. Kim. Mosaic: Processing a Trillion-Edge Graph on a [159] S. Ko and W.-S. Han. TurboGraph++ A Scalable and Single Machine. In Proceedings of the Twelfth European Fast Graph Analytics System. In Proceedings of the 2018 Conference on Computer Systems, EuroSys ’17, pages International Conference on Management of Data, 527–543, New York, NY, USA, 2017. ACM. pages 395–410, 2018. [182] G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, [160] V. Kolomičenko, M. Svoboda, and I. H. Ml`ynková. I. Horn, N. Leiser, and G. Czajkowski. Pregel: A System Experimental comparison of graph databases. In for Large-scale Graph Processing. In Proceedings of the Proceedings of International Conference on Information 2010 ACM SIGMOD International Conference on Integration and Web-based Applications & Services, Management of Data, SIGMOD ’10, pages 135–146, New pages 115–124, 2013. York, NY, USA, 2010. ACM. [183] J. Malicevic, A. Roy, and W. Zwaenepoel. Scale-up [205] I. Neo4j. What is openCypher? [Online; accessed Graph Processing in the Cloud: Challenges and 24-April-2020], 2018. Solutions. In Proceedings of the Fourth International [206] I. Neo4j. Cypher for Gremlin - Source Code. [Online, Workshop on Cloud Data and Platforms, CloudDP ’14, GitHub; accessed 24-April-2020], 2019. pages 5:1–5:6, New York, NY, USA, 2014. ACM. [207] I. Neo4j. Neo4j and GraphQL. [Online; accessed [184] MariaDB. Open Query GRAPH computation engine. 24-April-2020], 2020. [Online; accessed 24-April-2020], 2016. [208] I. Neo4j. What is a Graph Database? [Online; accessed [185] M. Mariappan and K. Vora. GraphBolt: 24-April-2020], 2020. Dependency-Driven Synchronous Processing of Streaming [209] Neo4j Inc. Neo4j - Source Code. [Online; accessed Graphs. In Proceedings of the Fourteenth EuroSys 24-April-2020], 2020. Conference 2019, EuroSys ’19, pages 25:1–25:16, New [210] Neo4j, Inc. Neo4j GraphQL to Cypher - Source Code. York, NY, USA, 2019. ACM. [Online, GitHub; accessed 24-April-2020], April 2020. [186] M. Mariappan and K. Vora. GraphBolt - Source Code. [211] M. Newman. Networks: An Introduction. Oxford [Online, GitHub; accessed 24-April-2020], 2020. University Press, Inc., New York, NY, USA, 2010. [187] C. Martella, D. Logothetis, A. Loukas, and G. Siganos. [212] Objectivity. Objectivity/DB. [Online; accessed Spinner: Scalable graph partitioning in the cloud. In 24-April-2020], 2016. 2017 IEEE 33rd International Conference on Data [213] Ontotext. GraphDB | The Best RDF Database for Engineering (ICDE), pages 1083–1094. Ieee, 2017. Knowledge Graphs. [Online; accessed 24-April-2020], [188] N. Martinez-Bazan, S. Gomez-Villamor, and 2020. F. Escale-Claveras. DEX: A high-performance graph [214] OpenLink Software. Virtuoso Open-Source Edition - database management system. In 2011 IEEE 27th Source Code. [Online, GitHub; accessed 24-April-2020], International Conference on Data Engineering 2020. Workshops, pages 124–127. IEEE, 2011. [215] Optiver. Ruruki - In-Memory Directed Property Graph. [189] K. K. Matam, G. Koo, H. Zha, H.-W. Tseng, and [Online; accessed 24-April-2020], 2016. M. Annavaram. GraphSSD: graph semantics aware SSD. [216] Oracle. Spatial and Graph features in Oracle Database. In Proceedings of the 46th International Symposium on [Online; accessed 24-April-2020], 2020. Computer Architecture, pages 116–128, 2019. [217] OrientDB LTD. OrientDB - Source Code. [Online; [190] C. Mayer, M. A. Tariq, R. Mayer, and K. Rothermel. accessed 24-April-2020], 2020. GraphH - Source Code. [Online, GitHub; accessed [218] L. Page, S. Brin, R. Motwani, and T. Winograd. The 24-April-2020], 2016. PageRank Citation Ranking: Bringing Order to the Web. [191] C. Mayer, M. A. Tariq, R. Mayer, and K. Rothermel. Technical Report 1999-66, Stanford InfoLab, 1999. GrapH: Traffic-aware graph processing. IEEE [219] B. Panda, J. S. Herbach, S. Basu, and R. J. Bayardo. Transactions on Parallel and Distributed Systems, Planet: massively parallel learning of tree ensembles with 29(6):1289–1302, 2018. MapReduce. Proceedings of the VLDB Endowment, [192] R. Meusel, S. Vigna, O. Lehmberg, and C. Bizer. The 2(2):1426–1437, 2009. graph structure in the web–analyzed on different [220] J. Paredaens, P. Peelman, and L. Tanca. G-Log: A aggregation levels. The Journal of Web Science, 1, 2015. graph-based query language. IEEE Transactions on [193] S. M. Meyer, J. Degener, J. Giannandrea, and Knowledge and Data Engineering, 7(3):436–453, 1995. B. Michener. Optimizing Schema-last Tuple-store Queries [221] J. R. G. Paz. Introduction to Azure Cosmos DB. In in Graphd. In Proceedings of the 2010 ACM SIGMOD Microsoft Azure Cosmos DB Revealed, pages 1–23. International Conference on Management of Data, Springer, 2018. SIGMOD ’10, pages 1047–1056, New York, NY, USA, [222] R. Pearce, M. Gokhale, and N. M. Amato. Scaling 2010. ACM. techniques for massive scale-free graphs in distributed [194] A. Mhedhbi, P. Gupta, S. Khaliq, and S. Salihoglu. A+ (external) memory. In 2013 IEEE 27th International Indexes: Lightweight and Highly Flexible Adjacency Lists Symposium on Parallel and Distributed Processing, for Graph Database Management Systems. arXiv pages 825–836. IEEE, 2013. preprint arXiv:2004.00130, 2020. [223] R. Pearce, M. Gokhale, and N. M. Amato. HavocGT - [195] A. Mhedhbi and S. Salihoglu. Optimizing Subgraph Source Code. [Online, GitHub; accessed 24-April-2020], Queries by Combining Binary and Worst-Case Optimal 2019. Joins. Proc. VLDB Endow., 12(11):1692–1704, July 2019. [224] J. Pérez, M. Arenas, and C. Gutierrez. Semantics and [196] Y. Miao, W. Han, K. Li, M. Wu, F. Yang, L. Zhou, complexity of SPARQL. ACM Transactions on Database V. Prabhakaran, E. Chen, and W. Chen. Immortalgraph: Systems (TODS), 34(3):1–45, 2009. A system for storage and analysis of temporal graphs. [225] Y. Perez, R. Sosič, A. Banerjee, R. Puttagunta, ACM Transactions on Storage (TOS), 11(3):1–34, 2015. M. Raison, P. Shah, and J. Leskovec. Ringo: Interactive [197] O. Michail. An introduction to temporal graphs: An Graph Analytics on Big-Memory Machines. In algorithmic perspective. Internet Mathematics, Proceedings of the 2015 ACM SIGMOD International 12(4):239–280, 2016. Conference on Management of Data, SIGMOD ’15, [198] Microsoft. Graph Engine (GE): Serving Big Graphs in pages 1105–1110, New York, NY, USA, 2015. ACM. Real-time. [Online; accessed 24-April-2020], 2017. [226] Y. Perez, R. Sosič, A. Banerjee, R. Puttagunta, [199] J. J. Miller. Graph database applications and concepts M. Raison, P. Shah, and J. Leskovec. Ringo - Source with neo4j. Proceedings of the Southern Association for Code. [Online, GitHub; accessed 24-April-2020], 2016. Information Systems Conference, Atlanta, GA, USA, [227] F. Petroni, L. Querzoni, K. Daudjee, S. Kamali, and 2324(36), 2013. G. Iacoboni. Hdrf: Stream-based partitioning for [200] R. K. Mishra and S. R. Raman. GraphFrames. In power-law graphs. In Proceedings of the 24th ACM PySpark SQL Recipes, pages 297–315. Springer, 2019. International on Conference on Information and [201] M. H. Mofrad, R. Melhem, and M. Hammoud. Revolver: Knowledge Management, pages 243–252, 2015. vertex-centric graph partitioning using reinforcement [228] C. Pfluke. A history of the Five Eyes Alliance: Possibility learning. In 2018 IEEE 11th International Conference for reform and additions: A history of the Five Eyes on Cloud Computing (CLOUD), pages 818–821. IEEE, Alliance: Possibility for reform and additions. 2018. Comparative Strategy, 38(4):302–315, 2019. [202] J. I. Munro, Y. Nekrich, and J. S. Vitter. Dynamic Data [229] S. Plantikow. Towards an International Standard for the Structures for Document Collections and Graphs. In GQL Graph Query Language. [Online; accessed ACM Symposium on Principles of Database Systems 24-April-2020], March 2019. (PODS), pages 277–289, 2015. [230] J. Pokorn`y.Graph databases: their power and [203] D. G. Murray, F. McSherry, R. Isaacs, M. Isard, limitations. In IFIP International Conference on P. Barham, and M. Abadi. Naiad: A Timely Dataflow Computer Information Systems and Industrial System. In Proceedings of the Twenty-Fourth ACM Management, pages 58–69. Springer, 2015. Symposium on Operating Systems Principles, SOSP ’13, [231] M. M. Rathore, A. Ahmad, A. Paul, and U. K. pages 439–455, New York, NY, USA, 2013. ACM. Thikshaja. Exploiting real-time big data to empower [204] G. Navarro. Compact data structures: A practical smart transportation using big graphs. In 2016 IEEE approach. Cambridge University Press, 2016. Region 10 Symposium (TENSYMP), pages 135–139. IEEE, 2016. Knowledge and Data Engineering, 30(1):29–42, 2017. [232] M. Research. Naiad - Source Code. [Online, GitHub; [256] X. Shi, X. Luo, J. Liang, P. Zhao, S. Di, B. He, and accessed 24-April-2020], 2018. H. Jin. Frog - Source Code. [Online, GitHub; accessed [233] C. D. Rickett, U.-U. Haus, J. Maltby, and K. J. 24-April-2020], 2018. Maschhoff. Loading and querying a trillion rdf triples [257] X. Shi, Z. Zheng, Y. Zhou, H. Jin, L. He, B. Liu, and with cray graph engine on the cray xc. Cray User Group, Q.-S. Hua. Graph processing on GPUs: A survey. ACM 2018. Computing Surveys (CSUR), 50(6):1–35, 2018. [234] I. Robinson, J. Webber, and E. Eifrem. Graph [258] D. Shimpi and S. Chaudhari. An overview of graph Databases. O’Reilly Media, Inc., 2013. databases. In Int. Conf. on Recent Trends in [235] I. Robinson, J. Webber, and E. Eifrem. Graph databases: Information Technology and Computer Science, pages new opportunities for connected data. " O’Reilly Media, 16–22, 2012. Inc.", 2015. [259] J. Shun and G. E. Blelloch. Ligra: a lightweight graph [236] M. A. Rodriguez. The gremlin graph traversal machine processing framework for shared memory. In Proceedings and language (invited talk). In Proceedings of the 15th of the 18th ACM SIGPLAN symposium on Principles Symposium on Database Programming Languages, and practice of parallel programming, pages 135–146, pages 1–10, 2015. 2013. [237] M. A. Rodriguez. The Gremlin Graph Traversal Machine [260] J. Shun and G. E. Blelloch. GraphH - Source Code. and Language. CoRR, abs/1508.03843, 2015. [Online, GitHub; accessed 24-April-2020], 2020. [238] A. Roy, L. Bindschaedler, J. Malicevic, and [261] N. B. Silva, R. Tsang, G. D. Cavalcanti, and J. Tsang. A W. Zwaenepoel. Chaos: Scale-out Graph Processing from graph-based friend recommendation system using genetic Secondary Storage. In Proceedings of the 25th algorithm. In IEEE congress on evolutionary Symposium on Operating Systems Principles, SOSP ’15, computation, pages 1–7. IEEE, 2010. pages 410–424, New York, NY, USA, 2015. ACM. [262] Y. Simmhan, A. Kumbhare, C. Wickramaarachchi, [239] A. Roy, L. Bindschaedler, J. Malicevic, and S. Nagarkar, S. Ravi, C. Raghavendra, and V. Prasanna. W. Zwaenepoel. Chaos - Source Code. [Online, GitHub; GoFFish: A sub-graph centric framework for large-scale accessed 24-April-2020], 2016. graph analytics. In European Conference on Parallel [240] A. Roy, I. Mihailovic, and W. Zwaenepoel. X-Stream: Processing, pages 451–462. Springer, 2014. Edge-centric Graph Processing Using Streaming [263] Y. Simmhan, A. Kumbhare, C. Wickramaarachchi, Partitions. In Proceedings of the Twenty-Fourth ACM S. Nagarkar, S. Ravi, C. Raghavendra, and V. Prasanna. Symposium on Operating Systems Principles, SOSP ’13, GoFFish - Source Code. [Online, GitHub; accessed pages 472–488, New York, NY, USA, 2013. ACM. 24-April-2020], 2017. [241] A. Roy, I. Mihailovic, and W. Zwaenepoel. X-Stream - [264] G. M. Slota, K. Madduri, and S. Rajamanickam. PuLP: Source Code. [Online, GitHub; accessed 24-April-2020], Scalable multi-objective multi-constraint partitioning for 2015. small-world networks. In 2014 IEEE International [242] M. Rudolf, M. Paradies, C. Bornhövd, and W. Lehner. Conference on Big Data (Big Data), pages 481–490. The graph story of the SAP HANA database. IEEE, 2014. Datenbanksysteme für Business, Technologie und Web [265] A. Smola and S. Narayanamurthy. An architecture for (BTW) 2037, 2013. parallel topic models. Proceedings of the VLDB [243] S. Sahu, A. Mhedhbi, S. Salihoglu, J. Lin, and M. T. Endowment, 3(1-2):703–710, 2010. Özsu. The Ubiquity of Large Graphs and Surprising [266] N. M. Soudani, A. Fatemi, and M. Nematbakhsh. An Challenges of Graph Processing. Proc. VLDB Endow., investigation of big graph partitioning methods for 11(4):420–431, Dec. 2017. distribution of graphs in vertex-centric systems. [244] S. Sakr, F. M. Orakzai, I. Abdelaziz, and Z. Khayyat. Distributed and Parallel Databases, pages 1–29, 2019. Large-Scale Graph Processing Using Apache Giraph. [267] Sparsity Technologies. Scalable high-performance graph Springer, 2016. database. [Online, GitHub; accessed 24-April-2020], 2015. [245] S. Salihoglu and J. Widom. GPS - Source Code. [Online, [268] I. Stanton and G. Kliot. Streaming graph partitioning for Stanford; accessed 24-April-2020], 2013. large distributed graphs. In Proceedings of the 18th [246] S. Salihoglu and J. Widom. GPS: A Graph Processing ACM SIGKDD international conference on Knowledge System. In Proceedings of the 25th International discovery and data mining, pages 1222–1230, 2012. Conference on Scientific and Statistical Database [269] Stardog. Enterprise Knowledge Graph platform. [Online; Management, pages 1–12, 2013. accessed 24-April-2020], 2020. [247] H. Samet. Foundations of multidimensional and metric [270] P. Stutz, A. Bernstein, and W. Cohen. Signal/collect: data structures. Morgan Kaufmann, 2006. graph algorithms for the (semantic) web. In [248] SAP SE. SAP HANA Graph Academy - Source Code. International Semantic Web Conference, pages 764–780. [Online, GitHub; accessed 24-April-2020], 2018. Springer, 2010. [249] A. Schätzle, M. Przyjaciel-Zablocki, T. Berberich, and [271] P. Stutz, M. Verman, L. Fischer, and A. Bernstein. G. Lausen. S2X: graph-parallel querying of RDF with TripleRush: a fast and scalable triple store. [Online; GraphX. In Biomedical Data Management and Graph accessed 24-April-2020], 2013. Online Querying, pages 155–168. Springer, 2015. [272] N. Sundaram, N. R. Satish, M. M. A. Patwary, S. R. [250] Securities and Exchange Commission. One billion. Dulloor, S. G. Vadlamudi, D. Das, and P. Dubey. [Online; accessed 05-May-2020], 2019. GraphMat: High performance graph analytics made [251] R. Sedgewick and K. Wayne. Algorithms. Addison-wesley productive. arXiv preprint arXiv:1503.07241, 2015. professional, 2011. [273] N. Sundaram, N. R. Satish, M. M. A. Patwary, S. R. [252] D. Sengupta, N. Sundaram, X. Zhu, T. L. Willke, Dulloor, S. G. Vadlamudi, D. Das, and P. Dubey. J. Young, M. Wolf, and K. Schwan. GraphIn: An online GraphMat - Source Code. [Online, GitHub; accessed high performance incremental graph processing 24-April-2020], 2017. framework. In European Conference on Parallel [274] V. Surveillances. The epidemiological characteristics of Processing, pages 319–333. Springer, 2016. an outbreak of 2019 novel coronavirus diseases [253] L. Sheng, Z. M. Ozsoyoglu, and G. Ozsoyoglu. A graph (COVID-19)—China, 2020. China CDC Weekly, query language and its query processing. In Proceedings 2(8):113–122, 2020. 15th International Conference on Data Engineering [275] Systap. Blazegraph High Performance Graph Database. (Cat. No. 99CB36337), pages 572–581. IEEE, 1999. [Online; accessed 24-April-2020], 2020. [254] X. Shi, B. Cui, Y. Shao, and Y. Tong. Tornado: A [276] A. S. Teixeira, P. T. Monteiro, J. A. Carriço, System For Real-Time Iterative Analysis Over Evolving M. Ramirez, and A. P. Francisco. Spanning edge Data. In Proceedings of the 2016 International betweenness. In Workshop on mining and learning with Conference on Management of Data, SIGMOD ’16, graphs, volume 24, pages 27–31, 2013. pages 417–430, New York, NY, USA, 2016. ACM. [277] C. Tesoriero. Getting started with OrientDB. Packt [255] X. Shi, X. Luo, J. Liang, P. Zhao, S. Di, B. He, and Publishing Ltd, 2013. H. Jin. Frog: Asynchronous graph processing on GPU [278] F. C. Thomas Schmidts, Jan Steemann. ArangoDB - with hybrid coloring model. IEEE Transactions on Source Code. [Online, GitHub; accessed 24-April-2020], 2020. [279] Y. Tian, A. Balmin, S. A. Corsten, S. Tatikonda, and Management Experiences and Systems, GRADES ’13, J. McPherson. From "Think Like a Vertex" to "Think pages 2:1–2:6, New York, NY, USA, 2013. ACM. Like a Graph". Proc. VLDB Endow., 7(3):193–204, Nov. [301] D. Yan, Y. Huang, M. Liu, H. Chen, J. Cheng, H. Wu, 2013. and C. Zhang. Graphd: Distributed vertex-centric graph [280] TigerGraph. The World’s Fastest and Most Scalable processing beyond the memory limit. IEEE Transactions Graph Platform. [Online; accessed 24-April-2020], 2020. on Parallel and Distributed Systems, 29(1):99–114, 2017. [281] C. Tsourakakis, C. Gkantsidis, B. Radunovic, and [302] K. Yang and L. Toni. Graph-based recommendation M. Vojnovic. Fennel: Streaming graph partitioning for system. In 2018 IEEE Global Conference on Signal and massive scale graphs. In Proceedings of the 7th ACM Information Processing (GlobalSIP), pages 798–802. international conference on Web search and data IEEE, 2018. mining, pages 333–342, 2014. [303] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, [282] Twitter, Inc. Quarterly results. [Online; accessed and I. Stoica. Spark: Cluster Computing with Working 05-May-2020], 2020. Sets. In Proceedings of the 2Nd USENIX Conference on [283] UC Berkeley, MIT, and Databricks. GraphFrames - Hot Topics in Cloud Computing, HotCloud’10, pages Source Code. [Online, GitHub; accessed 24-April-2020], 10–10, Berkeley, CA, USA, 2010. USENIX Association. 2020. [304] D. Zerbino and E. Birney. Velvet: algorithms for de novo [284] C. Unsalan and B. Sirmacek. Road network detection short read assembly using de Bruijn graphs. Genome using probabilistic and graph theoretical methods. IEEE Res, 18:821, 2008. Transactions on Geoscience and Remote Sensing, [305] K. Zhang, R. Chen, and H. Chen. NUMA-aware 50(11):4441–4453, 2012. graph-structured analytics. In Proceedings of the 20th [285] A. Vaikuntam and V. K. Perumal. Evaluation of ACM SIGPLAN Symposium on Principles and Practice Contemporary Graph Databases. In Proceedings of the of Parallel Programming, pages 183–193, 2015. 7th ACM India Computing Conference, COMPUTE ’14, [306] K. Zhang, R. Chen, and H. Chen. Polymer - Source pages 6:1–6:10, New York, NY, USA, 2014. ACM. Code. [Online, GitHub; accessed 24-April-2020], 2018. [286] O. van Rest, S. Hong, J. Kim, X. Meng, and H. Chafi. [307] Y. Zhang, X. Liao, H. Jin, L. Gu, and B. B. Zhou. Pgql: a property graph query language. In Proceedings of FBSGraph: Accelerating asynchronous graph processing the Fourth International Workshop on Graph Data via forward and backward sweeping. IEEE Transactions Management Experiences and Systems, pages 1–6, 2016. on Knowledge and Data Engineering, 30(5):895–907, [287] L. Vaquero, F. Cuadrado, D. Logothetis, and C. Martella. 2017. xDGP: A dynamic graph processing system with adaptive [308] H. Zhao, Q. Yao, J. Li, Y. Song, and D. L. Lee. partitioning. arXiv preprint arXiv:1309.1049, 2013. Meta-graph based recommendation fusion over [288] VESoft Inc. Negula Graph - Source Code. [Online; heterogeneous information networks. In Proceedings of accessed 24-April-2020], 2020. the 23rd ACM SIGKDD International Conference on [289] K. Vora, R. Gupta, and G. Xu. KickStarter: Fast and Knowledge Discovery and Data Mining, pages 635–644, Accurate Computations on Streaming Graphs via 2017. Trimmed Approximations. In Proceedings of the [309] D. Zheng, D. Mhembere, R. Burns, J. Vogelstein, C. E. Twenty-Second International Conference on Priebe, and A. S. Szalay. FlashGraph: Processing Architectural Support for Programming Languages and billion-node graphs on an array of commodity SSDs. In Operating Systems, ASPLOS ’17, pages 237–251, New 13th {USENIX} Conference on File and Storage York, NY, USA, 2017. ACM. Technologies ({FAST} 15), pages 45–58, 2015. [290] G. Wang, W. Xie, A. J. Demers, and J. Gehrke. [310] Zheng, Da and Mhembere, Disa and Burns, Randal and Asynchronous Large-Scale Graph Processing Made Easy. Vogelstein, Joshua and Priebe, Carey E and Szalay, In CIDR, volume 13, pages 3–6, 2013. Alexander S. FlashGraph - Source Code. [Online, [291] Y. Wang, A. Davidson, Y. Pan, Y. Wu, A. Riffel, and GitHub; accessed 24-April-2020], 2014. J. D. Owens. Gunrock: A high-performance graph [311] X. Zhu, W. Chen, W. Zheng, and X. Ma. Gemini: A processing library on the GPU. In Proceedings of the computation-centric distributed graph processing system. 21st ACM SIGPLAN Symposium on Principles and In 12th {USENIX} Symposium on Operating Systems Practice of Parallel Programming, pages 1–12, 2016. Design and Implementation ({OSDI} 16), pages [292] Y. Wang, A. Davidson, Y. Pan, Y. Wu, A. Riffel, and 301–316, 2016. J. D. Owens. Gunrock - Source Code. [Online, GitHub; [312] X. Zhu and Z. Ghahramani. Learning from labeled and accessed 24-April-2020], 2020. unlabeled data with label propagation. [Online; accessed [293] Y. Wang, Y. Pan, A. Davidson, Y. Wu, C. Yang, 24-April-2020], 2002. L. Wang, M. Osama, C. Yuan, W. Liu, A. T. Riffel, et al. Gunrock: GPU graph analytics. ACM Transactions on Parallel Computing (TOPC), 4(1):1–49, 2017. [294] J. Webber. A Programmatic Introduction to Neo4J. In Proceedings of the 3rd Annual Conference on Systems, Programming, and Applications: Software for Humanity, SPLASH ’12, pages 217–218, New York, NY, USA, 2012. ACM. [295] WhatsApp Inc. One billion. [Online; accessed 05-May-2020], 2016. [296] B. Wilkinson and M. Allen. Parallel programming. Pearson India, 2004. [297] M. Winter, D. Mlakar, R. Zayer, H.-P. Seidel, and M. Steinberger. faimGraph: high performance management of fully-dynamic graphs under tight memory constraints on the GPU. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 754–766. IEEE, 2018. [298] P. T. Wood. Query languages for graph databases. ACM Sigmod Record, 41(1):50–60, 2012. [299] W. Xiao, J. Xue, Y. Miao, Z. Li, C. Chen, M. Wu, W. Li, and L. Zhou. Tux2: Distributed Graph Computation for Machine Learning. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17), pages 669–682, 2017. [300] R. S. Xin, J. E. Gonzalez, M. J. Franklin, and I. Stoica. GraphX: A Resilient Distributed Graph System on Spark. In First International Workshop on Graph Data