A Unified Metamodel for NoSQL and Relational Databases∗,†
A Preprint
Carlos J. Fernández Candel Diego Sevilla Ruiz Jesús J. García-Molina Faculty of Computer Science Faculty of Computer Science Faculty of Computer Science University of Murcia University of Murcia University of Murcia Murcia, Spain Murcia, Spain Murcia, Spain [email protected] [email protected] [email protected]
May 18, 2021
Abstract The Database field is undergoing significant changes. Although relational systems are still predominant, the interest in NoSQL systems is continuously increasing. In this scenario, polyglot persistence is envisioned as the database architecture to be prevalent in the future. Therefore, database tools and systems are evolving to support several data models. Multi-model database tools normally use a generic or unified metamodel to represent schemas of the data model that they support. Such metamodels facilitate developing database utilities, as they can be built on a common representation. Also, the number of mappings required to migrate databases from a data model to another is reduced, and integrability is favored. In this paper, we present the U-Schema unified metamodel able to represent logical schemas for the four most popular NoSQL paradigms (columnar, document, key-value, and graph) as well as relational schemas. We will formally define the mappings between U-Schema and the data model defined for each database paradigm. How these mappings have been implemented and validated will be discussed, and some applications of U-Schema will be shown. To achieve flexibility to respond to data changes, most of NoSQL systems are “schema-on-write,” and the declaration of schemas is not required. Such an absence of schema declaration makes structural variability possible, i.e., stored data of the same entity type can have different structure. Moreover, data relationships supported by each data model are different; For example, document stores have aggregate objects but not relationship types, whereas graph stores offer the opposite. Through the paper, we will show how all these issues have been tackled in our approach. As far as we know, no proposal exists in the literature of a unified metamodel for relational and the NoSQL paradigms which describes how each individual data model is integrated and mapped. Our arXiv:2105.06494v2 [cs.DB] 17 May 2021 metamodel goes beyond the existing proposals by distinguishing entity types and relationship types, representing aggregation and reference relationships, and including the notion of structural variability. Our contributions also include developing schema extraction strategies for schemaless systems of each NoSQL data model, and tackling performance and scalability in the implementation for each store. Keywords Unified Metamodel · NoSQL Databases · Schemaless · Schema Inference · Model-Driven Engineering
∗This work has been funded by the Spanish Ministry of Science, Innovation and Universities (project grant TIN2017-86853-P). †Formatted for arXiv.org. A Unified Metamodel for NoSQL and Relational Databases A Preprint
1 Introduction be considered for database tooling, i.e., tools should support multiple data models. With the advent of modern data-intensive applications Data models determine how data can be organized N (e.g., Big Data, social networks, or IoT), NoSQL ( ot and manipulated in databases. They are applied to a o SQL nly ) database systems emerged to overcome particular domain by defining schemas that express the limitations that relational systems evidenced to the structure and constraints for the domain entities. support such applications, namely, scalability, avail- Such information, provided by schemas, is necessary ability, flexibility, and ability to represent complex ob- to implement many database tools. However, most jects. These systems are classified in several database NoSQL systems are schemaless (a.k.a “schema-on- paradigms, but commonly the term NoSQL refers to read”), that is, data can be directly stored without columnar key-value document four of them: , , , and requiring the previous declaration of a schema. This graph systems. NoSQL systems of the same paradigm feature is motivated by the fact that the pace of data can have significant differences in features and in the structure changes is considerably faster in the new structure of the data. This is because there is no data-intensive applications. Being schemaless does specification, standard, or theory that establishes the not mean the absence of a schema, but that the schema data model of a particular paradigm. Therefore, we information is implicit in data and code. Therefore, will assume here the data model of the most popu- the schemas implicit in NoSQL stores must be reverse lar stores of each category. Table 1 shows the main engineered in order to build a cohesive set of util- features of the four mentioned NoSQL paradigms. ities for NoSQL databases such as schema viewers, Over last years, as the popularity of NoSQL systems 3 query optimizers, or code generators. This reverse increased, polyglot persistence (a new term coined engineering process must tackle the fact that “schema- for heterogeneous database systems) has been gain- on-read” systems can store data with different struc- ing acceptance as the data architecture of the future: ture even belonging to the same database entity type applications using the set of databases that better (and relationship type in graphs), i.e., each entity or fit their needs. Today, relational databases are still relationship type can have one or more structural vari- clearly the most used by a wide margin, but most pop- ations. Recently, several NoSQL schema extraction ular relational systems are evolving to support NoSQL 4 approaches have been published [43, 36, 52], and some features. Two facts have motivated this interest in data modeling tools such as those mentioned above polyglot persistence [48, 42]: (i) the complexity and provide some kind of reverse engineering functionality. variety of data to be managed by software systems, When building multi-model database tools, the def- and (ii) a single type of database system does not inition of a generic, universal, or unified metamodel fit all the needs of an increasing number of systems can provide some benefits [20, 10, 7, 35, 51]. It of- (e.g., learning management systems, online retail sys- fers a unified view of different data models, so that tems, or social networks.) their schemas will be represented in a uniform way. The successful adoption of NoSQL requires database This uniformity facilitates building generic tools that tools similar to those available for relational systems. are database-independent. With the predominance This entails to investigate how common database util- of relational databases, the interest in multi-model ities can be available for NoSQL systems. In addition, tools declined, and little attention has been paid to these tools should be built taking into account the the definition of unified metamodels. A remarkable expected predominance of polyglot persistence. Thus, proposal is the DB-Main approach [20, 34] that de- they should support widespread relational databases fined the GER generic metamodel based on the EER as well as NoSQL databases. In the case of data (Extended Entity Relationship) data model. More modeling tools, the shift towards multi-model solu- recently, some universal metamodels [7, 35] have been tions is evident: the most popular relational modeling created in the context of “Model Management” [10, 9]. tools are being extended to integrate NoSQL stores 5 6 With the emergence of NoSQL systems, some unified (e.g. ErWin and ER/Studio provide functionality metamodels have been proposed to have a uniform for MongoDB) and new tools supporting a number access to data [7, 5], and the idea of an unified meta- of relational and NoSQL databases have appeared 7 model for data modeling tools was outlined in [51]. (e.g. Hackolade ). This multi-model nature should In the past years, we defined a reverse engineer- 3Four of the top 10 being NoSQL systems in the DB ing strategy to infer logical schemas from document engines ranking (https://db-engines.com/en/ranking) NoSQL databases [43], and have presented approaches as of January 2021: MongoDB (5th), Redis (7th), Elastic- to visualize inferred schemas [30] and automatically search (8th), and Cassandra (10th). generate object-document mappers code [32] from 4Top-8 systems in the DB engines ranking (https: schemas. Currently, we are tackling the definition of //db-engines.com/en/ranking are multi-model. a synthetic data generation approach [33], and the 5http://erwin.com/products/erwin-data-modeler. extraction of implicit physical schemas from NoSQL 6https://www.idera.com/er-studio-enterprise- data [38]. As our intention is to build multi-model architecture-solutions. database utilities, we have tackled the definition of a 7https://hackolade.com/. unified metamodel named U-Schema, which is capable
2 A Unified Metamodel for NoSQL and Relational Databases A Preprint
NoSQL System Types Type Data structure Appropriateness Database Systems Key-value Associative array of key- Frequent small read and writes with simple Redis, Memcache value pairs data Columnar Tables of rows with varying High performance, availability, scalability, HBase, Cassandra columns. Column-based and large volumes of data for OLAP queries physical storage Document JSON-like document collec- Nested Objects, structural variation, and MongoDB, Couchbase tions large volumes of heterogeneous data Graphs Data connected in a graph Highly connected objects, references prevail Neo4J, OrientDB over nested objects
Table 1: Types of NoSQL systems and some example implementations.
of representing schemas for the four most common Our strategy also takes into account scalability and NoSQL data models, as well as the relational model. performance, using MapReduce processing on the In this paper, we present U-Schema jointly with a native data. data model for each NoSQL paradigm, and the map- 4. We provide insights on how U-Schema can ease the ping from those data models to U-Schema (forward implementation of database utilities in multi-model mappings), and from U-Schema to the data models and multi-database environments such as database (reverse mappings). Common strategies are defined migration, schema queries, data generation for test- to implement and validate the mappings. Several ing, query optimization, and schema visualization. applications of U-Schema are then commented. 5. We have defined the U-Schema metamodel with research contributions The of this work are as fol- the Ecore metamodel, which is the central element lows: of the Eclipse Modeling Framework (EMF) [47], a 1. To our knowledge, we present the first unified meta- widely used open-source platform to develop Model- model able to represent logical schemas both for Driven Software Engineering (MDE) solutions [11]. the four most common kinds of NoSQL systems Thus, the proposal is open to be used and incorpo- and relational systems. Two salient features of our rated in any future database tool development. proposal are: (i) U-Schema includes the notion of The rest of this paper is organized as follows. Next structural variation for entity and relationship types, section will present the U-Schema data model. Then, as most NoSQL systems are schemaless, and (ii) un- Section 3 will describe the common process devised like other proposals, the four kinds of relationships to reverse engineer implicit schemas from data, intro- between entities that are typical in logical data duce a running example database, and will explain modeling are supported by U-Schema: aggregation, the common strategy applied to validate and assert references, graph relationships, and generalization. the performance of the schema extraction algorithms. Capturing structural variability allows us to accu- Next five sections will be devoted to define a logical rately describe the structure of the stored data. data model, formally specify the mapping between 2. Defining the forward and reverse mappings between U-Schema and the data model defined, and show the U-Schema and the data models that it integrates, implementation and validation of the corresponding we have established the notion of canonical map- forward mapping. Once presented the unified meta- ping in which there is a natural correspondence model and the mappings, we will discuss how they between each element of a data model and elements can be applied in common database tasks. Finally, we of U-Schema. For each data model, its canonical will contrast our proposal with related works, draw mapping has been formally defined, as well as the some conclusions, and outline further work. reverse mapping for characteristics not present in the considered data model (e.g. graph systems do 2 The U-Schema Unified Data Model not support aggregate relationships, or structural variation is not possible in relational tables.) 2.1 Logical Modeling Concepts in U-Schema 3. A common strategy is proposed to extract uni- A data model provides a set of concepts to specify fied schemas from databases. This strategy has the structure and constraints of a database type, and been applied to implement canonical mappings for a schema results of applying a concrete data model each paradigm integrated in U-Schema. As far on a domain or problem. A schema is therefore an as we know, all the schema extraction proposals instance of a data model. Given a particular data for NoSQL stores, only consider one kind of store, model, textual and graphical languages can be defined normally document-based [36, 52] or graph [17]. to express schemas.
3 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Data models (and therefore schemas) can be defined and it would be an aggregation or reference if it is at different levels of abstraction. Typically, they are associated to an entity type ε0 whose objects are, re- classified in three categories: conceptual, logical, and spectively, embedded in or referenced from objects of ε physical. Conceptual schemas represent the domain of the entity type ε, to which the Pi property belongs. an application in a platform-independent way. Logical Keys are a special kind of attribute able to record schemas describe data structures and constraints, but values used as identifiers of instances of entity types. providing physical independence. Finally, Physical Graph data models include relationship types in ad- schemas include all details needed to implement a dition to entity types. While nodes are instances of logical schema on a specific database system. entity types, arcs are instances of relationship types, At the logical or physical level, a unified or generic which can have attributes. Hereafter, we will use data model can be defined to integrate concepts from “schema type” to gather both entity and relationship several data models with the purpose of offering a types. uniform representation. When using a unified model Schemas play a similar role to types in programming for n data models, instead of managing n × (n − 1) languages. Given a database schema S, only data mappings (each data model with the others), only conforming to S can be stored in the database. There- n + n mappings are needed (between the unified and fore, all data of an entity type E (resp. a relationship each of the integrated data models in both directions.) type R) will have the same structure, that defined for U-Schema is a unified logical model that integrates the E (resp. R) in S. In absence of schema declarations, concepts and rules of both the relational model and however, data of E and R can have different struc- the four most common NoSQL data models: colum- ture, that is, E and R will have one or more structural nar, document, graph, and key-value. While the re- variations. lational model is a well-defined data model, there is A structural variation of a schema type ε is formed no specification, standard, or theory that establishes by a set of properties Qε ⊂ P ε, and each pair of the data model of a particular NoSQL paradigm. In variations of ε differ, at least, in one property. The set fact, NoSQL systems of the same kind can have sig- P ε is therefore the union of the sets of properties of nificant differences in features and in the structure of each of its variations. The set P ε is commonly called the data. We have therefore defined a logical model union schema of a schema type in a schemaless system. for each NoSQL category by abstracting from the The properties of an schema type can therefore be logical/physical data organization of the most popu- classified as common or specific, depending on whether lar stores of each category. This section will present they are present in all the variations, or in a subset of U-Schema, while the logical model defined for each them. It is worth noting that specifying a schema type particular NoSQL paradigm will be presented in the as the union schema, the information on its structural section devoted to describe how that data model has variability is lost. We have considered convenient been integrated into U-Schema. to record this structural variability in data models U-Schema includes the basic concepts traditionally defined for NoSQL databases, and therefore the notion used to create logical schemas, which are part of of “variation” will be part of U-Schema. well-known formalisms such as Entity-Relationship When structural variations are considered, entity and (ER) [44] and UML Class Models [41]: entity type, relationship types can be defined as follows. simple and multivalued attributes, key attribute, and three kinds of relationships between entity types: ag- • An entity type has a name and is formed by a set gregation, reference, and inheritance. Also, U-Schema of structural variations. incorporates some additional concepts, such as rela- • A relationship type (only for graph stores) has tionship types (as they are considered in the graph a name, is formed by a set of structural variations, data model [3]), and structural variations of entity and and refers to both a source and a target entity type. relationship types. Before presenting the U-Schema • A structural variation is formed by a set of metamodel, we will define all these concepts. Not all named properties. The kind of properties depend concepts will be present in all of the data models sup- on the data model, and can be: attributes, keys, ported by U-Schema. For example, the relationship aggregates, and references. type is exclusive of the graph model, but conversely, it does not define aggregation. Table 2 shows the correspondence between concepts of In data models, an entity type ε is normally char- each of the considered database kinds and the logical ε ε modeling concepts that we will use in U-Schema. We acterized by a set P = {Pi }, i = 1 . . . n of named properties. Properties can be of several kinds depend- will use these concepts to abstract a logical data model ing on the type of the object or value a property can for the most popular systems of each NoSQL paradigm. hold. Three common kinds are: attributes, aggrega- These models will be presented in sections 4 to 7. P ε tions, and references. Given a property i , it would 2.2 The U-Schema Metamodel be an attribute if it can take values of scalar type (e.g. Number) or structured type (i.e. Array or Set), Data models are commonly expressed formally in form of metamodels. A metamodel is a model that de-
4 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Logical modeling Relational Columnar Document Graph Key/Value concepts Schema Schema Database or Database Graph Database or Keyspace Namespace Entity Type Table Table with Collection and Node label Multirow column families nested object entities Relationship Type Relationship Ta- N/A N/A Relation type N/A ble Structural Variation Table (only one Rows with Documents with Same label with Multirow variation) different different different entities with structure within structure in a structure different column families collection structure Key Primary key Row key Document key N/A Pair key Reference Foreign key Join between Join between N/A Join between tables documents pairs Aggregation N/A Nested object Nested object N/A Nested object Attribute Column Column Document Node and Pair Value property Relation property Primitive Types Scalar Types Scalar Types Scalar Types Scalar Types Scalar Types Structured Types N/A Collections Collections Array Collections
Table 2: Mapping between logical modeling concepts and NoSQL/Relational Database Systems.
scribes a set of concepts and relationships between relational systems. An aggregation has a name, a them. That description determines the structure of cardinality (upper and lower bound), and refers to models that can be instantiated from the metamodel the structural variation it aggregates, or to a list of elements, i.e. a metamodel is a model of a model. variations, if the aggregated object is an heterogeneous Object-oriented conceptual modeling is usually ap- collection. plied to create metamodels: concepts and their prop- Unlike aggregations, references refer to an entity type erties are modeled with classes, and reference, ag- (via refsTo), and one or more attributes that match gregation, and inheritance relationships are used to the set of key attributes of the referenced object (all model relationships between concepts. Figure 1 shows the variations of an entity type must have the same the metamodel of the U-Schema data model in form key.) References also have a name, a cardinality, of a UML class diagram. Below, we describe this and an optional inverse reference (opposite). Ad- metamodel. ditionally, references can have their own attributes A U-Schema model represents a schema formed by when they represent graph arcs. This entails that a collection of types (SchemaType) that can be ei- a reference has to specify which variation (of its ther entity types (EntityType) or relationship types RelationshipType) its set of attributes corresponds (RelationshipType). Both types have two common to (isFeaturedBy). Key represents the set of at- properties: They include one or more structural vari- tributes playing the role of key for an entity type, ations (StructuralVariation), and they can form a holding a unique set of values for each element of the type hierarchy (parent relationship). type. Reference also points to the set of attributes A StructuralVariation has an identifier and is that form the referenced key (attributes property). characterized by a set of logical and structural fea- The aggregation relationship allows objects to be re- tures. StructuralFeatures determine the structure cursively embedded, then forming aggregation hierar- of database objects, and include Attributes and chies. In these hierarchies, the identification of the Aggregates, while logical features specify what identi- root element is important. Thus, an entity type in- fies an object (Key), and which References an object cludes a boolean attribute named root to indicate has to other objects. whether or not their entities are aggregated by others Each attribute has a name and a data type. The (aggregates relationship). data types included are: Primitive (e.g., Number, U-Schema also records information that could be use- String, Boolean), collections (sets, maps, lists, and ful to implement some database tasks. For example, tuples), and the special Null type. Also, the JSON as shown in Figure 1, StructuralVariations have and BLOB primitive types are included to support a count attribute to record the number of objects
5 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 1: U-Schema Metamodel.
that belong to each variation, and two timestamps 2.3 U-Schema Flavors: Full Variability vs. that hold the creation dates for the first and last Union Schema stored object of a variation (firstTimestamp and U-Schema allows to accommodate the definition of lastTimestamp). two model flavors: The U-Schema metamodel has been defined with the Ecore metamodeling language. Ecore is the central • Full Variability: All structural variations of all element of Eclipse Modeling Framework (EMF) [47], entity and relationship types are stored. a framework widely used to develop Model-Driven • Union Schema: Only one structural variation Software Engineering (MDE) solutions [11]. EMF- is stored for each schema type. Structural vari- provided tools such as model transformation lan- ability is recorded by using the optional boolean guages, model comparison and diff/merge tools, or attribute of Feature to indicate if a feature is workbenches for the creation of domain-specific lan- present or not in all the objects of an schema type. guages (DSLs) could be used to build database tools Union schemas are the schemas normally obtained based on U-Schema models. Metamodeling has tra- in NoSQL schema discovering processes [36, 52], ditionally been applied to define data models, and and visualized in NoSQL modeling tools. transformational approaches have been proposed to tackle problems involving schema mappings [27, 9]. Note that it is easy to convert a U-Schema model from However, the database engineering community has the Full to the Union flavors. This conversion loses paid little attention to MDE techniques and tools, information, and thus it is not reversible: Given a although metamodeling and model transformations schema type t with a set of n variations t.variations = t foundations are well established in the MDE field. {Vi }, i = 1 . . . n,, then t will be replaced by a schema Using Ecore, we obtain two benefits: leveraging the type s with the same name (t.name = s.name), and EMF tooling to develop database utilities, and favor the set s.variations will have a single variation W s s Un t U their interoperability with other tools [8]. with W .features = i=1 Vi , where is a function that returns the union set of all the features of all the variations with the following rules: 1. If the same structural feature appears in all varia- t tions Vi , then it is included in the result set with its
6 A Unified Metamodel for NoSQL and Relational Databases A Preprint
optional attribute set to false (common structural a U-Schema model that contains elements not present features). in a given data model. Specialized forward and reverse 2. Each structural feature that appears at least in a mappings could also be defined for each data model, variation is included in the result set, but with its and even for a given database implementation. These optional attribute set to true. mappings could be devised for specific needs within a development such as a database migration that in- 3. Structural features that appear with the same volves different source and target data models. The name StructuralFeature name ( attribute of ) but need for these mappings raises the interest in creating with different type (they belong to a different sub- a mapping language able to specify how the constructs StructuralFeature metatype of or have different of a given database paradigm are translated into the values of their attributes), are included with a nu- abstractions of U-Schema, and vice versa. This is out name meric identifier appended to their , and with of the scope of this work. their optional attribute set to true. In the following Section, the common strategies de- Example of an union schema for the running exam- vised to implement and validate all the forward map- ple presented in Section 3.2 is shown in Figure 9b. pings will be described. In sections 4 to 8, we will StructuralVariations are omitted for clarity, and define a logical model for each database paradigm optional features are shown in cursive and green color. addressed and formally express the canonical mapping We will use the Full Variability flavor through the rest between each data model and U-Schema. Additionally, of the article, as it contains more information and can reverse mapping examples will be shown for charac- be trivially converted to the Union Schema if desired. teristics not supported in each of the data models. For each paradigm, the forward mapping implementa- 2.4 Mappings between U-Schema and the tion and validation will be commented. Here, we will Logical Data Models introduce the notation used to define the mappings.
A unified metamodel is intended to represent all the • A mapping between an element u of U-Schema and concepts of the individual data models that it inte- an element m of a data model is expressed as: grates. Therefore, a mapping must be established between the unified metamodel and each data model. u ↔ mk{list of property relations} We will call forward mapping to a mapping from a reverse NoSQL or relational model to U-Schema, and where property relations are expressed as indicated mapping to a mapping in the opposite direction. below, and the ↔ operator is commutative. As indicated above, we had to define a logical data model for each NoSQL paradigm. As most NoSQL • A property relation p1 = p2 expresses that a prop- databases are schemaless, the schemas are implicit erty p1 of u and an property p2 of m have the same in data and code. Therefore, the implementation of value. The = operator is commutative. a forward mapping must first capture all the logical • A property relation p ← v expresses that the value information of the implicit schema, as described in v is assigned to the p property of u or m. Section 3, and then apply the mapping to obtain the U-Schema schema (i.e., a U-Schema model). • Let e1 be a property of u and let e2 be a property As U-Schema is richer in concepts than each individ- of m, a property relation e1 ↔ e2 expresses an ual data model, forward and reverse mappings are enclosed mapping between e1 and e2. not unique for a particular data model. In addition, • In a property relation that expresses a mapping U-Schema concepts not present in a specific data between two elements, the map(e, t) function can model could be mapped in different ways in a reverse be used to obtain the target element of type t that mapping. This has led us to introduce the notion of maps to the source element e; if e maps to a sin- canonical mapping . A canonical mapping satisfies two gle target element, then the second argument is conditions: optional. 1. It must be forward-complete, that is, the rules must • Given an instance of a meta-class in U-Schema, dot correctly map all the characteristics of the data notation is used to refer to its properties. For ex- model to U-Schema concepts. ample, given an instance e of EntityType, e.name 2. As a consequence, it must be trivially bidirectional refers to the attribute name. within a data model. This is because given a U- Schema model, the original database schema could • Functions will be applied on elements of the data always be reproduced (as the U-Schema model holds model to obtain the value of their properties. Func- all its information.) tions will have the same name as the property. For example, given an entity type e, name(e) will refer While the canonical mapping rules cover the character- to its name property. Other functions will be in- istics of each of the logical data models, there may be troduced in some rules, and their proper definition cases where a reverse mapping has to be performed on will be shown.
7 A Unified Metamodel for NoSQL and Relational Databases A Preprint
3 A Common Strategy for the the data structure of a structural variation: a set of Implementation and Validation of pairs formed by the name of a property and its data the Extraction of U-Schema type. Given an object O stored in the database of an entity type e, its raw schema is obtained by applying Models the following 4 rules on the values of its properties pi:
In this Section, we will first explain how U-Schema R1 Each value vi of a pi property is replaced by a value models are built from NoSQL databases or relational representing its type according to the rules R2 and schemas. Then, a conceptual schema will be presented R3. as a running example to be used to illustrate the ex-R2 If vi is of scalar or primitive type, it is replaced by planations of the following five sections. Finally, the a value that denotes the primitive type: "s" for experiments used to validate the U-Schema model String, 0 for numeric types, true for Boolean, and building process will be exposed. The implementa- so on. tion and validation strategies are common for all the R3 If vi is an embedded object, the rules R1, R2, and paradigms, but some stages or experiments are not R3 are recursively applied on it. required in the case of the relational model. R4 If vi is an array of values or objects, rules R2 and 3.1 Building U-Schema Models R3 are applied to every element, and the array is As indicated in Section 2.4, in the case of NoSQL replaced with an array of values representing types. stores, applying a forward mapping first requires in- In the case of document systems, where the key is ex- ferring the schema that is implicit in the data and plicitly included in the documents, the representation code. These schemas conform to the logical data of the structure will contain one scalar property with model abstracted for each NoSQL paradigm. There- the name “_id”, representing the key of the entity fore, U-Schema models are built in a 2-stage process, type. Additionally, the following rule is applied to as illustrated in Figure 2. First, a MapReduce opera- infer references between objects: tion is performed on the database to infer its logical schema. This stage is not needed for the relational R5 Some commonly used conventions and heuristics model. In the second stage, the forward mapping rules are taken into account to identify references. For are applied to create a U-Schema model from the pre- example, if a property name (with an optional prefix viously inferred schema. Next, we explain these two or suffix) matches the name of an existing entity stages. type and the property values match the values of the “_id” property of such an entity. The value of the property is replaced concatenating the value indicated in rule R2 with the name of the entity type and the suffix “_ref.” The process is repeated to obtain the raw schemas of the relationship types in the case of graph databases. Figure 2 shows how the above rules are applied to User and Movie objects of a document store. A raw schema is obtained for each User object with identical structure, and the same for Movie objects. Once the map function is performed, the reduce func- tion collects all the identical raw schemas and outputs a single representative raw schema for each structural variation of an entity type, to which we will refer, hereafter, as variation schema. Figure 2 shows the variation schemas obtained for User and Movie ob- jects. Note that a variation schema will be generated for each structural variation of the objects. In the case of graph and key-value systems, a prelimi- nary stage is needed to achieve an efficient MapReduce processing, as explained in Sections 4.4 and 6.4. Figure 2: Generic Schema Extraction Strategy. We decided to build U-Schema models directly from the intermediate representation of the MapReduce output instead of building specific metamodels for Inferring the Logical Schema In the map opera- each paradigm, because U-Schema already contains tion, a raw schema is obtained for each object stored the abstractions present in each of the individual in the database. We call raw schema to an intermedi- data models, and the transformation would have been ate representation (JSON-like format) that describes redundant.
8 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Generating a U-Schema Model In the second 3.3 Validation of the Schema Building stage, variation schemas are analyzed to build the Process U-Schema model. For this, a parsing process is con- nected to a schema construction process by applying To validate our schema building process, we have ap- the Builder pattern [24]. Variation schemas are parsed plied the same validation for the four kinds of NoSQL to identify its constituent parts: properties and rela- paradigms. For each system, we used two databases, a tionships, as well as the entity type (or relationship synthetic one based on the running example, an a real type) to which they belong. Whenever the parser dataset. In each one of them, two experiments were recognizes a part, it passes it to a builder that is in carried out: (i) a round-trip strategy to check that the charge of creating the schema. A builder has been obtained U-Schema model is equivalent to the schema implemented for each data model, which captures how used to synthesize the database or the schema of the parts are mapped to U-Schema. The same parser is real existing database; and (ii) two queries are issued used for all the data models as its input are variation on the real and synthesized databases to assure that schemas. In the case of relational databases, only at least a data object exists for each inferred struc- this second stage is needed, as schemas are already tural variation (“all variations exist” query) and that declared. the extraction process correctly calculates the number of data objects of each variation (“correctness count” 3.2 The “User Profiles” Running Example query). In the case of the relational model, only the second experiment is performed, as only the canonical Figure 3 shows a “User Profiles” conceptual schema forward mapping must be implemented, because there that will be used to build a database example for each is no need to infer the logical model of the database. paradigm integrated in our unified model. In each The round-trip experiment consisted in the following case, a “User Profiles” database will be populated to steps. First, we manually created a U-Schema model execute the algorithm that creates the corresponding (i.e. a schema) with the desired database structure (or U-Schema model and also to validate this algorithm the existing structure in the case of the real database). as explained in the next subsection. The running example model covers all the elements “User Profiles” schema could be an excerpt of the con- that can be mapped into the logical data model of ceptual schema of a movie streaming platform, which the corresponding paradigm, but this may not be the is expressed as a UML class model. It has 3 entities case for the real dataset. To populate the initial run- labeled Movie, User, and Address, and 3 relationships: ning example database, we randomly created elements a user aggregates an address, a user has zero or more according to the defined model. Afterwards, we in- favorite movies, and a user has zero or more watched ferred the implicit schema, and finally verified that movies. User has the attributes name, surname, and this schema was equivalent to the original U-Schema email; Address has city, street, number, and postcode; model: the resulting model can differ in the ordering and Movie has title, year, and genre. of the different variations found for each entity or When instantiating each database, we will sup- relationship type, this is why in this case we could pose that there are 2 variations for the Ad- not use standard model comparison tools, so we built dress entity type: {street, number, city}, and a custom U-Schema model compare utility. {street, number, city, postcode}; and 2 variations for To evaluate the scalability and performance of the U- User that vary in the relationships: either favorite- Schema model building algorithms, we have generated Movies and watchedMovies coexist, or only watched- four datasets of different size for the running example. Movies is present, and in the attributes: the surName The larger database contains 800,000 objects for the attribute is only present when both relationships are. User and Address entities, 400,000 for Movie, and a mean of 20 watched movies and 20 favorite movies. User and Address have the same number of objects in each of their variations. The rest of datasets reduce the number of objects and relationships in a factor of 2, 4, and 8, as shown in Table 3. All the performance tests were run on an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 48 GB of RAM and using SSD storage. To give a meaningful expression of the scalability of the schema inference process, instead of comparing absolute times, we used as a time baseline an aggregate query that calculates the average of watched movies by users. This query could be representative of those obtaining periodic Figure 3: “User profile” running example schema. reports, so we suppose that the database is not op- timized for it. In this way, we can get results that are independent of the different configurations in the
9 A Unified Metamodel for NoSQL and Relational Databases A Preprint
deployment. Table 4 show the different times for the 1. All variations exist The database must store, queries, schema inference, and the normalized value at least, a database object for each entity type (inference time divided by query time) for the database variation (and relationship type variation in the sizes in Table 3. We expected the ratio to diminish as case of a graph store) present in the extracted U- the size of the database increases, as the initialization Schema model. time of the MapReduce framework becomes smaller 2. Count correctness No other variations are with respect to the total inference time. Moreover, present in the database, i.e., the total number of in all cases the ratio stays in the range of 17.58x objects in the database matches the sum of objects (MongoDB, smaller case) to 2.04x (HBase, biggest that belong to each structural variation of the en- case), and for the biggest case, the inference reaches tity types present in the extracted model (count a maximum of about 10x slower (MongoDB). This attribute included in the StructuralVariation is expected as the query only has to process a part metaclass of the U-Schema metamodel.) Also, this of the database while the inference treats the whole check would be performed for relationship type vari- database. In the following sections, these results will ations in the case of graph stores. be studied.
Watched/ 4 Representing Graph Databases as Size/Item User Movie Favorite Movies Nodes Relationships U-Schema Models Larger 800k 400k 20/user 2,000k 24,800k Large 400k 200k 10/user 1,000k 6,400k 4.1 A Data Model for Graph Databases Medium 200k 100k 5/user 500k 1,700k Small 100k 50k 3/user 250k 550k In graph systems (e.g., Neo4j and OrientDB), a Table 3: Database Sizes. database is organized as a graph whose nodes (a.k.a. vertex) and edges (a.k.a. arcs) are data items that correspond to database entities and relationships between them, respectively. Edges are directed from an origin node to a destination node, and more than one edge can exist for the same pair of nodes. Both nodes and edges can have labels and properties. Labels denote the entity or relationship type to which nodes or relationships belong, and properties are key-value pairs. This is the so called labeled property graph data model [3], that most NoSQL graph systems implement. Graph databases are commonly schemaless, so there may exist nodes and relationships with the same label but different set of properties. Moreover, the same label can be used to name relationships that differ in the type of the origin and/or destination nodes. Figure 4: Inference to Query time ratio. Thus, graph databases can have structural variations as explained in Section 2.1. For this kind of graph store, we have abstracted the DB Small Medium Large Larger following notion of logical graph data model, which is Neo4j Query (ms) 686 1,213 3,165 12,016 represented in form of UML class diagram in Figure 5: Inference (ms) 11,821 20,177 41,814 109,724 Normalized 17.23 16.63 13.21 9.13 i) A graph schema has a name (that of the database) MongoDB Query (ms) 295 380 840 2,366 and is formed by a set of entity types and a set of Inference (ms) 5,187 6,730 10,226 23,452 relationship types. Normalized 17.58 17.71 12.17 9.91 ii) An entity type denotes the set of nodes with the HBase Query (ms) 931 1,942 6,419 24,023 Inference (ms) 5,615 6,840 11,526 49,042 same label (or set of labels). Normalized 6.03 3.52 1.80 2.04 iii) Entity types can be single-label or multi-label de- Redis Qyery (ms) 1,002 2,833 10,091 43,888 pending on whether they have one or more labels. Inference (ms) 11,487 22,013 61,505 252,794 Normalized 11.46 7.77 6.10 5.76 iv) A relationship type denotes the set of relationships with the same label (or set of labels). A relationship Table 4: Times for inference and queries for all the type has origin and destination entity types. database implementations. v) Entity and relationship types can have structural variations. With the extracted U-Schema model, we build a set vi) A structural variation is characterized by a set of of queries on the databases to perform the second properties that is shared by elements with the same experiment: set of labels.
10 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 5: Graph Data Model. vii) A property is a pair that mimics the property of a Reference. The former represents a type or clas- node or relationship in the graph, having a key and sifier whose instances are relationships between a pair the scalar data type that corresponds to the values of nodes, and can have variations based in their set of the property. of attributes, while the latter denotes a particular link between two nodes. Note that Aggregation and Table 2 shows the correspondence between graph Key U-Schema elements do not have a direct corre- database elements and the graph model elements ex- spondence to elements of the graph model. Next, we pressed above. Note that a graph schema is obtained express the set of rules that defines the Graph to by the MapReduce operation of the schema extraction U-Schema canonical mapping. process described in the previous section. R1. A graph schema G corresponds to an instance uS Figure 6 shows a graph database for the “User of the uSchemaModel metaclass of U-Schema (i.e., a Profiles” running example. It has three entity schema or model) with the same name: types labeled Movie, User, and Address, and three relationship types labeled FAVORITE_MOVIES, uS ↔ G k {uS.name = name(G)} WATCHED_MOVIES, and ADDRESS. In the fig- R2. Each different single-label entity type e that ure, nodes are represented as circles, and relationships exists in G maps to a root EntityType et in the uS as arrows. Nodes having the same labels (i.e. entity schema, whose name is that of the label associated to type) are filled with the same color. In this example, e: gray for Address, white for User, and black for Movie. Nodes only show a property for each entity type: title et ↔ ek{et.name = name(e), et.root ← true} for Movie, name for User, and street for Address. Re- EntityType instances are included in the uS.entities lationships are tagged with their relationship types, collection. and no properties are shown. We suppose that there R3 e are the variations indicated in Section 3.2. . Each different multiple-label entity type that exists in G maps to a root EntityType et in the uS schema whose name is formed by concatenating the names of the set of n > 1 labels L = {l1, . . . , ln}, and et inherits from each entity type e1, . . . , en that corresponds to labels in L: et ↔ e k {et.name = concat(L), et.root ← true,
et.parents = set{map(e1), . . . , map(en)}} R4. Each relationship type r that exists in G maps to a RelationshipType rt and a Reference rf in the uS schema, which are named the same as the label associated to r. Figure 6: “User Profiles” Graph Database Example. r ↔ rt k {rt.name = name(r)}, r ↔ rf k {rf.name = name(r)} 4.2 Canonical Mapping between Graph RelationshipType instances are included in the Model and U-Schema uS.relationships collection, and Rule R7 specifies Each element of the graph model defined above has how references are connected to other elements of a natural mapping to a U-Schema element, with the U-Schema schema. the exception of relationship types, that map to R5. Each “variation schema” v of an entity or rela- two U-Schema elements: RelationshipType and tionship type in G maps to a StructuralVariation
11 A Unified Metamodel for NoSQL and Relational Databases A Preprint
sv in the uS schema, which is identified by means of a ag.aggregates is the aggregated variation av, could unique identifier index (an integer ranging from 1 to be mapped to a relationship type whose name is |EV | or |RV |). Structural variations are included in ag.name adding the prefix “AGGR_”, its origin entity the collection variations that both entity types and type being sv.container, and its destination entity relationship types have in a U-Schema schema. type being av.container. Origin and destination R6. Let P v be the set of properties of a “variation entity types should be created if they do not exist schema” v which maps to a StructuralVariation sv. in the graph schema. Also, properties of av should v v sv Each property pi ∈ P will map to an Attribute ati be mapped using rules R2 to R7, as well as this with the same name, which is included in the collection rule (if an aggregate is part of the properties of sv.features. The type of the property will map to one av). Figure 7 shows an example JSON document of the types defined in the Type hierarchy defined in of an U-Schema entity type Person that aggregates U-Schema, and a mapping has to be specified for each an object of the entity type Address. Figure 8 il- graph store. The property mapping can be expressed lustrates the reverse mapping where a relationship type named AGGR_address_address1 connects a as: v sv sv v pi ↔ ati k {ati .name = name(pi ), Person and Address nodes (we suppose that the sv v ati .type ↔ type(pi )} aggregated variation identifier is 1.) R7. Each reference in a U-Schema schema uS has to be connected to other elements of uS. Let rf be Person: a Reference which maps to a relationship type r { R4 name:"Diego", according to Rule , address:{ street:"Espinardo Campus", number: i) rf must be linked to the EntityType that maps 2} } to the entity type that denotes the destina- tion nodes for the relationship r: rf.refs_to ← map(destination(r)). Figure 7: Example Person Data with Address Aggre- ii) Let oe the EntityType of uS that maps to the gate. origin entity type of a relationship type r in G (oe ↔ map(origin(r))), rf will be present in the set of features of the variations of oe whose nodes are origin of edges that are instances of r. iii) rf must be linked to the structural variation which features: rf.isF eaturedBy ← sv, where sv is the StructuralVariation that belongs to the relation- ship type that returns map(r, RelationshipT ype). iv) The lowerbound cardinality of rf would be 1 (rf.lowerBound ← 1) and the upperbound cardi- nality could be 1 (ref.upperBound ← 1) or ∞ Figure 8: Person Aggregates Address in Neo4j. (ref.upperBound ← ∞) depending on whether the instances of r in the database (i.e. arcs of type r) have one or more destination nodes for a given 4.4 Implementation and Validation of the origin node. Forward Mapping for Neo4j A slightly revised strategy to that described in Sec- 4.3 Reverse Mapping Completeness tion 3 has been applied to implement the forward The graph model does not include the Key and canonical mapping for Neo4j. We chose this store Aggregate elements. Next, we provide a possible because it is the most popular graph database.8 It mapping for these two concepts. is schemaless and fits into the labeled property graph data model. • Key. Remember that the Key concept in U-Schema refers to those attributes that act as an object key The strategy had to be revised because graph or the set of attributes that form part of a ref- databases usually do not offer facilities to efficiently erence to another object. As references between process the whole graph, and sometimes they even objects (nodes) in graphs are explicit in arcs, there fail because of lack of resources. So we devised a pre- is no need to include key information into the graph liminary stage that serialized the graph obtaining all schema. However, that information could be in- the nodes along with their outgoing relationships. Of cluded in the nodes, for example, using a special each arch, the data included the source node with its _keys property holding the set of properties that properties, the properties of the arc, and the ID of the map act as key. destination node. We modified the operation of • Aggregate. An Aggregate ag that belongs 8DB-Engines Ranking https://db-engines.com/en/ to a particular StructuralVariation sv, where ranking, (January, 2021).
12 A Unified Metamodel for NoSQL and Relational Databases A Preprint
the generic strategy to construct all the raw schemas For document databases, we have abstracted the fol- for nodes and edges with this serialization format. lowing notion of document data model, which is repre- The serialization was organized in batches by using sented in form of a UML class diagram in Figure 10: Spark Neo4j connector [46]. This way, an efficient schema extraction process was achieved. The reduce i) A document schema has a name (that of the operation did not need any modification from that database) and is formed by a set of entity types. described in the generic strategy, generating variation ii) An entity type denotes a collection of documents schemas for both entity and relationship types from stored in the database. nodes and arcs, respectively. iii) Entity types have one or more structural variations. The process finalizes with creating the U-Schema model by applying the mapping rules to the previous iv) A structural variation is characterized by a set of output (i.e. the logical graph model). The resulting properties that are shared by documents of the same schema for the “User Profiles” running example is collection. shown in Figure 9a. We also show the union schema v) Properties have a name and a type, and can be in Figure 9b. attributes, aggregates, or references. The two experiments introduced in Section 3.3 were vi) Attributes denote object’s fields whose value is of successfully carried out on the Neo4j database cre- scalar or array type. An attribute is specified by the ated for the running example and a Movies dataset name of the field and the type of its value. We sup- 9 available at the Neo4j website. pose that there exists an attribute that acts as the Regarding scalability and efficiency of the model cre- key of the Entity type (e.g., “_id” in MongoDB). ation process, Table 4 show that the relative timesvii) Aggregates denote object’s fields whose value is an with the reference query decrease as the size of the embedded object. An aggregate is specified by the database increases. Neo4j, jointly with MongoDB name of the field and the variation schema of the show the worst ratio cases. This is because the query embedded object. (average of watched movies by user) is, by chance, easily optimized by the database. In any case, as theviii) References denote object’s fields whose values are database grows, the factor is never beyond 10x. references. A reference is specified by the name of the field and the type of its value. 5 Representing Document Databases Table 2 shows the correspondence between document database elements and the document model elements as U-Schema Models expressed above. Note that a document schema is obtained by the MapReduce operation of the schema 5.1 A Data Model for Document Databases extraction process described in Section 3. Figure 11 shows how the “User Profiles” running exam- Document databases (e.g., MongoDB and Couchbase) ple would be stored in a document database. Instead are organized in collections of data recorded for a of using JSON notation, we depicted the database Movie User Ad- particular database entity (e.g., , , and objects in a representation that remarks their nested dress in the running example). Data are stored in the structure and the references between objects. There form of semi-structured objects or documents [2, 13] are two collections: User and Movie objects, and the that consist of a tuple of key-value pairs (a.k.a. fields). relationships are as follows. User objects aggregate Keys denote properties or attributes of the entity, and watchedMovies objects with two properties: the stars the values can be atomic data (e.g. Number, String, attribute and the movie_id reference that records the or Boolean), nested or embedded documents, or an id value of a movie object (arrow from movie_id to array of values. Also, a string or integer value can Movie objects); watchedMovies objects are recorded act as a reference to another document, similar to in an array. To record favorite movies, User has the foreign keys in relational systems, although usually favoriteMovies array of references to Movie objects. no support for consistency is provided. The user addresses are stored as an address aggregate Semi-structured data is characterized by having its object of users. While graph databases rely on refer- schema implicit in itself [13]. Thus, document ences (i.e. relationships in graph store terminology) databases are commonly schemaless, and a collection to connect data items, and aggregation is normally can store different variations of the entity documents. not available to compose data, the opposite is true in Usually, document databases maintain data in some document database systems. JSON-like format. 5.2 Canonical Mapping between Document Model and U-Schema 9No longer available at the original site, a copy can be obtained from https://github.com/catedrasaes- Each element of the document data model defined umu/NoSQLDataEngineering/blob/master/data/Neo4j/ above has a natural mapping to a U-Schema element. Movies/. Next, we present the rules for the canonical mapping.
13 A Unified Metamodel for NoSQL and Relational Databases A Preprint
(b) Union Schema.
(a) Complete Schema. Figure 9: “User Profiles” Schema for Graph Stores.
Figure 10: Document Data Model.
R1. A document schema D corresponds to an instance which is identified by means of a unique identifier uS of the uSchemaModel metaclass of U-Schema index (an integer ranging from 1 to |EV |). Each v (i.e., a schema or model) with the same name: property pi of v will be mapped according to rules R4–R6. uS ↔ D k {uS.name = name(D)} sv ↔ v k {sv.variationId = idgen(), R2. Each entity type e that exists in D maps to a sv.features ↔ properties(v)} root EntityType et with the same name: StructuralVariation instances are included in the et ↔ ek{et.name = name(e), et.root ← true} collection et.variations. v uS.entities holds the set of instances of EntityType. R4. If pi is an attribute, sv R3. Each variation schema v of e corresponds to i) it will map to an Attribute ati with the a StructuralVariation sv of et in the uS schema, same name, which is included in the collection
14 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 11: “User Profiles” Document Database Example.
sv.features. The mapping is the same as that i) A Reference rf with the same name, which is defined in Rule R6 of the mapping between the included in sv.features. The mapping is the same graph model and U-Schema. defined in Rule R7 of the mapping between graph ii) Additionally, if the attribute is the key of the entity models and U-Schema models. type, a Key instance also exists in sv.features and ii) An Attribute at according to the mapping ex- sv is connected to the corresponding attribute ati . pressed in Rule R4-i, and at and rf appear con- v nected in the uS schema: at exists in rf.attributes R5. If pi is an aggregate that has a set of n properties v v and rf is part of at.references. G = {gi }, i = 1..n, it will map to three elements in the U-Schema model: 5.3 Reverse Mapping Completeness i) A non-root EntityType nr with the same name but capitalized and stemmed (function name∗()), The only element of U-Schema that is not di- which is included in the collection uS.entities: rectly supported by the document model is the RelationshipType. RelationshipTypes have struc- v ∗ v nr ↔ pi k {nr.name = name (pi ), nr.root ← false}tural variations, and some References can spec- ify (via its isFeaturedBy property) to which ii) A StructuralVariation instance av included in v StructuralVariation of a RelationshipType they nr.variations, and each property g is mapped re- i belong. cursively according to rules R4 to R6: rt v v Given a RelationshipType in a U-Schema model, av ↔ pi k {av.features ↔ G } the reverse mapping for documents would map to iii) An Aggregate ag with the same name as the prop- an entity type e whose name is rt.name + ”_REF ”. erty, which is included in sv.features. This aggre- Each variation of rt will correspond to a variation in gate ag is connected to the structural variation av e, applying mapping rule R3 (i.e., each Attribute that it aggregates. The mapping would be: in rt maps to an attribute of the corresponding e p v v variation of ). A reference property will exist ag ↔ pi k {ag.name = name(pi ), in all the variations of e that will map with rule av ∈ ag.aggregates} R6. Then, each Reference rf that belongs to a StructuralVariation v of the entity type et to which The cardinality of ag would be established as indi- origin(rt) maps, where ro is the relationship type cated in Rule R7-ii of the mapping between graph such that ro = map(rt), and whose isFeaturedBy is models and U-Schema models. a variation vt in rt.variations, will map to a refer- v R6. If pi is a reference, it corresponds to two elements ence property r named name(e)+”_ref” by applying of the U-Schema model: rule R6.
15 A Unified Metamodel for NoSQL and Relational Databases A Preprint
database. The ratio also goes down as the size of the rf ↔ rk{rf ∈ v.features, database doubles, with the exception of the Small and et ↔ origin(rt), Medium times, that are similar (17.58x and 17.71x). v ∈ map(et).variations, The ratio then goes down from around 18x to 10x for the biggest case. vt = rf.isF eaturedBy, vt ∈ rt.variations, 6 Representing Key-Value Databases r.name ← name(e) + ”_ref”} as U-Schema Models Figure 12 illustrates the application of the reverse 6.1 A Data Model for Key-Value databases mapping explained above for a U-Schema model con- taining a RelationshipType for the watchedMovies Key-Value (K/V) stores conform to the simplest phys- relationship type of the running example. It can ical data model of NoSQL systems. A K/V store be appreciated how the document schema would is an associative array, dictionary, map, or keyspace, contain an entity type named WatchedMovie_REF, that holds a set of key-value pairs, usually lexico- which has a structural variation for the single graphically ordered by key. As such, they are used StructuralVariation of the RelationshipType to record data with a simple structure, and refer- that exists in the U-Schema schema. That varia- ences and aggregations are not primitive constructs tion is connected to the attributes named stars and to build up data. They usually store a single entity movie_id. Also, there exists a reference to the en- type (e.g. user profile, user login, or a shopping cart), tity type Movie, and a reference and attribute named although data of several entity types could co-exist in watchedMovie_REF are present in the structural vari- the same keyspace. ations of the origin entity type (User in our example). Like document and columnar systems, K/V stores can The reference will connect the User objects with the record semi-structured objects. Several techniques can WatchedMovie_REF objects. be used to encode a tree-like structure into key-value Some document systems provide the dbref construct pairs, which use normally namespaces to build hierar- to record references between documents, which can chical key values. We chose one of the most commonly 11 include fields. In these systems, the document data used encoding patterns to define the canonical map- model shown in Figure 10 could be extended to con- ping between K/V databases and U-Schema, to which sider that references can have attributes. Then, the we will call the flattened key pattern of compound document model would include all the U-Schema ele- objects or simply flattened object-key pattern. ments, as it would also support relationship types. When using this pattern, the key of every pair not only acts as the identifier of the object, but also 5.4 Implementation and Validation of the encodes the name of a property of the entity type, in Forward Mapping for MongoDB a similar format to XPath or JSONPath [25]. Keys Once the output of the MapReduce described in Sec- are built with a separator to differentiate between tion 3.1 is produced, i.e., the set of variation schemas, the object identifier and the property name (e.g., a the generation of the U-Schema model is achieved by colon: “
16 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Documents:
// User Collection { "name":"Brian", ... "watchedMovies":[ 978 ] }
// WatchedMovie_REF Collection { " _id ":978, "stars":3, "movie_id":202 }
// Movie Collection { " _id ":202, "title":"The Matrix" ... }
Figure 12: Example of Application of the Reverse Mapping from a RelationshipType of U-Schema to a Document Schema.
K/V systems are schemaless, and several structural tion being that Rule R4-ii should be removed, and a variations of an entity type can therefore exist in the new rule has to be added because the notion of key is database. In Figure 14, the variations of the running different in this data model. example can be observed. R7. Each StructuralVariation sv in the U-Schema Taking into account the use of the flattened object-key model contains a Key instance k in sv.features whose pattern, the document data model presented in Sec- value of k.name is “_id”, and it is not connected to tion 5.1, and shown in Figure 10, can also be used any Attribute. for K/V systems by modifying the Key notion. In this case, every database object also has a key, but 6.3 Reverse Mapping Completeness it is not associated to any attribute. A namespace The same reverse completeness mapping rules exposed would correspond to an entity type or either, if only in Section 5.3 for the document model are applicable one namespace is used, each different entity type will in this case. have a different “
17 A Unified Metamodel for NoSQL and Relational Databases A Preprint
(b) Union Schema.
(a) Complete Schema. Figure 13: “User Profiles” Schema and Union Schema for Document Stores.
same object. The result is a set of JSON objects that times, the Redis implementation is the slowest, which are similar to those stored in a document database. confirms that the calculation of an aggregate value Now, the two stages of the common strategy are per- is not an appropriate operation for a K/V store. As formed: a MapReduce processing to obtain the set of in previous implementations, the ratio goes down variation schemas, followed by the generation of the from 11.46x to 5.76x as the database doubles. U-Schema model, which is similar to the document model with the exception of the key generation using 7 Representing Columnar Databases the rule R7. as U-Schema Models The schema extracted for the running example is the same as for documents, shown in Figure 13a. The 7.1 A Data Model for Columnar Databases union schema is shown in Figure 13b. In columnar databases, data is structured in a sim- The schema extraction process was validated using ilar way to relational databases. In the most pop- a database built for the running example, as well as ular columnar databases (Hbase [29] and Cassan- using a real-world dataset. The same EveryPolitician 13 dra [14]), a database or Keyspace schema S is com- dataset used with MongoDB (Section 5.4) was inserted posed of a set of tables T = {ti}, i = 1..n, and each into Redis. table ti usually stores data of a single entity type. As The performance of the Redis schema inference process in relational databases, each table has a name, and is implementation versus the query gets better than in organized in rows and columns, but rows have a more MongoDB or Neo4j. This is because the query itself has to process the whole database, as Redis does not 13This can be observed in https://db-engines.com/ include a query language. Note also that in absolute en/ranking. Cassandra appears in the 10nd position and HBase in the 22nd position as of March, 2021.
18 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 14: Key-Value Database Example for running example.
complex structure than in relational tables because in the case of Cassandra, the set of attributes in the they are organized in column families. A table t is table that are not UDTs will form the default column therefore defined in terms of a set of column families family.) In the example, the default column family is t t F = {Fj }, j = 1..m. Moreover, each row r belonging User, with Address and WatchedMovies as aggregated to a table t contains a row key. Figure 15 shows an ex- entities. Note that WatchedMovies aggregates an ample of columnar database for the running example, array of objects, so the name of the columns is which has the User and Movie tables. The User table formed by using the flattened object-key pattern14 contains three column families: User, Address, and (“
19 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 15: Columnar database example for the running example.
relationship types do not have a direct mapping to 8 Representing Relational Databases the model, and the same approach used in documents as U-Schema Models can be implemented: the new entity type with a name convention to hold the structure residing in the ref- 8.1 The Relational Data Model erences, and the reference itself on the origin entity type variations. Unlike NoSQL logical data models, there exists a standard relational data model which is formally de- fined through relational algebra and calculus. Being 7.4 Implementation and Validation of the “schema-on-write” is another significant feature that Forward Mapping for HBase and differentiates relational databases from NoSQL stores: Cassandra schemas must be declared prior to store data in tables. The relational model is based on the mathematical We implemented the forward mapping for Hbase and concept of relation and its representation in form of Cassandra. In the case of HBase, we applied the tables [15]. A detailed description of the relational common strategy of Section 3.1, with the MapReduce model can be found in [49, 19]. operation identifying the default and aggregated col- umn families, and building the variation schemas. In A relational schema consists of a set of relation the case of Cassandra, the API was used to retrieve schemas. Each relation schema specifies the relation the database schema, and then build the U-Schema name, the attribute names and the domain (i.e., type) model. of each attribute. Relationships between relations are implicitly represented by key propagation from Validation was carried out as described in Section 3.3. a relation schema to another (one-to-one and one-to- A database was created for the running example, and may relationships) or either by a separated relation EveryPolitician the same real world dataset used in schema (many-to-many relationships). Therefore, rela- MongoDB and Redis, introduced in Section 5.4, was tion schemas can represent entity types or relationship injected into Hbase and Cassandra. types. A relational schema is instantiated by adding Figures 13a and 13b show the variation schemas ob- tuples to each relation. Each relation has one or more tained for the running example, which are the same attributes that form the key (primary key), and each for all the aggregation-based stores. tuple is uniquely identified by the values of the key As shown in Table 4 and Figure 4, Hbase shows the attributes. Relations are represented as tables, and best performance of the inference regarding the ratio the term column is used to refer to the attributes, relative to the aggregated query. As with all systems, while rows name the tuples of a relation. A table with a slight difference in the two bigger databases can declare foreign keys: one or more columns that (1.8x to 2.04x), the ratio decreases as the size of the reference to the primary key of another table in a key database increases. This confirm the scalability of propagation. Figure 16 shows a relational database the schema extraction approach. HBase, like Redis, example for the schema of the running example. User is specialized in fast random-access queries, but the and Movie tables represent the entity types of identi- aggregated query has to process most of the database, cal name, WatchedMovies and FavoriteMovies tables making the times very close to the full process of the represent the many-to-many relationships from User database performed in the schema extraction. Thus, and Movie in the conceptual schema of the running ratios go from just 6x slower to around 2x slower in example, and User aggregates Address by incorpo- the case of the Larger databases. rating its attributes. Note that Address could be a The performance of building the Cassandra model separate table related by foreign key, but it has been was not recorded as no inference process is required integrated into User because they hold a one-to-one because the schema is already declared. relationship.
20 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Figure 16: “User Profile” relational example.
In the last four decades, conceptual and logical with the same name, and a StructuralVariation sv schemas for relational systems have been extensively that represents the only structure of the table that studied, and a lot of methods and tools are available exists in the database. The mapping between r and for using them in the whole database life cycle. Entity- sv is solved as in rule R2. Relationship (ER) [49], Extended ER (EER) [19] and rt ↔ r k {rt.name = name(r)}, Object-Orientation modeling are the most widely used sv ↔ r k {sv.id ← idgen , sv.features ↔ columns r } formalisms to model conceptual and logical schemas () ( ) for relational databases. As explained in Section 2.1, RelationshipType instances are included in the main concepts of such formalisms are included in uS.relationships and sv is included in et.variations. U-Schema, which redefines most of them, and adds R4. Each column c of a table t is mapped to an some other concepts. Attribute at with the same name, and the data type 8.2 Canonical Mapping between Relational of the column will map to one of types defined in Type Model and U-Schema Models the hierarchy of U-Schema (a mapping between types has to be specified for each relational system.) The relational model is completely integrated in U- The mapping can be expressed as follows: Aggregate Schema, but the latter has the element at ↔ c k {at.name name c , at.type ↔ type c } which is not present in relational schemas. Moreover, = ( ) ( ) all the tuples have the same structure, so that the Attributes of an EntityType et are included in the number of structural variations for an entity type is collection sv.features, where sv is the only structural limited to one. Next, we expose a set of rules that variation that et has. specify the canonical mapping between relational and R5. The primary key pk of a table t is mapped to U-Schema models. We will use the terminology of a Key k and the collection k.attributes includes the table data models. attributes that maps to the columns that form pk. R1. A relational schema D corresponds to a The name of k is the name of the attribute (if there uSchemaModel instance uS in U-Schema (i.e., a U- is just one), or t.name + ”_pk” otherwise (pkname() Schema model) with the same name: function): uS ↔ R k {uS.name = name(R)} pk ↔ k k {k.name = pkname(pk), R2. Each table t in R representing an entity type k.attributes ↔ columns(pk)} uS EntityType et maps to two elements of : a root For each attribute at ∈ k.attributes, at.key = k. k is StructuralVariation sv with the same name, and a also included in sv.features. that represents the only structure of the table that R6 fk t exists in the database. An identifier is generated for . Each foreign key of a table to a table s target fk Reference rf the variation sv and its features are mapped to the = ( ) is mapped to a , and rf.attributes columns of the table t by applying the rules R4 to R6. the collection includes the attributes fk This mapping can be expressed as follows: that map to the columns that form . The name of fk is the name of the attribute (if there is just one), or et ↔ t k {et.name = name(t), et.root ← true}, s.name + ”_fk” otherwise (fkname() function). The sv ↔ t k {sv.id ← idgen(), sv.features ↔ columns(t)}reference rf is included in sv. It also refers to the entity type that maps to the target table s: EntityType instances are included in uS.entities and sv is included in et.variations. fk ↔ rf k {rf.name = fkname(fk), R3. Each table r in R representing a relationship type rf.attributes ↔ columns(fk), maps to two elements of uS: a RelationshipType rt rf.refsT o = map(s)}
21 A Unified Metamodel for NoSQL and Relational Databases A Preprint
8.3 Reverse Mapping Completeness Finally, References are created for foreign keys in Reference The U-Schema elements that are not present in tables, and each will be connected to ele- the relational model are Aggregate, and (multiple) ments previously created according to the U-Schema StructuralVariation. Next, we describe some pos- metamodel (R6). sible mappings for these elements. The validation has been performed on the Sakila database available at the MySQL official website.15 • The canonical mapping only takes into ac- Sakila contains 16 tables, and the average numbers count a StructuralVariation per schema type of columns and references between tables are, respec- (resp. table). If an schema type has several tively, 5.6 and 1.4. The smallest table has 3 columns, StructuralVariations, then two possible alterna- and the biggest one 13 columns. We have checked tives are: (i) mapping each variation to a table with the correction of the U-Schema model generation by a distinctive naming scheme, and (ii) mapping all comparing the model obtained with the information variations to a single table where the columns re- on the database available at the MySQL website (SQL sult of the union of the set of properties of each creation files and official diagrams). In the study of structural variation. In the latter case, the tuples performance and scalability, as with Cassandra, re- of the table will have NULL values in the columns lational databases have not been considered because not corresponding to their structural variation. Ob- schemas are already available. taining the different entity variations from a table would require the analysis of all the tuples to regis- 9 Related work ter all the different set of non-NULL columns. This could be carried out with a similar operation to the In this section, the U-Schema metamodel will be con- MapReduce described in the common strategy of trasted to some relevant generic metamodels defined Section 3. for database schemas, and the schema inference strat- • Each Aggregate ag in an StructuralVariation egy to others published for NoSQL stores. sv of a given EntityType et could be mapped to elements of the relational model also in sev- 9.1 Generic Metamodels eral ways: (i) an additional table t with the DB-Main was a long-term project aimed at tackling name of the aggregate ag.name and the columns the problems related to database evolution [34, 28]. mapped to properties in the StructuralVariation The DB-Main approach was based on three main el- ag.aggregates using rules R4 to R6. A foreign key ements: (i) The Generic Entity/Relationship (GER) column is added to t, and a primary key to the metamodel to achieve platform-independence; (ii) A table mapped to et. (ii) If the aggregate cardinality transformational approach to implement operations is one-to-one, the attributes of the ag.aggregates such as reverse and forward engineering, and schema variation could be incorporated into the table that mappings; and (iii) A history list to record the schema maps to et. The aggregation relationship between changes [34]. Here, our interest is focused on the two User and Address in the running example schema former elements. The generic GER metamodel was has been mapped using the second alternative, as defined as an extension of the ER metamodel [49]. shown in Figure 16. Conceptual, logical, and physical models could be rep- resented in GER. Models for a particular paradigm, 8.4 Implementing and Validating the system, or methodology were obtained by means Relational Schema Extraction Process of (i) selecting necessary GER elements, (ii) defin- In the case of relational databases, it is not necessary ing structural predicates to establish legal assemblies to infer schemas: U-Schema models can be obtained of that elements, and (iii) choosing an appropriate from relational schema declarations. We chose MySQL visual diagram. Regarding schema transformations, a to implement the set of rules exposed above for the set of basic transformations were defined, and the sig- relational to U-Schema mapping. Rule R3 cannot be nature of each of them (name, input, and output) was applied as the schema does not distinguish between specified in a particular format to be used to record relationship and entity tables. This information could changes in the history list. Our proposal differs of the be provided, for example, through name conventions, GER approach in several significant aspects. which could also be used to specify aggregation tables. A. Support of semi-structured data in NoSQL The model generation process is straightforward, and stores it works following the described mapping rules. First, It is convenient to remark that our approach R1 is applied to create and name the model, then an shares objectives with DB-Main. However, DB-Main EntityType and a StructuralVariation are created was focused mainly on relational systems, and also on for each table (R2). An Attribute is created for each earlier database systems. Instead, we are interested in column of a table (R4). Next, Keys are created for 15Sakila can be downloaded from https://dev.mysql. primary keys in tables, which will have references to com/doc/index-other.html, and documentation is avail- Attributes that have been instantiated previously able at https://dev.mysql.com/doc/sakila/en/sakila- for the columns that are part of the primary key (R5). structure.html.
22 A Unified Metamodel for NoSQL and Relational Databases A Preprint
both structured and semi-structured data, specially in et al. [7] described a universal metamodel based on a the emerging NoSQL stores and relational databases. three-level architecture similar to those defined in the EMF framework and used in our work: a metamod- B. Physical and conceptual level separation in eling language (Ecore) is used to define metamodels different metamodels U-Schema is intended to (U-Schema in our case), which, in turn, are used to represent logical schemas, so that conceptual and create models (schemas in our case). In [7], a set physical schemas are separately modeled. Instead of of 13 meta-constructs were defined to represent the mixing all the information in a single metamodel, we concepts used in different data formalisms. For ex- have considered more convenient to separate the large ample, Abstract is proposed to model autonomous amount of physical concepts in their own metamodel concepts such as ER entities or OO classes, Abstrac- and to have a simpler conceptual model. Because of tAttribute to model references, and Generalization to this concern separation, reusability is promoted, and model Abstract hierarchies. This proposal overlooked models are kept simple and readable. The conceptual the already existing MDE frameworks, in particu- and physical metamodels are out of the scope of this lar EMF/Ecore. Instead, the authors started from paper. At this moment, we have defined a physical scratch, and they even proposed a dictionary structure metamodel for MongoDB, as described in [38]. to store models as instances of the universal meta- model. Schemas are expressed by indicating, for each C. Concrete schemas are directly represented element, the construct at the level of the data model in U-Schema Unlike GER, we do not have to define from which is instantiated, and for each of these con- a sub-model of U-Schema for each database system. structs its meta-construct at the level of the universal U-Schema acts as a pivot representation, able to repre- metamodel. The metamodel was accompanied by a sent NoSQL and relational schemas for all paradigms. basic tooling for textual and graphical visualization. The set of rules that maps each data model to U- The main differences between this universal meta- Schema determines the U-Schema elements involved, model and U-Schema are the following: and therefore the valid structures. A. A different purpose and meta-modeling D. Structural variation representation A cen- architecture While the universal metamodel of tral notion of U-Schema is structural variation. Vari- Atzeni et al. is aimed to instantiate data models, ations of entity and relationship types can be repre- U-Schema is a unified metamodel able to represent sented. This information can be useful in different schemas of a variety of databases. Therefore, the meta- tasks. For example, variations are used to identify modeling architectures are different: Universal meta- whether an entity type contains a type hierarchy (e.g. a model/Data Model/Database Schemas vs. Ecore/U- Product hierarchy) in [31]. Variations also allow to Schema/Database Schemas. It is worth noting that analyze the database evolution, or can be used to our approach does not prevent the definition of meta- generate test datasets, among other tasks. models for representing any existing data model that E. Solution based on MDE specification As is integrated in U-Schema. However, as indicated in indicated in Section 2, we have defined U-Schema with Section 3.1, we have considered that creating these the Ecore metamodeling language with the purpose metamodels would not provide any benefit as inter- of taking advantage of MDE technology integrated in mediate representation, as the variation schema to the EMF framework [47]. data model transformations would be very close to the variation schema to U-Schema transformation. F. Schema extraction In DB-Main, a different B. Availability of tools for basic model oper- schema extractor had to be developed for each ations database system. In our case, a common strategy While we used the EMF metamodeling ar- have been defined which address the scalability and chitecture to create U-Schema, Atzeni et al. had performance issues. to implement their own metamodeling architecture from scratch, as well reporting and visualization tools. Model Management MM ( ) is an approach aimed Instead, EMF provide tools supporting model com- data programmability to solve problems which nor- parison (EMF Compare)16 and model diff/merge mally involve complex mappings between data operations (EMF Diff/Merge),17 as well as model- schemas of different sources [10, 9]. A set of oper- transformation languages to implement the modelgen match ators between models are proposed, such as , operator. union, merge, diff, or the modelgen operator that gen- erates a schema from another. In [10], building a C. Relationship types and structural varia- universal metamodel is considered a feasible way of tions The expressiveness of the Universal meta- developing tools to specify mappings, although it does model is covered by U-Schema elements. In addition, not seems the more adequate alternative because of U-Schema includes the notions of relationship types the large complexity of the required metamodel. Two universal metamodels for applying Model Man- 16https://www.eclipse.org/emf/compare/. agement are presented in [7] and [35]. Paolo Atzeni 17https://wiki.eclipse.org/EMF_DiffMerge.
23 A Unified Metamodel for NoSQL and Relational Databases A Preprint
and structural variations, which are convenient to a higher level of abstraction than SOS and NoAM. represent schemas of NoSQL stores. These representations are closer to the physical level GeRoMe is another generic metamodel proposed than the logical. Thus, some key aspects for a logical for Model Management [35]. A role-based modeling schema are neglected, such as the relationships be- is applied to define a metamodel able to represent tween entities. In addition, the existence of structural different data models. In mid-nineties, role-based variations is not considered. Finally, MDE technology modeling approaches received attention in the context was not used in their definition. of object-oriented programming to model the multiple- ERwin Unified Data Modeler (ModelSet) is a project classification and object collaborations [40]. However, outlined in an article in infoQ [51] whose purpose is that interest has decreased over the years because very close to ours. However, to our knowledge, results languages and tools do not support the notion of role. of that project have not been published yet. In [51], Extended ER, Relational, OWL-DL, XML Schema, Allen Wang, responsible of the ERwin project, pointed and UML were analyzed in GeRoMe with the aim out on the importance of “using logical models describ- to identify their similarities and differences. Then, a ing business requirements and de-normalizing schema set of roles was established, and the role-based meta- to physical data models” in database design. A sim- model created. U-Schema clearly differs of GeRoMe ple unified logical model is shown to represent three in its purpose and the kind of representation of the kind of schemas: columnar, document, and relational generic metamodel. Our unified metamodel has been schema. The metamodel only includes four elements. defined by applying object-oriented conceptual model- The three basic modeling constructs: Entity, Rela- ing, the technique commonly used currently to create tionships, Properties (of entities), and Tags are used metamodels, and using a well-know metamodeling to add additional information to basic constructs. A architecture. physical model should be built for each system. Query As far as we know, neither of the three generic meta- and data production patterns are defined on the logi- models here considered (GER, Atzeni et al., and cal model for its transformation into physical model. GeRoMe) has evolved to include elements specific Several significant differences are found between U- of NoSQL stores. Therefore, none of them has ad- Schema and the ERwin metamodel: (i) U-Schema is dressed the representation of structural variations or not only able to represent aggregate-based systems, relationship types. In the case of DB-Main, the tool but also graph stores; (ii) U-Schema is more expressive, can currently be acquired from the Rever company18 ModelSet only includes the three basic constructs of as a tool to simplify data engineering tasks. modeling, but this is similar to our variation schemas More recently, several metamodels have been pro- that are input to the analysis process; (iii) Being U- posed to represent NoSQL schemas. SOS is a meta- Schema a representation at higher level of abstraction, model designed to represent schemas of aggregate- the definition and implementation of operations such based stores [5]. With this uniform representation, as schema mapping, visualization, or schema discovery a NoSQL schema consists of a set of collections (Set are easier; (iv) U-Schema represents structural varia- metaclass), which can contain Structs and Attributes. tions; (v) Instead of a proprietary tool, U-Schema is An Attribute represents a key-value property, and a part of a free data modeling tool. 19 group of key-value pairs is modeled as a Struct. Struct The Typhon project is an European project aimed and Set can be nested. Later, SOS evolved to the to create a methodology and tooling to design and NoAM (NoSQL Abstract Model) metamodel, which develop solutions for polystore database systems. As was defined as part of a design method for aggregate- part of this project, the TyphonML [50] language based NoSQL databases [6, 4]. NoAM was designed as has been built, which allows schemas to be defined an intermediate representation to transform aggregate in a database system-independent way. Columnar, objects of database applications into NoSQL data. A document, key-value, graph, and relational schemas NoAM database is a set of collections that contains a can be defined with TyphonML. Typhon schemas can set of blocks. A block contains a set of key-value pairs, also express mappings from schemas to the physical and each block is uniquely identified. In [6], several representation. Some remarkable differences with U- strategies are described to represent a collection of Schema are: (i) TyphonML is not a language defined aggregate objects in form of a NoAM database. on a unified metamodel, instead U-Schema is sepa- NoAM and SOS were designed with a purpose dif- rated from any schema declaration language. In fact, we have created the Athena language on U-Schema, ferent to U-Schema. SOS aims to achieve a uniform 20 accessing, and NoAM is part of a design method. and other languages could be defined ; (ii) The ex- Instead, U-Schema has been devised to have a uni- istence of structural variation in NoSQL systems is form representation able to capture data models of not considered; (iii) As can be observed in [50], for NoSQL and relational data models, with the aim of each paradigm, the TyphonML metamodel includes facilitating the building of database tools supporting several database systems. Therefore, U-Schema offers 19https://www.typhon-project.org/. 20https://catedrasaes-umu.github.io/ 18https://www.dataengineers.eu/en/db-main/. NoSQLDataEngineering/tools.html.
24 A Unified Metamodel for NoSQL and Relational Databases A Preprint
logical and physical aspects; Instead, our choice is to process; (iii) Our inference strategy discovers relation- separate both levels of abstraction in two metamodels ships between entities, and structural variations of as pointed out in [38]; (iv) Although graph stores are entities. (iv) The output of our inference process is a represented, the concept of relationship type is not model that conforms to an Ecore unified metamodel. included in TyphonML; (v) Aggregates are not repre- In this way, we can take advantage of benefits offered sented as a separate concept, but the same metaclass by MDE, which were commented in Section 2.2. Reference represents both aggregates and references An MDE-based reverse engineering approach for ex- by using the boolean attribute isComposite to record tracting conceptual graph schemas is described in [17]. the kind of relationship; Instead U-Schema represents CREATE Cypher statements are analyzed to obtain aggregates and references with two metaclasses, which a graph model: a graph is formed by nodes and allows us to have a complete semantics. The logical edges, and nodes have incoming and outgoing edges. elements of TyphonML are limited to Entities that Then, a model-to-model transformation generates an aggregate Attributes and References, while our unified Extended Entity-Relation (EER) conceptual schema metamodel has a wider and richer set of semantic model, whose elements are entities, relationships, at- concepts. tributes, and IS-A relationships. Our inference process A 2-step model transformation chain aimed to trans- differs of this strategy in several significant aspects, form conceptual models into physical schemas (Cassan- apart from being a generic strategy and use a MapRe- dra stores are considered) is described in [1]. Logical duce to considerably improve the efficiency: (i) We models are generated in an intermediate step. Con- access stored data instead of using CREATE statements, ceptual schemas and logical schemas are represented which might not be available. (ii) Instead of building by means of very simple metamodels. A conceptual a graph model, we create a table with all relation- schema is formed by a set of classes and datatypes, and ships as input to the MapReduce operation; this table classes include attributes and relationships. A rela- allows us to obtain the cardinality of each relation- tional model is used as logical model, to which relation- ship (iii) We obtain a logical schema that include ships are added. This proposal has some flaws such structural variations for relationships and entities. as (i) relationship types and references are not distin- guished, which is necessary for graph schemas, (ii) ref- 9.3 Data Modeling Tools erences have not properties, and (iii) the separation between logical and conceptual model is not justified With the emergence of NoSQL systems, multi- because they include the same concepts but differ- paradigm data modeling commercial tools have pro- ent names; this can be observed in the very simple liferated. In our study of some of the most relevant conceptual-to-logical transformation shown in the pa- of these tools, we have found no evidence showing per. the use of a unified metamodel. Next, we contrast Table 5 summarizes the discussion made above, and features of these tools with those considered in our compares the generics metamodels considered accord- U-Schema approach. ing to several criteria. These tools can be classified in two categories. A first category are existing tools for relational databases 9.2 NoSQL Schema Extraction Strategies which are incorporating some NoSQL systems. At this moment, these tools have only added support Recently, several approaches to extract schemas from for document systems, being MongoDB the system NoSQL document stores have been published [52, 36, integrated in the most of them. 16]. A detailed study of these works can be found For example, ER/Studio [21] and ERwin [22] provide in [23] where they were contrasted to our previous utilities to extract and visualize schemas for MongoDB approach for document stores [43]. Moreover, some and CouchDB since 2015. They extract schemas as works on schema extraction from JSON datasets have a set of entity types whose properties are the union also been presented. In [16], a MapReduce is used to of all fields discovered in objects of that entity, but obtain a collection of key-value pairs from an input variations and relationships are not addressed. Re- JSON dataset. In each pair, the key is a document cently, ERwin Data Modeler provides an integrated specifying the structure or type of a JSON object view of conceptual, logical and physical data models in the dataset, and the value records the number of to help stakeholders understand data structures and elements of the that type. In a second step, heuristics meaning. are applied to merge similar types. The second group is formed by new tools developed The main differences of the approach described here with the purpose of offering data modeling for polyglot with previous extraction strategies are the follow- persistence. As far as we know, Hackolade [26] is the ing: (i) They are focused on document stores. In- only tool that integrates database systems for the four stead, we have defined a general strategy applicable most common NoSQL paradigms as well as a wide to the four main NoSQL paradigms and the relational number of relational systems and other leading data model; (ii) Like [16], our approach use a MapReduce technologies. Recently, it has been announced the operation to improve the efficiency of the inference creation of a unified model named “Polyglot Data
25 A Unified Metamodel for NoSQL and Relational Databases A Preprint
Model” but no details have been published. Unlike files (e.g. JSON or CSV) or can act as a source feed U-Schema, Hackolade does not address variation and for a streaming process. Likewise, U-Schema models references in the NoSQL schema extraction. Entities could automate the data ingestion procedure using extracted are represented as the union of all the fields bulk insertion utilities from files, generated insert discovered in different variations of the entity. The queries, or even help to build the ingestion as the last collision of fields with the same name but different stage of a streaming process. type is not considered but that modeler should make The next step is to specify and execute the mapping a decision. rules between source and destination elements. The DBSchema [18] is a tool similar to Hackolade: It mapping rules introduced in sections 4 to 8 should allows the developer to define schemas with a graphical be adapted to the specificities of the migration. For layout, but also to apply a reverse engineering process example, an alternate mapping could be devised for to an existing database in order to extract the schema, characteristics not present in the destination data as long as there is a JDBC Java driver for it. Queries model, as was the case we showed with aggregates in can be created in an intuitive way or either using a graph data model in Section 4.3. The migration rules SQL. In this tool, variations are not considered at all, could be hardcoded, or either specified with a ad-hoc since it applies a SQL approach to infer the schema, language. This language would be defined taking into in which variations are not taken into account. account the abstractions of U-Schema. The migration rules would include the U-Schema source element, 10 Applications of the U-Schema the target data model element, and the mappings between the parts that constitute the source and target Metamodel elements, similarly to how we expressed the canonical mappings before. The usefulness of generic metamodels is well known, and has been extensively discussed in the database literature for more than 30 years. In this section, we 10.2 Definition of a Generic Schema Query shall show how U-Schema metamodel favors the imple- Language mentation of some database operations that involve Schema query languages help developers to inspect SQL and NoSQL systems. We will present a migra- and understand large and complex schemas. In the tion process and discuss how U-Schema facilitates the case of relational systems, SQL is used to query development of database migrations, generic solutions schemas represented in form of tables in the data to query schemas, generate synthetic data for testing dictionary. In NoSQL stores, a similar query facil- purposes, and visualize schemas. ity is provided by some systems that require to de- clare schemas, for example Cassandra [14] and Ori- 10.1 Database Migrations entDB [39]. In the case of schemaless NoSQL systems, Database migration is a typical task in which a unified the number of variations can be very large in some or generic representation provides a great advantage. domains, for example 21,302 variations for the Com- Given a set of m database systems, the total number pany entity type of DBPedia are reported by Wang of migrators required is m + m instead of m × (m − 1). et al. [52]. Using U-Schema, a generic query language Here we will describe how U-Schema models can be could be defined which would allow querying rela- used to help migrate databases when the source and tionships and structural variations for any kind of target systems are different. NoSQL store, unlike existing solutions. As far we To perform a migration, the source and destination know, querying variations has been only addressed in databases have to be specified, as well the mapping the mentioned work of Wang et al. [52], which focused rules that determine how source data are moved to the on MongoDB, and only suggested a couple of queries to illustrate the idea. A first version of our language destination database. A migration tool usually has 21 to read all the data in the original database, perform can be downloaded. some processing, and write the resulting data in the The U-Schema query language allows to query the destination database. These steps can be carried out schema of any type of database system under a unique in different ways, that can be simplified by using U- language, and even make it possible in scenarios where Schema models, as they contain all the information the data is stored in different database systems (poly- of entities, attributes, and relationships. Therefore, glot persistence). Some examples of the most com- the U-Schema model has to be obtained prior to the mon queries that a developer might need are: (i) get aforementioned steps. an overview of the entities and the relationships be- There are several options when reading the original tween them, (ii) search variations with a set of prop- data. A set of queries could be constructed to extract erties, (iii) check all shared properties of all variations the data guided by its structure (i.e., its schema). The of a specific entity. The results of the queries could inferred U-Schema model from the source database can be used to automatically generate those queries. 21https://github.com/catedrasaes-umu/ The queries can produce a set of interchange format NoSQLDataEngineering.
26 A Unified Metamodel for NoSQL and Relational Databases A Preprint
be displayed as text or a graphic representation in the 11 Conclusions and Future Work form of tables, graphs or trees (hierarchical data). Multi-platform database engineering tools commonly 10.3 Generation of datasets for testing define a unified metamodel to represent database purposes schemas of a variety of systems. In this paper, we have presented a proposal of unified metamodel that Automatic database generation is a point of interest in integrates data models for relational and NoSQL sys- designing, validating, and testing of research database tems (key-value, document, wide column, and graph). tools and deployments of data intensive applications. These systems cover most of current database applica- Often, researchers in the data-engineering field lack of tions. Our work is motivated by the growing interest real-world databases with the required characteristics, in multi-model database tooling and systems as poly- or they can not access them. glot persistence is considered essential to satisfy needs Some works have addressed the generation of synthetic of modern applications. With U-Schema, we have data on relational systems, and some restriction lan- defined a representation able to express inferred or guages have been proposed to this purpose [12, 45]. declared schemas at a similar abstraction level to EER With U-Schema, a database paradigm-independent and Object-Orientation logical models. restriction language could be defined to tailor the U-Schema is the first logical unified metamodel de- generation of data. In this way, a given specification fined for NoSQL and relational systems taking into could be used to generate data for different databases. account structural variation, relationship types, aggre- Note that the language constructs would be at the gations, and references. Through forward and reverse level of abstraction of U-Schema, and not aligned to mappings, we have formally shown how U-Schema is elements of any concrete paradigm. able to represent each considered data model, and This is of special importance in the case of distributed how U-Schema models can be converted to schemas systems, as most NoSQL deployments are. In this of the individual models. We would like to remark context, a cost model to evaluate query efficiency that the extraction of schemas from databases (for- is very difficult to build, given all the variables in- ward mappings) have been implemented for the most volved [37]. Generating different sets of data with widely used NoSQL systems (Neo4j, MongoDB, Redis, different characteristics can help fine-tuning applica- Cassandra, and HBase), as well as for one of the most tion intended queries. For example, just changing the used relational systems (MySQL). For each extraction relationships between the entities of a schema (for algorithm, scalability and performance were assessed. example, changing references into aggregations or vice Having used the de facto standard Ecore to represent versa), new data that follows this change could be the schemas turns the framework in a reusable and generated to test the queries, helping the developers adaptable tool, and Eclipse modeling tooling can be to find opportunities for optimization. used to build database tooling. Finally, another advantage of our approach around U- Future work can be divided in two lines, depending on Schema is that in the case of existing databases, their whether they have to do with the unified metamodel- schema can be inferred into a model, and then used ing approach, or with applications based on U-Schema. to generate data that can be for the same or different We approached the unified representation of schemas databases, matching the schema or even introducing by separating logical and physical aspects. The meta- changes, either for performance tuning or for testing model presented here concerns to the logical view, purposes. and a new metamodel will represent the unified phys- We developed an initial version of a U-Schema-based ical schemas. Thus, we will have U-Schema-Physical data generation language with the described charac- and U-Schema-Logical, where physical schemas will teristics [33]. be extracted from data stores, and logical schemas could be directly obtained either from stores or from 10.4 U-Schema Schema Visualization physical schemas, as described in [38]. Physical data models for each system will include data structures at When schemas are extracted they must be expressed in physical abstraction level, indexes, physical data dis- a graphical, textual, or tabular format to be shown to tribution, among others. Regarding improvements of stakeholders. Normally, they are shown as a diagram U-Schema, we will extend the metamodel to represent (e.g., ER or UML). In a previous work, we explored constraints to support new logical validation character- the visualization of document schemas and proposed istics in some NoSQL databases, such as the MongoDB several kinds of diagrams for document systems [30]. Schema Validation. Finally, it is planned to build sev- Now, it is possible to take advantage of U-Schema eral tools and languages around U-Schema: (i) The to define common diagrams for logical schemas tak- migration approach outlined in Section 10.1, which ing into account the existence of variations if needed. will include a U-Schema based schema mapping lan- Moreover, U-Schema could be mapped to other for- guage to express specialized mappings as described in mats with the purpose of visualizing schemas in exist- Section 2.4; (ii) Complete the generic Schema Query ing tools. Language introduced in Section 10.2; (iii) Define some
27 A Unified Metamodel for NoSQL and Relational Databases A Preprint diagramming tools from the results in [30]; (iv) A [11] M. Brambilla, J. Cabot, and M. Wimmer. Model- universal schema definition language, that, by using Driven Software Engineering in Practice. Morgan U-Schema, allows to define schemas homogeneously & Claypool Publishers, 2012. through NoSQL and relational datastores; (v) Support [12] Nicolas Bruno and Surajit Chaudhuri. Flexible an approach of platform-independent schema evolu- Database Generators. In 31st International Con- tion through the definition of a change taxonomy ference on VLDB, pages 1097–1107, 2005. implemented by a schema change operation language; and (vi) This language could be used to explore the [13] P. Buneman. Semistructured Data. In Sixteenth impact of schema changes on the set of queries of ACM SIGACT-SIGMOD-SIGART Symposium an application, following automated reinforcement on Principles of Database Systems, pages 117– learning techniques. 121, Arizona, USA, 1997. ACM. [14] Cassandra webpage, 2016. Accessed Septem- References ber 2020. URL: http://cassandra.apache. org/. [1] Fatma Abdelhédi, Amal Ait Brahim, Faten [15] E. F. Codd. A Relational Model of Data Atigui, and Gilles Zurfluh. Logical Unified Mod- for Large Shared Data Banks. Commun. eling for NoSQL Databases. In ICEIS (1), pages ACM, 13(6):377–387, 1970. URL: http: 249–256. SciTePress, 2017. //doi.acm.org/10.1145/362384.362685, doi: 10.1145/362384.362685. [2] Serge Abiteboul, Peter Buneman, and Dan Suciu. Data on the Web: From Relations to Semistruc- [16] Dario Colazzo, Giorgio Ghelli, and Carlo Sartiani. tured Data and XML. Morgan Kaufmann, 1999. Typing massive JSON datasets. In International Workshop on Cross-model Language Design and [3] R. Angles, M. Arenas, P. Barceló, A. Hogan, Implementation, volume 541, pages 12–15, 2012. J. Reutter, and D. Vrgoč. Foundations of Modern Query Languages for Graph Databases. ACM [17] Isabelle Comyn-Wattiau and Jacky Akoka. Model Computing Surveys, 50(5), 2017. driven reverse engineering of nosql property graph databases: The case of neo4j. In 2017 [4] Paolo Atzeni, Francesca Bugiotti, Luca Cabibbo, IEEE International Conference on Big Data, and Riccardo Torlone. Data modeling in the 2017, Boston, MA, USA, December 11-14, 2017, NoSQL world. Comput. Stand. Interfaces, 67, pages 453–458, 2017. 2020. [18] Dbschema webpage, 2016. Accessed Octo- [5] Paolo Atzeni, Francesca Bugiotti, and Luca Rossi. ber 2018. URL: http://www.dbschema.com. Uniform Access to Non-relational Database Sys- [19] Ramez Elmasri and Shamkant B. Navathe. Fun- 24th International tems: The SOS Platform. In damentals of Database Systems. Pearson, 7th Conference on Advanced Information Systems edition, 2015. Engineering (CAiSE), pages 160–174, Gdansk, Poland, June 2012. [20] Vincent Englebert and Jean-Luc Hainaut. DB- Main: A next generation meta-case. Information [6] Paolo Atzeni, Francesca Bugiotti, and Luca Systems, 24(2):99–112, 1999. Rossi. Uniform access to NoSQL sys- tems. Information Systems, 43:117–133, [21] ER-Studio Webpage, 2015. Accessed Febru- https://www.idera.com/er- 2014. URL: http://www.sciencedirect.com/ ary 2021. URL: studio-enterprise-data-modeling-and- science/article/pii/S0306437913000719. architecture-tools. [7] Paolo Atzeni, Giorgio Gianforme, and Paolo Cap- [22] ERwin Data Modeler Webpage, 2016. Ac- pellari. A universal metamodel and its dictionary. cessed February 2021. URL: http://erwin.com/ Trans. Large Scale Data Knowl. Centered Syst. , products/erwin-data-modeler. 1:38–62, 2009. [23] Severino Feliciano. Inferring NoSQL Data [8] Francisco J. Bermudez, Jesús García Molina, and Schemas with Model-Driven Engineering Tech- Oscar Díaz. On the Application of Model-Driven niques. PhD thesis, Faculty of Informatics. Uni- Engineering in Data Reengineering. Information versity of Murcia, Spain, 2017. Systems, 72:136–160, 2017. [24] Erich Gamma, Richard Helm, Ralph Johnson, [9] Philip A. Bernstein, Alon Y. Halevy, and Rachel and John M. Vlissides. Design Patterns: El- Pottinger. A vision of management of complex ements of Reusable Object-Oriented Software. models. SIGMOD Rec., 29(4):55–63, 2000. Addison-Wesley Professional, 1994. [10] Philip A Bernstein and Sergey Melnik. Model [25] Stefan Gössner. JSONPath – XPath management 2.0: manipulating richer mappings. for JSON, July 2020. URL: https: In Proceedings of the ACM SIGMOD Int. Confer- //tools.ietf.org/id/draft-goessner- ence on Management of data, pages 1–12, 2007. dispatch-jsonpath-00.html.
28 A Unified Metamodel for NoSQL and Relational Databases A Preprint
[26] Hackolade Webpage, 2016. Accessed: Septem- Business, Technology, and Web (BTW), pages ber 2020. URL: https://hackolade.com/. 425–444, 2015. [27] Jean-Luc Hainaut. The transformational ap- [37] Michael J. Mior, Kenneth Salem, Ashraf Aboul- proach to database engineering. In Genera- naga, and Rui Liu. NoSE: Schema Design for tive and Transformational Techniques in Soft- NoSQL Applications. IEEE Trans. Knowl. Data ware Engineering, International Summer School, Eng., 29(10):2275–2289, 2017. GTTSE 2005, Braga, Portugal, July 4-8, 2005. [38] Pablo Muñoz, Carlos J. Fernández, Jesús Garcia- Revised Papers, pages 95–143, 2005. doi:10. Molina, and Diego Sevilla Ruiz. Extracting Phys- 1007/11877028\_4. ical and Logical Schemas for Document Stores. [28] Jean-Luc Hainaut, Vincent Englebert, Jean In CoMoNoS Workshop in Conceptual Modeling Henrard, Jean-Marc Hick, and Didier Roland. Int. Conf., 2020. Database Evolution: the DB-Main Approach. In [39] OrientDB Community Webpage, 2020. Accessed Pericles Loucopoulos, editor, Entity-Relationship September 2020. URL: http://orientdb.com/ Approach - ER’94, Business Modelling and Re- orientdb/. Engineering, 13th International Conference on the Entity-Relationship Approach, Manchester, [40] Trygve Reenskaug, Per Wold, and Odd Arild Working with objects - the OOram soft- UK, December 13-16, 1994, Proceedings, volume Lehne. ware engineering method 881 of Lecture Notes in Computer Science, pages . Manning, 1996. 112–131. Springer, 1994. [41] James E. Rumbaugh, Ivar Jacobson, and Grady The unified modeling language reference [29] Hbase webpage, 2007. Accessed September 2020. Booch. manual URL: https://hbase.apache.org/. . Addison-Wesley-Longman, 1999. [42] P.J. Sadalage and M. Fowler. NoSQL Distilled. [30] Alberto Hernández Chillón, Severino Feliciano A Brief Guide to the Emerging World of Polyglot Morales, Diego Sevilla Ruiz, and Jesús García Persistence. Addison-Wesley, 2012. Molina. Exploring the Visualization of Schemas for Aggregate-Oriented NoSQL Databases. In [43] Diego Sevilla Ruiz, Severino Feliciano Morales, ER Forum 2017, 36th Int. Conf. on Conceptual and Jesús García Molina. Inferring Versioned Modelling (ER), pages 72–85, Valencia, Spain, Schemas from NoSQL Databases and Its Appli- November 2017. cations. In 34th International Conference on Conceptual Modeling (ER), pages 467–480, Stock- [31] Alberto Hernández Chillon, Diego Sevilla Ruiz, holm, Sweden, October 2015. and Jesus Garcia-Molina. Discovering Entity Inheritance Relationships in Document Stores. [44] Peter Pin shan Chen. The entity-relationship 2021. Manuscript under review. model: Toward a unified view of data. ACM Transactions on Database Systems, 1:9–36, 1976. [32] Alberto Hernández Chillón, Diego Sevilla Ruiz, Jesús García Molina, and Severino Feliciano [45] Yannis Smaragdakis and et al. Scalable Satisfia- Morales. A Model-Driven Approach to Generate bility Checking and Test Data Generation From Schemas for Object-Document Mappers. IEEE Modeling Diagrams. Automated Software Engi- Access, 7:59126–59142, May 2019. neering, 16(1):73, 2009. [33] Alberto Hernández Chillon, Diego Sevilla Ruiz, [46] Apache spark webpage, 2021. Accessed and Jesús Garcia-Molina. Deimos: A Model- March 2021. URL: https://spark.apache. based NoSQL Data Generation Language. In org/. CoMoNoS Workshop in Conceptual Modeling Int. [47] David Steinberg, Frank Budinsky, Marcelo Pater- Conf., 2020. nostro, and Ed Merks. EMF: Eclipse Modeling [34] Jean-Marc Hick and Jean-Luc Hainaut. Strat- Framework 2.0. Addison-Wesley Professional, egy for database application evolution: The DB- 2009. MAIN approach. In International Conference on [48] Michael Stonebraker. The case for polystores. Conceptual Modeling, pages 291–306. Springer, ACM Sigmod Blog, 2015. URL: https://wp. 2003. sigmod.org/?p=1629. [35] David Kensche, Christoph Quix, Mo- [49] Dionysios C. Tsichritzis and Frederick H. Lo- hamed Amine Chatti, and Matthias Jarke. chovsky. Data Models. Prentice Hall Professional Gerome: A generic role based metamodel Technical Reference, 1982. for model management. J. Data Semantics, [50] Typhon Project. Hybrid Polystore Mod- 8:82–117, 2007. elling Language (Final Version). Techni- [36] Meike Klettke, Uta Störl, and Stefanie cal report, University of L’Aquila, 2018. Scherzinger. Schema Extraction and Structural URL: https://4d97e142-6f1b-4bbd-9bbb- Outlier Detection for JSON-based NoSQL Data 577958797a89.filesusr.com/ugd/d3bb5c_ Stores. In Conference on Database Systems for 3394b40f9cb54bcbb873f2c4ea1f2298.pdf.
29 A Unified Metamodel for NoSQL and Relational Databases A Preprint
[51] Allen Wang. Unified Data Modeling for Rela- tional and NoSQL Databases. infoq, 2016. URL: https://www.infoq.com/articles/unified- data-modeling-for-relational-and-nosql- databases/. [52] Lanjun Wang, Oktie Hassanzadeh, Shuo Zhang, Juwei Shi, Limei Jiao, Jia Zou, and Chen Wang. Schema management for document stores. Proc. VLDB Endow., 8(9):922–933, 2015. doi:10. 14778/2777598.2777601.
30 A Unified Metamodel for NoSQL and Relational Databases A Preprint LogicalPhysical (Conceptual separated) and modeled U-Schema models inNeo4j form graphs of Common strategy ofMapReduce 2 and steps: U-Schema model building process NoSQL stores Instances of the metamodel U-Schema Relational andand key-value) NoSQL Set ofNoSQL conceptsschemas to and cover relational Framework, EMF) Ecore / U-Schema / Models (use of mapping rules) (columnar, document, graph, (schemas of a data model) Conceptual and Logical Not addressedtion Structural variationscept are thechine leaning and usetics statis- are obtained of ma- Instances ofmodel (proprietary the solu- tion) meta- Entity, Property,Relationship and UnifiedModels metamodeldata models) (schemas / of Modeler (ModelSet) Modeling tooland columnar Database engineering toolkit (Physical separated) representation tions are supported, but not extracted or represented Programmatically, Collection,or Struct Block,tribute and (veryto physical At- model) near Abstract metamodel Database Design NoSQL databases Relational, document, Java instances / Instances Model elements play one of their roles Set of 48all data roles models to cover Use of role-baseding model- Anymalism metadataschema and database for- Table 5: Approaches defining a Generic Metamodel. ER diagrams and textNot addressed Own visualization N/A Not addressed N/A Unified ModelSet nota- No details provided, ex- Conceptual and Logical Conceptual and Logical Very simple uniform Modelannotated elementsconstruct with are belong meta- it to which constructsall data to models cover SuperMetamodel/Data Model/Schema (Own architecture) Anymalism metadataschema and database for- Not addressed NotER addressedtext diagrams and A schemafor extractor each system Not addressedNot addressed Not addressed Structural varia- Not addressed N/A No details provided MapReduce operation on Conceptual, Logical and Physical Selection of GERements el- andtion of defini- constraints GER based on ER SetFrom scratch of 13GER meta- and restrictionsdefine metamodel From data scratch to model From scratch N/A From scratch Ecore (Eclipse Modeling DB-MainEvolution toolRelational, OO, ER and Early Universal databases Metamodel Model Management GeRoMe Model Management Uniform access / SOS/NOAM ERwin Unified Data Schemaless supported Output Schemation extrac- ScalabilityPerformance and Schema Levels Defining concrete schemas Unifiedmodel Meta- Metamodeling Language Levelsmodeling of meta- Aim Supported Databases Paradigms
31