D4M: Bringing Associative Arrays to Database Engines Vijay Gadepally†‡, Jeremy Kepner†‡+, William Arcand†, David Bestor†, Bill Bergeron†, Chansup Byun†, Lauren Edwards†, Matthew Hubbell†, Peter Michaleas†, Julie Mullen†, Andrew Prout†, Antonio Rosa†, Charles Yee†, Albert Reuther† † MIT Lincoln Laboratory, ‡ MIT Computer Science and Artificial Intelligence Laboratory, + MIT Math Department and variety. Numerous tools exist that allow users to store, Abstract—The ability to collect and analyze large amounts of query and index these massive quantities of data. Each storage data is a growing problem within the scientific community. The or database engine comes with the promise of dealing with growing gap between data and users calls for innovative tools complex data. Scientists and engineers who wish to use these that address the challenges faced by big data volume, velocity systems often quickly find that there is no single technology and variety. Numerous tools exist that allow users to store, query that offers a panacea to the complexity of information [2, 3]. and index these massive quantities of data. Each storage or When using multiple technologies, however, there is significant database engine comes with the promise of dealing with complex data. Scientists and engineers who wish to use these systems trouble in designing the movement of information between often quickly find that there is no single technology that offers a storage and database engines to support an end-to-end ap- panacea to the complexity of information. When using multiple plication. In this article, we present the Dynamic Distributed technologies, however, there is significant trouble in designing the Dimensional Data Model - a technology developed at MIT movement of information between storage and database engines Lincoln Laboratory. Previous articles on D4M [4, 5] have to support an end-to-end application along with a steep learning showcased the ability of D4M to interact with the popular curve associated with learning the nuances of each underlying Apache Accumulo database. Recent advances in D4M now technology. In this article, we present the Dynamic Distributed allow D4M to operate on a variety of back end storage or Dimensional Data Model (D4M) as a potential tool to unify database engines while providing a federated look to the end database and storage engine operations. Previous articles on user through the use of associative arrays. Associative arrays D4M have showcased the ability of D4M to interact with the popular NoSQL Accumulo database. Recently however, D4M provide a mathematical interface across different database now operates on a variety of backend storage or database technologies and can help solve one of the largest problems of engines while providing a federated look to the end user through working with numerous backend storage or database engines the use of associative arrays. In order to showcase how new - how do we correlate information that may be spread across databases may be supported by D4M, we describe the process of different storage or database engines? building the D4M-SciDB connector and present performance of The Intel Science and Technology Center (ISTC) on Big this connection. Data [6] is centered at the MIT Lincoln Laboratory and Keywords—Big Data, Data Analytics, Dimensional Analysis, supports five major research themes: Big Data Databases and Federated Databases Analytics, Big Data Math and Algorithms, Big Data Visual- ization, Big Data Architectures, and Streaming Big Data. One I. INTRODUCTION of the core goals of the ISTC is to develop the next generation The challenges associated with big data are commonly software stack required to manage heterogenous data in order referred to as the 3 V’s of Big Data - Volume, Velocity to enable large scale data analytics on data from the Internet and Variety [1]. The 3 V’s provide a guide to the largest of Things (IoT). This solution stack is known as the Big outstanding challenges associated with working with big data Data Working Group (BigDAWG) stack [7]. The BigDAWG arXiv:1508.07371v1 [cs.DB] 28 Aug 2015 systems. Big data volume stresses the storage, memory and solution stack is a vertically integrated stack that supports compute capacity of a computing system and requires access to numerous hardware platforms, analytics libraries, database a computing cloud. The velocity of big data stresses the rate at and storage engines, software development through the Big which data can be absorbed and meaningful answers produced. Dawg Query Language (BQL) and Compiler, visualization Big data variety makes it difficult to develop algorithms and and presentation of data through a variety of applications. tools that can address that large variety of input data. The BQL will provide software and analytics developers an The ability to collect and analyze large amounts of data abstraction of the underlying database and storage engines, is a growing problem within the scientific community. The analytics libraries and hardware platforms. A key feature of growing gap between data and users calls for innovative tools BQL is to develop the API required to provide a federated that address the challenges faced by big data volume, velocity look to developers. Vijay Gadepally is the corresponding author and can be reached at vijayg Federated databases have the ability to abstract away details [at] mit.edu. about the underlying storage or database engine. Very often, This material is based upon work supported by the National Science Founda- federated databases are used to provide some mutual benefit. tion under Grant No. DMS-1312831. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and This feature can be quite appealing to scientists who wish to do not necessarily reflect the views of the National Science Foundation. write complex analytics and are not necessarily database or 978-1-4799-6233-4/14/$31.00 ©2014 IEEE storage experts. There has been much promise of federated databases [8]. Federated databases provide the ability to give array by simply adding and subtracting values. With this users the feel of a data warehouse without physically moving definition, it is assumed that only a finite number of tuples data into a central repository [9]. As an example of a federated will have values, and all other tuples have a default value database, consider Myria [10, 11], a distributed database that of the additive-identity/multiplicative-annihilator 0. Further, uses SQL or MyriaL as the language all of which was devel- the associative array mapping should support operations that oped at the University of Washington. One of the challenges resemble operations on ordinary vectors and matrices such as in database federation has been in developing a programming matrix multiplication. In practice, associative arrays support API that can be used to interact with the ever-increasing variety a variety of linear algebraic operations such as summation, of databases and storage engines [12]. union, intersection, multiplication and element wise operations. D4M’s mathematical foundation, associative arrays, have Summation of two associative arrays, for example, that do not the ability to to help alleviate the challenges associated with have any common row or column key performs a union of open problems in federated database. Having a one-to-one their underlying non-zero keys. Element wise multiplication as relationship with triple store or with key-value store systems an example performs an operation similar to an intersection. allows a flexible representation that can be supported by many Associative arrays have a one-to-one relationship with key- databases. The ability to perform linear algebraic operations on value store or triple store databases, sparse matrices, and associative arrays (and thus data stored in different database adjacency or incidence matrix representations of graphs. These engines) opens up big-data analytics to non-computer scien- relations allow complex datasets to be easily converted to tists. We believe that an API based on mathematical operations associative array representation. Linear algebraic operations on is easy to learn. The software implementation in popular associative arrays can be used to perform graph algorithms as languages such as MATLAB, Octave, and Julia allows the described in [13]. rapid prototyping of new and complex analytics with minimal NoSQL database tables can be exactly described using the effort. mathematics of associative arrays [14]. In the D4M schema, In this article, we present our work on developing asso- a table in a NoSQL database, such as Apache Accumulo, is ciative arrays as the datatype for big data in Section II. In an associative array. In this context, the primary differences Section III, we present D4M and provide examples of how between associative arrays and sparse matrices are: associative database operations such as context and cast can be done array entries always carry their global row and column labels with D4M and associative arrays through the D4M MATLAB while sparse matrices do not. Another difference between toolbox. In Section IV, in order to motivate the ease at which associative arrays is that sparse matrices can have empty rows new database support can be built into D4M, we detail the or columns while associative arrays do not. D4M-SciDB connector. In order to demonstrate the use of Using associative arrays as a datatype for big data has many D4M, associative arrays, and database engines, we provide a benefits such as: simple case study of developing an analytic for medical data • Using associative arrays as the base datatype will that spans across three different storage engines in Section V. make database development easier. DB developers Finally, we conclude in Section VI. will only need to provide an optimized interface to associative arrays; II. ASSOCIATIVE ARRAYS • Associative arrays can limit the programming Associative arrays are used to describe the relationship language-DB connectors that are required. Currently, between multidimensional entities using numeric/string keys if there are N programming languages and M database and numeric/string values.