PAPER EMBEDDED DATABASE MANAGEMENT FOR IOT AND EDGE DEVELOPERS By John K. Waters www.actian.com :: PAPER EXECUTIVE SUMMARY The advent of the Internet of Things (IoT) and the proliferation of Edge Computing have presented software developers with a range of significant challenges—not the least of which is the management of the massive amount of data fueling these disruptive trends. Third parties, organizational systems, and billions of end-point devices—everything from industrial sensors and actuators to smart phones and autonomous vehicles—are generating unprecedented volumes and varieties of data at previously unheard of velocities. These “three Vs” define Big Data, and developers are expected to make the most of it in the IoT and Edge applications they build—not just manage it, but generate near real-time analytics at the point of action for business insights, while providing overall data governance and security. This analysis layer adds complexity for developers, and the old client-server models won’t give them what they need in this environment. Whether you’re push- ing your data to a terrestrial server right next door or the cloud, the performance lag inherent in a system that puts the app on one machine and the data it needs on another becomes a defeating factor when the data needs to be processed in near real time. In manufacturing, health care, and financial services, for example, milliseconds matter. Neither will the practice of simply manipulating data in some memory allocated space using file systems provide a solution going forward. The solution for a growing number of developers lies in the embeddable database, the database and management software that is effectively part of the application itself. This white paper examines the key concepts and enabling technologies of embeddable DBs that IoT and Edge developers need to understand. DEFINITIONS are becoming the norm, and developers Database expertise varies among devel- need to understand them. Marisha/ Shutterstock.com Image: By Cover opers, as does familiarity with the current terms of art in the IoT and Edge Comput- Embedded ing space, so it makes sense to define a The focus of this paper is the embed- few terms. The truth is, the database is dable database, which is a database (DB) no longer a hunk of shrink-wrapped soft- and various levels of database manage- ware deployed and tended by a database ment software integrated into, or very administrator (DBA). Data-intensive apps tightly bundled with, an application that [ 1 ] :: PAPER needs direct or fast access to the data it Edge Computing is manipulating. The DB and the app Edge Computing refers to distributed using it are running on the same machine, information architecture in which the and they communicate via procedure processing of data occurs where it is calls, so there’s very little latency. generated, far from a centralized data- For the purposes of this paper, the center, at the “edge” of the network. embedded database category does not That processing is done on IoT devices include in-memory databases (IMDB) in and remote gateways. Edge Computing which the data is stored entirely in the has also been described as a kind of main memory. Those systems are light, decentralized data persistence that simple, and fast, but provide no perma- occurs on or near the devices that nent storage. generate the data. THE SOLUTION FOR A GROWING NUMBER OF DEVELOPERS LIES IN THE EMBEDDABLE DATABASE, THE DATABASE AND MANAGEMENT SOFTWARE THAT IS EFFECTIVELY PART OF THE APPLICATION ITSELF. Also, an embedded database isn’t an Fog Computing, a term coined by embedded system, which is special- Cisco Systems in 2014, effectively means purpose software with a dedicated func- the same thing as Edge Computing, tion installed on a piece of hardware. though it’s more directly associated with As of this writing, embedded Cloud Computing. As Cisco defines it, the databases are used primarily by Fog “extends the cloud to be closer to independent software vendors (ISVs), the things that produce and act on IoT original equipment manufacturers data.” The term has been called a (OEMs), and systems integrators (SIs). marketing take on Edge, but it has been But the IoT and Edge ecosystem is catching on. (“Fogging” has even expanding at warp speed, and the become a verb.) market is likely to expand with it. Data analytics is the process of assess- IoT ing data for actionable insights. Embed- The Internet of Things is part of the ded analytics is the use of reporting and popular lexicon now, but just to be clear, analytic capabilities in transactional IoT refers to the vast and ever-expanding business applications. You can’t have network of physical devices linked embedded analytics without an through the Internet and via enterprise embedded database. intranets. There are billions of them now, [ 2 ] :: PAPER everything from smart phones to SQL temperature sensors, kitchen appliances The Structured Query Language (SQL) is to cars. The software installed on them the language most commonly associated gathers, stores, and analyzes data, either with relational databases. In fact, locally or through a connection with a “relational database” is almost separate database. synonymous with “SQL Databases.” An IoT gateway (sometimes called an SQL statements are used both for intelligent gateway or a control tier) can interactive queries for information from a take the form of a physical hardware relational database and for gathering data appliance or a piece of dedicated soft- for reports. It is considered a must-master ware. It serves as the connection between tool for developers working with the cloud and the universe of controllers, databases. sensors, and intelligent devices. NoSQL Database Management System A NoSQL database is a non-relational A database management system (DBMS) database—no tables with data organized is software that controls the storage, in rows and columns—and they come with retrieval, deletion, security, and integrity looser consistency models than traditional of data within a database. Embedded relational databases. NoSQL databases database solutions are sometimes called use a variety of data models, including embedded DBMSs. document, graph, key-value, in-memory, and search. They’re optimized for applica- Relational Database tions that require large volumes of data, A relational database is a DB that low latency, and flexible data models. organizes information into sets of tables with columns and rows. Each table Flat File contains data that relates to data in the A flat file is a plain text database from other tables. The relationships are which all word processing and other pre-defined and the data can be structure characters or markups have accessed or reassembled in different been removed. It contains a single table ways without having to reorganize the of data with one record per line. One of tables. These types of databases are the most common flat file examples is a designed for transactional and strongly comma-separated values (CSV) file, in consistent online transaction processing which table data is gathered in lines of (OLTP) applications. Relational database American Standard Code for Information management systems (RDBMSs) are Interchange (ASCII) text with the value preferred for OLTP, but not for online from each table cell separated by a analytical processing (OLAP). comma and each row represented by a new line. [ 3 ] :: PAPER “AS THE VOLUME AND VELOCITY OF DATA INCREASES, SO DOES THE INEFFICIENCY OF STREAMING ALL THIS INFORMATION TO A CLOUD OR DATACENTER FOR PROCESSING.” —SANTHOSH RAO, GARTNER ANALYST ACID Compliance a traditional datacenter or cloud. Gartner ACID (Atomicity, Consistency, Isolation, analyst Santhosh Rao summarized the and Durability) is an acronym for the four problem this trend is creating for IT attributes of a database transaction that organizations in a 2018 report (“What Edge guarantee its validity. All database trans- Computing Means for Infrastructure and actions must be ACID Compliant to Operations Leaders”): “As the volume and ensure data integrity. The attributes are: velocity of data increases, so does the Atomicity: in a transaction involving inefficiency of streaming all this information two or more discreet parts, if a part fails, to a cloud or datacenter for processing.” the whole transaction fails. And virtually every industry on the Consistency: data written to the data- planet is poised to invest significant base as part of the transaction must resources in reaction to these trends. adhere to all defined rules and restric- In another recently published report tions. (“Worldwide Semiannual Internet of Isolation: a transaction in process and Things Spending Guide”), IDC analysts not yet committed must remain isolated predict that IoT spending will reach $1.2 from any other transaction. Trillion in 2022. Durability: all the changes made to the These predictions should reassure database are permanent once a transac- developers considering their own IoT and tion is successfully completed. Edge strategies. While developers may be most interested in leveraging APIs that THE LANDSCAPE have functionality behind them that Why should developers care about IoT, delivers their prescribed outcomes, when Edge Computing, and all this arcane data- it comes to APIs for management of the base lore? They’re not DBA’s, after all, and data within their IoT and Edge applica- they’re already coping with some signifi- tions, investing in technologies that cant challenges associated with acceler- enable data processing closer to the edge ating application release cadences. of the network allows organizations to Ignoring these trends is simply not an analyze that data in near real-time, which option. According to the industry analysts has quickly become an essential capability at Gartner, within the next four years 75 across many industries, including percent of the data generated in the enter- manufacturing, health care, telecommuni- prise will be created and processed outside cations, and finance.