Embedded databases & Berkeley Shubham Batra, Nathan Liccardo December 18, 2017 Contents 1 Introduction 4 1.1 An introduction to data management . 4 1.2 Data access and data management . 4 1.3 Network Database . 5 1.4 Clients and servers . 5 2 Embedded databases 6 2.1 Introduction . 6 2.2 Definition . 6 2.3 Properties of embedded databases . 6 2.3.1 Minimisation of footprint . 6 2.3.2 High availability . 6 3 Berkeley Database 7 3.1 Introduction . 7 3.2 What is Berkeley Database ? . 7 3.2.1 Berkeley DB Library . 7 3.2.2 Berkeley DB Architecture . 7 3.3 Records & Requests . 8 3.4 Berkeley Database . 9 3.4.1 Creating Database . 9 3.4.2 Creating Table . 9 3.4.3 Linking Table to Data . 10 3.4.4 Inserting Data to Table . 11 3.4.5 Updating Data to Table . 11 3.4.6 Retrieving Data from the Table . 12 3.4.7 Example . 13 3.5 Data Access Services . 14 3.6 Data management services . 14 3.7 The Berkeley DB products . 14 3.8 Available access methods . 15 3.9 Berkeley DB subsystems . 15 3.10 Programming model . 16 3.11 Programmatic APIs . 16 3.12 Security . 17 3.13 Encryption . 17 3.14 Berkeley Database Customers . 18 3.14.1 Amazon.com . 18 3.14.2 Motorola Wireless network . 18 3.14.3 Cisco Systems Broadband Provisioning Register . 18 3.14.4 Openwave Mobile Messaging . 18 3.14.5 AT&T Network . 19 3.14.6 Bitcoin Database . 19 2 4 Conclusion 20 3 1 Introduction In this document, we are going to talk about embedded databases. The report will be separated into two main parts. We will firstly define and understand what is precisely an embedded database and why we are more and more speaking about this kind of databases today. Based on each previous definitions, we will then describe a real embedded database system named Berkeley. Those explanations would be accompanied by some of the most know embedded applications : Motorola,OpenWave & AT-T Network. 1.1 An introduction to data management Cheap, powerful computing and networking have created countless new applications that could not have existed a decade ago. The advent of the World-Wide Web, and its influence in driving the Internet into homes and businesses, is one obvious example. Equally impor- tant, though, is the shift from large, general-purpose desktop and server computers toward smaller, special-purpose devices with built-in processing and communications services. As computer hardware has spread into virtually every corner of our lives, of course, software has followed. Software developers today are building applications not just for conventional desk- top and server environments, but also for handheld computers, home appliances, networking hardware, cars and trucks, factory floor automation systems, cell phones, and more. Many computer systems must store and retrieve data to track history, record configuration set- tings, or manage access. Data management can be very simple. In some cases, just recording configuration in a flat text file is enough. More often, though, programs need to store and search a large amount of data, or structurally complex data. Database management systems are tools that programmers can use to do this work quickly and efficiently using off-the-shelf software. Data storage is a problem dating back to the earliest days of computing. 1.2 Data access and data management Fundamentally, database systems provide two services. The first service is data access. Data access means adding new data to the database (inserting), finding data of interest (searching), changing data already stored (updating), and removing data from the database (deleting). All databases provide these services. How they work varies from category to category, and depends on the record structure that the database supports. Each record in a database is a collection of values. For example, the record for a Web site customer might include a name, email address, shipping address, and payment information. Records are usually stored in tables. Each table holds records of the same kind. The second service is data management. Data management is more complicated than data access. Providing good data management services is the hard part of building a database system. When you choose a database system to use in an application you build, making sure it supports the data management services you need is critical. Data management services include allowing multiple users to work on the database simultaneously (concurrency), allowing multiple records to be changed instan- taneously (transactions), and surviving application and system crashes (recovery). 4 1.3 Network Database The "network model" is a fairly old technique for managing and navigating application data. Network databases are designed to make pointer traversal very fast. Every record stored in a network database is allowed to contain pointers to other records. Reorganization is often necessary in databases, since adding and deleting records over time will consume space that cannot be reclaimed without reorganizing. Without periodic reorganization to compact network databases, they can end up with a considerable amount of wasted space. 1.4 Clients and servers Database vendors have two choices for system architecture. They can build a server to which remote clients connect, and do all the database management inside the server. Alternatively, they can provide a module that links directly into the application, and does all database management locally. In either case, the application developer needs some way of communi- cating with the database (generally, an Application Programming Interface (API) that does work in the process or that communicates with a server to get work done). 5 2 Embedded databases 2.1 Introduction Our smartphones are the most striking example of the use of embedded databases. But, the exact notions of embedded systems and databases are often misunderstood and confused. The most common mistake is the confusion between Real-time systems and Embedded systems. In fact, they are absolutely not the same but are usually used together. 2.2 Definition Definition. An embedded database is a DBMS which is directly linked with an application. The database is completely hidden for the application's end-user and requires little mainte- nance. In other words, the main property of every embedded database is the hidden aspect. This type of architecture will built the DBMS into the application rather than provides external management. For this reason, in many cases every database interventions are directly man- aged from the application. Nowadays, we also often use the term of embedded database to designated embedded ap- plications on mobile devices. This definition is, however, not complete. In fact, real-time and embedded applications are only a small subset of embedded database systems. Embedded databases are not related to embedded platforms and can run on mobile devices but also on laptop and desktop. The only condition to be embedded is that the database is directly linked into the application. 2.3 Properties of embedded databases In traditional entreprise, the main objectives are mainly the flexibility, scalability, function- ality, etc. While we are talking about embedded systems (and databases), we are focusing on resources and processor usage which are relatively not important for traditional entreprise due to relatively cheap price of hardware. 2.3.1 Minimisation of footprint A footprint is commonly used to designated the amount of space a particular unit of hardware or software occupies. Embedded systems and databases focus on the use of smallest footprint. As example, a typical footprint size is within the range of kilobytes to megabytes. 2.3.2 High availability In embedded databases, we commonly do not have access to system administrator during the execution and after the system is deployed. In fact, we only have access to the database administration during the initial configuration and the ongoing maintenance would be realised by the program itself. The high availability added the considerations that the database would continue to function normally even when the hardware or the network failures within the system. 6 3 Berkeley Database 3.1 Introduction So far, we have discussed database systems in general terms. It is time now to consider Berkeley DB in particular and see how it fits into the framework. The key question for this part will be the following : what kinds of applications should use Berkeley DB ? 3.2 What is Berkeley Database ? Berkeley DB is an embedded database library that provides scalable, high performance, transaction-protected data management services to applications. It provides also a simple function-call API for data access and management. Berkeley DB is embedded because it links directly into the application and runs in the same address space as it. 3.2.1 Berkeley DB Library As a result, no inter-process communication, either over the network or between processes on the same machine, is required for database operations. Berkeley DB provides a simple function-call API for a number of programming languages, including C, C++, Java, Perl, Tcl, Python, and PHP and all database operations happen inside the library. Multiple processes, or multiple threads in a single process, can all use the database at the same time as each uses the Berkeley DB library. Low-level services like locking, transaction logging, shared buffer management, memory management, and so on are all handled transparently by the library. The Berkeley DB library is extremely portable. It runs under almost all UNIX and Linux variants, Windows, and a number of embedded real-time operating systems. It runs on both 32-bit and 64-bit systems. It has been deployed on high-end Internet servers, desktop machines, and on palmtop computers, set-top boxes, in network switches, and elsewhere. 3.2.2 Berkeley DB Architecture Once the databse is linked into the application, the end user generally does not know that there is a database present at all.