sign in sign up
<<
Home , Embedded database, Embedded system

International Journal of Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019

SQLite and its Application on Embedded Platform

Y.V. Sai Bharadwaj, Sai Bhageerath Yarrapatruni, Dr. Prasada Rao YVSSSV School of and Information Technology , University of Hyderabad National Institute of Technology, Warangal NRI Institute of Technology, Guntur, A.P.

Abstract are developed in python, International products like SQLite is an system, which is OracleLite of Oracle, DB2 and DB2 Satellite of IBM, popularly used in embedded environments like mobile Sybase iAnywhere of Sybase. devices, remote sensing, control systems, information Embedded database based on the open- appliances etc. It has turned out to be the focal point source platform like Berkeley DB, Mysql, SQLite for development works in related areas. Reliability, etc., that provide API‟ s which are easy-to-use, aid the robustness, security, fast and high efficiency, flexibility management and access of data and encourage the and so on, are unique to it among many other development of open-source . Adding, to the embedded . This paper outlines the basic emergence of research and application in some features, structure and the key technologies of main verticals like mobile devices, Communication embedded databases, analyses the characteristics, Systems, Automatic Test System (ATS) etc., the architecture and the functions of SQLite. development of efficient embedded databases will Also, a detailed from SQLite to ARM- become further important. Linux platform is discussed and a development case on In response to the requirement emerging in this home gateway over ARM-Linux is examined. hot reached vertical, major database vendors have developed their own products (embedded databases). Key-Words - Embedded Database; SQLite; Porting; Each have their own technical specifications and ARM-Linux; Home Gateway characteristics. Among these, SQLite has strongly I. INTRODUCTION attracted most of the developers due to its features like lightweight, portability, no copyright constraints With the advancement in information technology, a and so on. new development space for embedded database technology has opened up according to the II. FEATURES, FRAMEWORK, requirements of system. With the ARCHITECTURE AND THE KEY advancement of smart machines like Mobile Devices, TECHNOLOGIES OF EDS

Information Appliances etc., embedded database A. Features of Embedded Database Systems technology has entered into the application domain The architecture defines an embedded database from its earlier research vertical. Numerous variants of system [1]. It is a DMS which supports mobile blend of embedded systems and embedded database , and is integrated with specific Mobile technology are being worked on. Because of the and embedded Operating overreach of mobile devices, the demand for real time System, and runs on an embedded or mobile device. It data processing and data management has become the is also known as embedded mobile database as it need of the hour. This demand, has drawn the need for involves . Embedded database an efficient embedded database application into an technology combines several fields like the real time accelerating development phase. system, mobile communication, database, distributed Recently, many database products have appeared in computing etc. This has developed into a new area of the international arena such as SQL , Oracle, research in the vertical of database technology. The IBM DB2, Sybase etc. However, the power utilization structure of an , is shown in Fig.1. and memory consumption of the embedded systems are restricting these products from its full potential, especially when the volume of mobile users become elephantine. The traditional databases utilize enormous space and are much slow in real time, while on the other hand embedded database show exceptional performance in the concerned vertical. Currently, a large number of embedded databases have surfaced in both the domestic and international markets, and this has eased the development of various embedded devices. Domestic products include TinyDB, ZODB Fig.1 Embedded System Framework

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 1 International Journal of Computer Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019

An Embedded database, as a is The conceptual level provided a logic illustration bound to be restricted by various factors like speed, of DB. The external level is an interface between data application and other resources. Embedded database base system and users. Based on the structure of the system is very much similar to a traditional database external level, programmers store and operate data system, which can be hierarchical, network, and are not concerned with the logical illustration. relational, and also object-oriented database. The The internal (memory) level defines the physical typical distinction between the embedded database structure like the data types, indexes, methods of data and traditional database is that: an embedded database control and so on. The conceptual level is a is procedure driven where as a traditional database is middleware and restricts both the external and response driven (i.e., in an embedded database internal levels. respective API is called to access data, while in a Based on the need of Real-Time, conceptual level traditional response approach is can be bypassed (if real time demand is high) and used). An embedded database can‟t be considered as a directly deal with the physical data in the internal small scale traditional database, as an embedded (memory) level. database system and traditional database system are Embedded Database Management System is a not the same in the running environment, and have that manages the data in the their own characteristics [2][3][4]. embedded database system. It consists of three (1) Less space and memory requirement program modules, namely, the language The scale of the data structure (such as data tables, processing programs, system running control program records) is limited and less disk space is and the service program. All the operations on consumed. The memory utilization of an embedded DB are executed through the Embedded embedded system is also very little, so an DBMS. The three modules are described here [5]: embedded database can run in a finite memory (1) Language compiler processing program: It space. includes language processing and compiler (2) Reliability, robustness and security processing of DML all levels. As an embedded database is generally used in a (2) The system control program: It is used to mobile environment, the necessity is not only control the processes in the program that include, reliability, but also data integrity and security. real-time , data access, control and storage (3) Interoperability and portability into database, concurrent control program etc. Generally, an embedded database is specific to the (3) Service-oriented program: It included error application platform and the , so in recovery process, DB organization process and so on. order to ensure hassle free communication with other databases, interoperability is of great concern in the . The Key Technologies of EDS development process; at the same time portability The design of an embedded database, is balanced plays a major role in an embedded database (as OS and with the application system (Front end) and the data hardware platform used in various embedded systems which is a sub-set of the dataset (in the back end are different), which should be considered during server). So, for a robust, reliable and secured design phase. embedded database various key technologies are to be (4) Scalability considered during design phase [6]. An embedded database must be scalable, so as to (1) Replication and Synchronization achieve efficiency by reducing the disk space Data replication is employed when mapping utilization. with the DB at the back end server, so that data can be accessed by users any time. This replication of B. The Architecture of EDS data, necessitates synchronization between front Similar to that of a traditional database, there are end applications and back-end servers. three levels in an embedded DB: internal level, (2) recovery conceptual level and the external level [5], which is The backup process in an embedded DB is shown in Fig.2. different than that of a usual DB. It is in accordance with the procedure and is not independent of the service. (3) Transaction processing Processing of transactions in an embedded DBS is rationalized in the front-end. As accessing of required data does not necessitate hitting the back end DB every time, this helps in reducing the no of hits onto the back end server and in-turn saves time and improves the performance. But in the entire application system, it may need to be combined with Fig.2 Architecture of Embedded database System

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 2 International Journal of Computer Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019 the characteristics of mobile computing environment (7) It supports major programming languages to control and deal with transactions. including C/C++, PHP, , etc. These languages (4) Security interact with the DB using API call. Granting access control is the utmost need for an (8) SQLite employees more than 90% test embedded devices that is highly portable and mobile. coverage (i.e., test cases covering the complete The database system is precise with authority access. application code). (5) Quick start-up of the system There is a greater risk of fault occurrence in an 2.Data Structure of SQLite embedded system, so a Quick start up is ensured to list SQLite has several data structures to use in the every process running, this helps in the case of non- design of program. They are as follows: correction of Software Error. (1) In the program design, * XXX is (6) Real-time processing used to define a data structure pointing to a DB. To avoid processing delay, real time processing is SQLite is a data structure pointing to a database. to be employed and this forms the backbone (2) In the program design, sqlite_stmt *XXX requirement of an embedded system. is used to define a handle pointing to some SQL statements. Sqlite_stmt is a data structure that III. DATA STRUCTURES, ARCHITECTURE OF includes byte codes compiled by SQL statements SQLITE AND ITS API FUNCTIONALITIES and all the necessary resources which execute the byte codes. A. Features and Data Structure of SQLite (3) Sqlite_value is used to hold values of 1. Features of SQLite dynamic data type. SQLite is an open-source light weighted DB. It is written in C language. Compared to the traditional B. Analysis of SQLite Architecture databases like Oracle, SQL Server etc., SQLite does 1. SQLite Architecture not require additional components (as it encompasses Modular design is facilitated for up- complete embedded DB engine). It is suited for gradations and the complete DB is divided into embedded application development, and possess many several modules.The 8 primary subsystems are benefits, and hence it is preferred over many other depicted in Fig 3[8]. The architecture of SQLite embedded DB systems in the research and constitute three parts: Core, SQL Compiler and development domain. The main features are as follows Backend. In addition to this, Accessories are also [7][8][9]: included. (1) The open of this embedded DB is implemented in not more than 30,000 line of C code. In addition to this, its DB is compatible to binary formats and scales up to 2 terabytes in size. As this is open-sourced it minimizes the production costs. (2) Auto configuration and execution is an enabling feature of SQLite, as it does not require any thread to start and stop. Neither, does it require to create DB nor distribute access authority by the admin. At the time of system collapse, the restoration process involved in SQLite is automatic. Also, it provides easy-to-use API and accesses the DB directly through API functions. It also supports advanced languages.

(3) SQLite can access the database files on hard Fig.3 SQLite Architecture disk directly without calling an additional service.

Interoperability enables use of the same database files The top most level of SQLite interface is on different machines. implemented in ANSI C library, even though API‟s (4) SQLite does not differentiate data type i.e., it in different languages are used, C library is can assign any data to any column of any table, no executed in the bottom. The received SQL matter how the data is declared on the hard disk. statements from the interface, are decomposed into (5) SQLite enforces ACID properties and hence various identifiers by the tokenizer. Then the parser does not leave the system vulnerable during (lemon analyser) recomposes the identifiers, and unplanned collapse. inputs the result to the code generator to produce (6) SQLite is quick, scalable and with high VM code. The VM carries the VM code, and throughput. As it is platform independent, it can be finally completes the task of SQL statement. used on a variety of embedded OS, like uCLinux, SQL command has three independent Windows CE etc. components: Tokenizer, Parser and Code Generator.

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 3 International Journal of Computer Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019

The interface transfers these strings containing SQL statements to the tokenizer and the job of the tokenizer is to break the original string up into tokens and pass these broken tokens one by one to the parser. The Tokenizer and Parser transform the SQL statements into data structures that is handled at bottom layer. By using the parser generator, parser produces parse tree by recombining the tokens and transfers the parse tree to code generator. VM is the most important component of the internal structure of SQLite, which is called the Virtual Database Engine (VDBE), which is an engine designed to deal with library files. It performs not only operations like manipulation of data but also acts as an interface between the client and backend storage. The instructions of VDBE instruction are Fig.4 SQL statement Execution designated to complete various database operations, like insertion, deletion, querying, transaction (1) Preparation stage: Virtual Database Engine processing etc. [9]. It can also perform stack byte code is formed by compiling the SQL statements operations. The instruction sets of Virtual Database using the paser, tokenizer and code generator in the Engine analyses any SQL statement and firstly API function written in C. The compiler associates transforms it into corresponding with the sqlite_prepare() function and generates statements, and then completes the tasks given by the sqlite_stmt. sqlite_stmt includes Byte code, all the SQL statements. necessary resources required by executing SQL The sole works of Code generator is to convert the statement etc. parse tree into and then transfer it (2) Execution stage: Virtual DB Engine to the virtual machine for execution. implements the Byte-codes included in the The Virtual Database Engine executes the series sqlite_stmt step-by-step. This step by step process is of instructions of the mini-program (In assembly completed by the sqlite_step(). code) one by one, and fulfils the request as specified (3) Finalization stage: The execution of the SQL in the SQL statement. statement is still under process by the Virtual DB Backend includes three parts as depicted in Fig.3 Engine. Resources, are released during this time. [10][11]. The role of B-tree and pager is to transfer sqlite_finalize() determines this process. the data blocks to OS interface. B-tree is tasked to As shown in the above diagram, prepared, active order the data blocks, to ensure the relation among the and finalized represent the three stages of the data pages, so that it will be easy to locate. The data implementation. They imply the following: prepared blocks are stored in forms of B-Tree in the discs. The means the Virtual DB Engine Byte Codes are task of the pager is for data management, collapse prepared by compiling all the SQL statements. Active recovery etc., this requires collaboration with the OS means that the obtained Byte Codes are executed step interface. by step using sqlite_step() function. Finalized means OS interface provides a unified interface for porting that all the related resources are released after the end onto OS. As different OS use different methods to lock of the execution process. files, the task of OSI is to shield out these differences and provide an abstract layer for the other components C. SQLite API Functionalities of SQLite. SQLite uses many API function, which perform The architecture of SQLite as depicted in Fig 3 also various basic functions on the database, these are used includes Accessories, namely Utilities and Test code. for establishment of table, querying, modification, Utilities mainly perform no data type string insertion, deletion, sorting etc. The core functions of comparison, allocate memory, LEX functions, symbol API are four [12] which are listed herewith: table storage, functionalities of printing and the sqlite_open (), sqlite_exec (), sqlite_close (), and random number function of SQLite. sqlite_errcode(), which are used to execute SQL and Test Code: In case regression testing is counted on, acquire data, and realize efficient data storage and more than half of SQLite functions can be tested. management. In addition, it is extensible, allowing the During the system recovery, the fault recovery programmers to custom functions and pass on in the mechanism is simulated using os_test.c. form of callback. sqlite_open(): This establishes SQLite engine, where 2. SQL statement Execution using SQLite access mode and filename are provided. Then a SQLite is used to implement the execution callback function is implemented by SQLite for each ofSQL statements, this process involved three stages: record from the DB. Prepare, Step and Finalize which is shown in Fig.4.

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 4 International Journal of Computer Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019 sqlite_exec(): This deals with the result fetched after ported onto the ARM-Linux platform in order to executing the SQL statement. make DB application run on the ARM-Linux platform sqlite_close (): This function closes the DB files and (only then application development is possible). releases the SQLite engine. The following The porting of SQLite is possible and easy due to descriptions are the composition of four API its good characteristics, framework and architecture. functions. And as SQLite is developed in C language, it is (1) Sqlite_open (): highly portable. For this, on various platforms we perform cross compiling of the . For This function is used to open the database and porting SQLite onto ARM-Linux platform, establish SQLite engine. There are various standards corresponding ARM-Linux tools are needed and these of writing this function based on the UTF (Unicode tools are installed in usr / local / arm-lnx / bin / path. Transformation Format) being used. In case of UTF-8 The bottom OS is the ARM-Linux and porting is done encoding format we use: using 3 tools: arm-lnx-gcc, arm-lnx-ar and arm- intsqlite_open(const char *filename, sqlite3 lnxranlib. The specific course on porting are [12][13]: **ppDb); (1) Use command "echo PATH" to see if arm- lnx-gcc has been included in the PATH. But for UTF-16 encoding format we use: (2) Download the source code package from int sqlite3_open16 (const void *filename, sqlite3 http://www.sqlite.org/, then decompress the package. **ppDb) After the decompression process, the source code and a few attached files are available in the un-zipped The parameters used are FILENAME (name of the SQLite directory. file) and PATH. For permission on access mode we (3) To ensure that SQLite runs on ARM-Linux use two additional parameters: platform, a few necessary docs are to be int sqlite3_open_v2 (const char *filename, sqlite3 edited/modified. Rename Makefile.Linux-gcc to **ppDb, int flags, const char *zVfs) Makefile, and then edit the files as below { MODIFIED TO is denoted as [::-::] } ACCESS MODE determines (Read only/Read- Write/Create). If the file exists, SQLite_OK is returned TOP = .. / sqlite[::-::]read: TOP =.; by the OPEN function and ppDb parameter returns a TCC = gcc- O6 [::-::] read:TCC = armlnx-gcc-O6 legal database handle. In case, the file does not exist, it ; will establish the DB connection with the SQLite AR = ar cr [::-::] read: AR = arm-lnx-ar cr; engine and abnormal code is returned. RANLIB = ranlib [::-::] read: RANLIB = arm- lnx (2) Sqlite_close(): -ranlib ; intsqlite_close (sqlite3 *db) MKSHLIB = gcc-shared [::-::] read:MKSHLIB= This function shuts down the already opened DB arm- lnx -gcc-shared. connection with the SQLite engine. The db parameter determines the handle of the DB being closed. Delete the tclsqlite.o output file in the program (3) Sqlite_exec (): file, as the language binding with SQLite is intsqlite_exec (sqlite * db,char* not supported on ARM platform. ,int(*Callback)(void*,int,char **,char**),void *parg,char **errmsg); (4) Save all the modified files, and later run the SQL queries are dealt with this function which include commandsmake&&make installto generate sqlite.h, five parameters: database structure pointer, SQL libsqlite.aheader files. statement string, the first pointer point to Callback (5) Later, on the ARM-Linux platform run the functions, the first parameter pointer of Callback, the SQLite, for this copy it to the ARM board, and test a pointer to error string. program, and if the result is correct, it implies that (4) SQLite has been ported successfully onto the ARM intsqlite_errcode(sqlite*db): platform. After this application developments can be carried on this platform using SQLite. This Error handling function returns the error code. But for obtaining the error information we use: V. CONCLUSION const char * sqlite_errmsg(sqlite *); [for UTF-8] const char * sqlite_errmsg16(sqlite *)[for UTF-16] With the raising demand for intelligent appliances, the need for embedded database have become the IV. PORTING EDS (SQLITE) ONTO ARM- focal point of study. SQLite is a C language library LINUX PLATFORM and is open sourced, contains small core, DB is a Being an open sourced platform and advantaged with simple file, The DB files are easy to move and cross lightweight feature, SQLite has been the most widely platform, and is very suitable to build an EDS. Due to used EDS. It operates in three levels: external level, its unique advantages, SQLite has become the logic level and internal (storage) level. SQLite is mainstream database in embedded systems.

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 5 International Journal of Computer Trends and Technology ( IJCTT ) – Volume 67 Issue 2 – Feb 2019

In this paper, we have made a comprehensive analysis on the basic features, framework, architecture and key technologies of EDS, and discussed the key interface functions of SQLite, and then the porting of SQLite to ARM-Linux platform has been detailed; later discussed on the application of SQLite in home gateway using the API provided by SQLite. With enormous advantages, SQLite will have a wide spread use in fields like remote control, intelligent appliances, home medical equipment etc.

REFERENCES

[1] Chunyue Bi et al., “Research and Application of SQLite Embedded Database Technology”, WSEAS TRANSACTIONS on , Issue 1, Volume 8, January 2009, pp. 83-91. [2] B. Schneier, A. Shostack. Breaking up is hard to do: Modeling Security Threats for Smart Cards. USENIX Symposium on Smart Cards, 1999. [3] R. Munz: Usage Scenarios of DBMS, Keynote, 25th International Conference on Very Large Data Bases, Edinburgh, UK, 1999, http://www.dcs.napier.ac.uk/~vldb99/Industrial SpeakerSlides/SAPVLDB.pdf [4] Jiang MF et al., “Discovering Structure from Document Databases”, Lecture Notes in Computer Science, vol 1574. Springer [5] Rick F, et al., Introduction to SQL: Mastering the Relational Database Language, Addison Wesley, 2006. [6] Hector.Garcia-Molina, et al., Database System Implementation, Prentice Hall, 2001. [7] Mike Owens and Grant Allen, “The Definitive Guide to SQLite”, Apress 2nd edition, 2006. [8] SQLite homepage [EB/OL], http://www.sqlite.org. [9] Oracle Berkeley DB SQL API vs SQLite API- Integration, Benefits and Differences, Oracle White paper, Nov 2016, pp. 1-11. [10] Mike Owens, Embedding SQL Database with SQLite, Linux Journal, June 2003. [11] P. Bohannon, D. Lieuwen, R. Rastogi, A. Silberschatz, S. Seshadri, S. Sudarshan, "The Architecture of the Dalí Main- Memory Storage Manager", Tools and Applications, vol. 4, no. 2, pp. 115-151, 1997. [12] Michael A.Olson, Selecting and implementing an EDS, IEEE Computer, 2000,33(7). 27-34. [13] Ling-yun, Li-Jun, The Development and Application of Script Program Based on SQLite, China Academic Journal Electronic Publing House, 1994-2008. [14] Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2012-2017, Cisco, February 2013. [15] Chaudhuri, S., Narasayya, V., “AutoAdmin „WhatIf‟ Index Analysis Utility,” Proceedings of the ACM SIGMOD Conference, Seattle, 1998. [16] Rick F, van der Lans. Introduction to SQL: Mastering the Relational Database Language, Addison Wesley, 2006.

ISSN: 2231 – 2803 http://www.ijcttjournal.org Page 6