CORAL, A SOFTWARE SYSTEM FOR VENDOR-NEUTRAL ACCESS TO RELATIONAL DATABASES I. Papadopoulos, R. Chytracek, D. Dullmann,¨ G. Govi, IT Department, CERN, Geneva, Switzerland Y. Shapiro, ATLAS Database Group, PH Department, CERN, Geneva, Switzerland Z. Xie, Princeton University, Princeton, New Jersey 08544 USA Abstract user from the technology-specific APIs. The need for the maximal SQL insulation from the client software can be The COmmon Relational Abstraction Layer demonstrated simply showing the SQL statements for two (CORAL)[1] is a C++ software system,developed within technologies (Oracle[5] and MySQL[6]), which are used the context of the LCG persistency framework, which mostly in HEP, in rather simple tasks: provides vendor-neutral software access to relational databases with defined semantics. The SQL-free public in- 1. Creation of a table: terfaces ensure the encapsulation of all the differences that one may find among the various RDBMS flavours in terms MySQL syntax: ¡£¢¥¤§¦©¨ ¤ ¨©¦ ¤¨ of SQL syntax and data types. CORAL has been developed ¨§! #"! ¤$ following a component architecture where the various RDBMS-specific implementations of the interfaces are Oracle syntax: % loaded as plugin libraries at run-time whenever required. ¡£¢¥¤§¦©¨ ¤ ¨©¦ ¤%&¨ ' ,+.- 1 $/.0 ¦2¢43 ! 5" ¤$ The system addresses the needs related to the distributed "() £¤*¢ deployment of relational data by providing hooks for client-side monitoring, database service indirection and 2. Query fetching only the first rows of the result set: application-level connection pooling. MySQL syntax: 6 ¤©¤7¡8¨9:4¢8 #(;¨ ( ¨>= INTRODUCTION #¢¥ ¤*¢< ©3 CORAL is the set of software deliverables of the Oracle syntax: 6 ”Database Access and Distribution” work package of the ¤©¤7¡8¨@?A:4¢8 #( 6 % #¢¥ ¤*¢< ©3 $ POOL project[2]. The development of libraries for the ¤©¤7¡'¨9B:C¢* D(;%&¨ ¤*¢¥¤G¢8 "H(JILK E vendor-independent database access, collectively referred E>F to as the Relational Abstraction Layer (RAL), started within POOL in spring 2004. At that time the main require- The CORAL approach is to present an identical API for ment that needed to be met was the creation of an insulation such cases, which allows the development of software com- layer that would allow the development of software com- ponents that can be used without any code modification or ponents responsible for accessing data in RDBMS with- conditional constructs against multiple relational technolo- out the knowledge of the subtle differences among the gies. various technology-specific solutions. Since then RAL CORAL is expected to be used by applications run- was adopted by the relational components of POOL (File ning on a grid-enabled and distributed environment. It Catalog, Collections), the COOL (Conditions Database) is therefore defining the necessary interfaces for database project[3], and several components of the software frame- service indirection, multiple authentication mechanisms works of the LHC experiments. (certificate- and user/password pair-based), client-side con- In spring 2005 a formal review took place of the existing nection pooling and client-side monitoring. RAL API with feedback from the direct clients of RAL, CORAL is not a general purpose C++ connectivity li- within and outside POOL. During the review new emerg- brary, such as ODBC, JDBC, the python or perl DBI. It is ing use cases have been considered that are related to the neither a system to accomodate all access patterns to rela- database deployment and distrubution issues. The outcome tional data. Its scope is restricted to serve primarily the use was the design of an improved version of RAL, which is cases relevant to the data handling and analysis of the LHC now being developed and packaged independently of the experiments. rest of the POOL components under the new name CORAL (COmmon Relational Abstraction Layer). ARCHITECTURAL CHOICES PROJECT SCOPE The architecture of CORAL is component-based. The overall system consists of a set of abstract interfaces and The primary goal of CORAL is to provide function- several implementation components which have no depen- ality for accessing data in relational databases using a dency on each other. Data and control flow only through C++ API, free of SQL commands and types, shielding the the abstract interfaces in a user application. EnvironmentAuthenticationService MonitoringService OracleAccess XMLAuthenticationService MySQLAccess ConnectionService SQLiteAccess XMLLookupService FrontierAccess LFCLookupService RelationalService Developer−Level Interfaces and Common Implementations (CoralCommon) User−Level Interfaces (RDBMS Access, Types, Row Buffers, etc.) Client Software (RelationalAccess, CoralBase) Figure 1: Component architecture of the CORAL framework. The technology insulation is achieved through the set of tual data, as this is defined by the contact string, is fully the abstract interfaces; a client component depends only on decoupled from the authentication parameters and mecha- them. The functionality expressed by each interface is min- nism that are required for accessing them. imal but complete. The component architecture and the fact A user may enable client-side monitoring for a stated that the functionality expressed by each interface is mini- verbosity level. In this case the RDBMS plugin searches mal but complete, enable the parallel and independent de- for a loaded implementation of the IMonitoringService in- velopment of the various CORAL components, as well as terface and registers information such as begin and end of additional and user-provided implementations of the same user sessions, signaling of transaction boundaries and the interfaces. They also facilitate efficient unit and integration response times of the various SQL statements that are is- testing. sued. The static view of the CORAL architecture is shown in Some RDBMS implementations that are used for read- Fig. 1. The realization of the component architecture has only purposes be based on a web-cache server for the been based on the plugin management and the component fast delivery of the query results. With the use of the framework of the SEAL[4] project. The latter allows the IWebCacheControl interface a user may fully control the construction of context hierarchies onto which the various caching policies in order to ensure consistency of the data implementation components are hooked. In this configura- that are retrieved. tion if a component requires the functionality of a particular Alternatively to the format specified above, a user may interface, it simply queries its context tree for an available choose a free format which indicates a logical name for a implementation. database service. In this case a query to an implementation of the ILookupService interface has to be issued in order FUNCTIONALITY OF THE PUBLIC API to obtain the corresponding connection string that corre- sponds to an actual service. The best way of doing so is to A user may connect to a database schema by prividing a use an implementation of the IConnectionService interface contact string with the following format: which coordinates the control flow among the above com- MONQPSRUT/V&W VYX[Z \S]^V[M_V`PaV&Wcb dYdUR/VYefMgT[h`iNYbc\SV`]M TUjUi kYNS]d[e_PSR[NSih T[hUiHN ponents and provides application-level connection pooling. The technology may take values such as oracle, mysql, After having retrieved a valid handle to a database sqlite, etc. The protocol is optional and used only on em- schema the CORAL API provides the user with a large sub- bedded technologies such as SQLite[7]. It may take values set of the functionality that is possible through generic SQL such as file, http, etc. and RDBMS-specific connectivity APIs: The loaded implementation of the IRelationalService in- terface parses the connection string and searches for avail- Schema definition and manipulation operations, such able plugins providing the RDBMS functionality for the as creating, droping and redefining tables and views, particular technology. No authentication credentials ap- as well as defining indices, keys and constraints. It is pear in the connection string. These are provided by the also possible to fully describe a given schema and its loaded implementation of the IAuthenticationService inter- elements. face to the RDBMS plugin whenever an authentication is Columns are defined by specifying the correspond- attempted. This feature ensures that the location of the ac- ing C++ types (eg. int, float, double, string) and some storage hints such as the allocated size for string Two implementations of the interface set related to the variables. The translation into the actual SQL types retrieval of authentication credentials: one based on that are eventually used is performed by the RDBMS- XML files and another on environment variables. specific plugin which is employed for the technology in use. An implementation of the interface set responsible for peforming the necessary technology dispatching given Manipulating data in tables such as inserting, modify- a connection string and the available RDBMS-specific ing and deleting rows. Operations with BLOB types implementations found at run time. are facilitated through a Blob C++ type provided by CORAL. An XML-based implementation of the interface set re- The API supports the execution of such operations sponsible for peforming logical to physical database in bunches, where only the input data change be- service lookup operations. tween subsequent iterations. This minimizes the total An implementation based