DDaattaa AAcccceessss wwiitthh LLiinnuuxx l o i b b g ODBC o GNOME d n a o B Copyright Statement © 2001 Adam Tauno Williams ([email protected]) Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. You may obtain a copy of the GNU Free Documentation License from the Free Software Foundation by visiting their Web site or by writing to: Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. Credits The assistance of the following people was imperative to the completion of the presentation: Nick Gorham of the unixODBC project Rodrigo Moya of the GNOME-DB project Vivien Malerba of the gASQL project Giancarlo Capella, author of dbreport I was privileged to have e-mail exchanges with all of the above developers about various issues. They were consistently helpful, overlooking my ignorance, and were excellent representatives of the Open Source ethic of community. Thanks also is in order to the local Linux users' group http://www.kalamazoolinux.org The Topic There are a great variety of relational database systems available for Linux, both Open Source and proprietary. They range in features from the engineless SQLite, to the read-oriented MySQL, to the ACID compliant PostgreSQL, and to incredibly expensive Oracle, Informix, and DB2 Then there are LDAP servers, mail servers, syslog files, NNTP servers, XML files, delimited files, fixed-length files, text files, spreadsheets, etc... All these databases, servers, and files represent one thing: data, the real reason we have computers in the first place. The need, not only to record, but to sort, calculate, report, correlate, and "mine", is the basis of Information Technology. But while there seems to be no shortage of information on the "data sources" available for the Linux platform, corresponding information about tools to use that data seems to me to be lacking. The Scope This presentation is focused toward the Linux "power user", and attempts to provide a working understanding of the technologies and tools available for data access on the Linux platform. Hopefully, it has enough depth to be of interest to budding developers as well. The focus is on data-source independent tools. Therefore, tools linked to a specific database (or other source such as an LDAP server) are not covered. Along that same line, ODBC and GDA which enable most such tools, are covered "in-depth". Since the topic is restricted to data access on the Linux platform it does not include multi-platform data access tools written in Java or using the JDBC technology. Java is a registered trademark of Sun Microsystems Inc. MMeetthhooddss The overview Application Application Application O Bonobo A F / C Gnome DB O R ODBC Driver B Manager Libgda A Native ODBC Driver Library Text Any RDBMS NNTP Mail LDAP RDBMS File CORBA Pros & Cons Native Libraries Native Libraries Fastest Every DBMS has a different API. Fewest Dependencies Vendor may change API at will. Usually Portable Developer may need to deal with a lot of low-level issues. ODBC Uniform API ODBC App becomes quasi-independent Less efficient than native libraries of backend data store. Requires a driver/provider. Drivers available for non-relational Drivers/Providers vary in quality. data stores More dependencies GNOME DB (GDA) GNOME DB (GDA) Objects and signals replace APIs. Under active development Inherent scalability All disadvantages of ODBC Support for Batch Jobs All advantages of ODBC Terms Manager l Both ODBC and GDA have an application and a . r library known as a manager that knows about available drivers / s e v m providers, and allows the definition of new data sources as well as e y s n creating connections/instances with defined data sources. e o c r c u Driver/Provider a ODBC (drivers) and GDA (providers) accomplish d o d r pretty much the same purpose. They match up the ODBC or GDA t n a n API with that of a given datasource. Each type of data source used i , s (PostgreSQL, Oracle, Text File) requires a driver/provider. t A p D e c G DSN Both ODBC and GDA use the concept of a DSN n d o (Data Source Name) thar corresponds to a defined data source. All n c a , s the connection parameters are defined via the manager and are loaded C m when the application connects to a defined DSN. B r e D t O w e h t n o B NNaattiivvee LLiibbrraarriieess Native Libraries Native libraries are usually provided in a SDK by Application the producers of the database engine. The libraries are linked to the application when it is compiled. Due to the "direct" nature, native libraries are usually the fastest way to communicate with a database engine. Native An example is "libpq.so" to which PostgreSQL Library applications are linked, with a "-lpq" directive to the compiler. RDBMS FreeTDS (http://www.freetds.org) FreeTDS is an LGPL'd re-implementation of the Tabular Data Stream protocol (versions 4.2, 5.0, and 7.0) used by Microsoft SQL server and Sybase products. FreeTDS is used as the basis of unixODBC and Perl DBI drivers as well as a GDA provider. When you build FreeTDS, you can specify the default version of the protocol you wish to use. This depends upon the product to which you desire to connect. --with-tdsver={4.2, 4.6, 5.0, 7.0} This can be overridden at run time by using the TDSVER environment variable or the config file which permits you to specify the version used for a particular server. MDB Tools (http://mdbtools.sourceforge.net/) MDB Tools is a collection of utilities for working with Jet 3 (Access '97) and Jet 4 (Access 2000, Access 2002) data- bases. In version 4, a small SQL engine for use with MDB files as well as an ODBC driver for unixODBC is included. The libraries are released under LGPL while the utilities are released under GPL. Access, Jet 3, and Jet 4 are registered trademarks of Microsoft Corporation Inc. OODDBBCC ODBC Developed by the Microsoft corporation as a way for applications to access a variety of backend databases without source code modification “Maximum interoperability” Based upon open standards X/OPEN ISO/IEC Has become a ubiquitous standard, even on SQL non-Microsoft platforms, as is included with most Linux distributions. All the API translation, etc... is performed on the client making ODBC drivers platform specific. unixODBC http://www.unixodbc.org unixODBC is an implementation of ODBC for GNU Application platforms (Linux, FreeBSD, etc...) as well as UNIX (Sun, AIX, etc...). unixODBC is distributed under the LGPL so that commercial solutions can be based upon it without license concerns. ODBC Driver Manager unixODBC is compatible with the ODBC 3.5 standard, ODBC Driver can perform ODBC 3 to ODBC2 translation, as well as Native UNICODE to ANSI translation. Library Text NNTP File RDBMS Installing unixODBC 1. Download tar file from website. 2. Unpack tar file. tar -xzvf unixODBC-2.0.4.tgz 3. Configure. ./configure --prefix=/usr/local/odbc --with-qt-dir=/usr/lib/qt-2.2.0 Almost every 4. Compile. make current Linux 5. Install. make install distribution includes unix- 6. Add library to library path. echo "/usr/local/odbc/lib" >> /etc/ld.so.conf ODBC as a 7. Rebuild library index. /sbin/ldconfig package. /usr/local/odbc bin lib include etc Config tools ODBC Driver Header System-wide and command Manager and Files Drivers and DSNs line ODBC Drivers. client. ODBC Drivers Included Drivers PostgreSQL: libodbcpsql.so A normal unixODBC driver consists of libodbcpsqlS.so two shared libraries, a driver library and a setup library. NNTP: libnn.so libodbcnnS.so Typically the setup library ends in a SQI/Flat File: libodbctxt.so capital "S". libodbctxtS.so MiniSQL: libodbcmini.so Drivers not included: libodbcminiS.so Oracle: http://www.easysoft.org MySQL: http://www.unixODBC.org/myodbc.html Sybase: http://www.freetds.org M$-SQL: http://www.freetds.org Access: http://mdbtools.sourceforge.net Informix: http://www.ibm.com DB2: http://www.ibm.com unixODBC Libraries All these libraries should build The SQP library is a light-weight lex/yacc when you install unixODBC. based SQL parser used by unixODBC to permit Cursor Library: libodbccr.so SQL access to non-SQL sources such as NNTP. Linked List Library: liblstcl.la SQI Library: libsqilc.la The SQI library is a light-weight SQL engine SQP Library: libsqlpc.la that uses text files as its data source; this is the Logging Library: libloglc.la "backend" to the ODBC TXT driver. Tree Library: libtrelc.la PostgreSQL Setup: libodbcpsqlS.so Much of unixODBC's functionality has been split MiniSQL Setup: libodbcminiS.so into separate libraries to facilitate code reuse in MySQL Setup: libodbcmyS.so ODBC drivers and allow the code to be "borrowed" NNTP Driver Setup: libodbcnnS.so by other projects. TXT Driver Setup: libodbctxtS.so Setup libraries are typically built for all known esoob Setup: libesoobS.so ODBC drivers, even if the driver is not included or oplodbc Setup: liboplodbcS.so cannot be built due to lack of a SDK. This allows FreeTDS Setup: libtdsS.so you to drop a driver in and "just start using it". Oracle Setup: liboraodbcS.so The setup libraries are often maintained by the TXT Driver: libodbctxt.so unixODBC project and not the driver developers. NNTP Driver: libnn.so ODBCConfig ODBCConfig provides an easy GUI environment for defining personal DSNs, or if you're root, installing ODBC drivers and defining System DSNs.