#68
CONTENTS INCLUDE:
n Oracle Berkeley DB Family n Berkeley DB Java Edition Features Getting Started with n BDB JE Base API and Collections API n BDB JE Direct Persistent Layer n BDB JE Transaction Support and Performance Tuning Oracle Berkeley DB n BDB JE Backup and Recovery Visit refcardz.com n Hot Tips and more... By Masoud Kalali
entirely different set of features and characteristics (See Table About Oracle Berkeley DB 4). The base feature sets are shown in Table 1.
The Oracle Berkeley DB (BDB) family consists of three open Table 1: Family Feature Sets source data persistence products which provide developers Feature Set Description
with fast, reliable, high performance, enterprise ready local Data Store (DS) Single writer, multiple reader
databases implemented in the ANSI C and Java programming Concurrent Data Store (CDS) Multiple writers, multiple snapshot readers
languages. The BDB family typically stores key-value pairs but Transactional Data Store (TDS) Full ACID support on top of CDS is also flexible enough to store complex data models. BDB and High Availability (HA) Replication for fault tolerance. Fail over recovery support BDB Java Edition share the same base API, making it possible to easily switch between the two. Table 2 shows how these features are distributed between the We will review the most important aspects of Oracle BDB different BDB family members. family briefly. Then we will dig deep into Oracle BDB Java Table 2: Different Editions’ Feature Sets DS CDS TS HA Edition and see what its exclusive features are. We discuss BDB/BDB XML Edition Based API and Data Persistence Layer API. We will see how we can manage transactions DPL and Base API in addition BDB Java Edition to persisting complex objects graph using DPL will form the overall development subjects. Backup recovery, tuning, and Additional Features data migration utilities to migrate data between different editions and installations forms the administrations issues The BDB family of products has several special features and which we will discuss in this Refcard. offers a range of unique benefits which are listed in Table 3.
The BDB Family Table 3: Family Features and Benefits
Feature Benefit
Oracle BDB Core Edition Locking High concurrency
Berkeley DB is written in ANSI C and can be used as a library Data stored in application-native format Performance, no translation required www.dzone.com Get More Refcardz! to access the persisted information from within the parent Programmatic API, no SQL Performance, flexibility/control application address space. Oracle BDB provides multiple In process, not client-server Performance, no IPC required interfaces for different programming languages including Zero administration Low cost of ownership ANSI C, the Java API through JNI in addition to Perl, PHP, and ACID transactions and recovery Reliability, data integrity Python. Dual License Open/Closed source distributions Oracle BDB XML Edition Built on top of the BDB, the BDB XML edition allows us to easily store and retrieve indexed XML documents and to use XQuery to access stored XML documents. It also supports Get More Refcardz accessing data through the same channels that BDB supports. (They’re free!) BDB Java Edition BDB Java Edition is a pure Java, high performance, and flexible n Authoritative content embeddable database for storing data in a key-value format. It n Designed for developers supports transactions, direct persistence of Java objects using n Written by top experts EJB 3.0-style annotations, and provides a low level key-value n Latest tools & technologies retrieval API as well as an “access as collection” API. n Hot tips & examples n Bonus content online Key Features n New issue every 1-2 weeks Berkeley DB
Each of the BDB family members supports different feature sets. BDB XML edition enjoys a similar set of base features Subscribe Now for FREE! as the Core BDB. BDB Java edition on the other hand is Refcardz.com racle implemented in a completely different environment with an O
DZone, Inc. | www.dzone.com 2 Oracle Berkeley DB
In memory or on disk operation Transacted caching/ persisted data store db.get(null, keyValue, searchEntry, LockMode.DEFAULT);//retrieving record Similar data access API Easy switch between JE and BDB String foundData = new String(searchEntry.getData(), “UTF-8”); dataValue = new DatabaseEntry(“updated data content”. Just a set of library Easy to deploy and use getBytes(“UTF-8”)); Very large databases Virtually no limit on database size db.put(null, keyValue, dataValue);//updating an entry db.delete(null, keyValue);//delete operation db.close(); Features unique to BDB Java Edition are listed in Table 4. dbEnv.close();
Table 4: BDB Java Edition Exclusive Features There are multiple overrides for the Database.put method to Feature Benefit prevent duplicate records from being inserted and to prevent record overwrites. Fast, indexed, BTree Ultra fast data retrieval Java EE JTA and JCA support Integration with Java EE application servers DPL Sample Efficient Direct Persistence Layer EJB 3.0 like annotation to store Java Objects graph DPL sample consists of two parts, the entity class and the
Easy Java Collections API Transactional manipulation of Base API through entity management class which handle CRUD over the entity enhanced Java Collections class. Low Level Base API Work with dynamic data schema Entity Class JMX Support Monitor able from within parent application @Entity public class Employee { These features, along with a common set of features, make the @PrimaryKey Java edition a potential candidate for use cases that require public String empID; public String lastname; caching, application data repositories, POJO persistence, @SecondaryKey(relate = Relationship.MANY_TO_MANY, relatedEntity = Project.class,onRelatedEntityDelete = queuing/buffering, Web services, SOA, and Integration. DeleteAction.NULLIFY) public Set
DZone, Inc. | www.dzone.com 3 Oracle Berkeley DB
Configurations like read-only access, record duplication Table 5: Base API Features handling, creating in-memory databases, transaction support, Key value store retrieval, value can be anything etc. are provided through DatabaseConfig. Cursor API to traverse in a dataset forward and backward In DPL StoreConfig and EntityStore objects provide access to JCA (Java Connectivity Architecture) support object storage and how the object storage can be accessed. JMX (Java Management eXtension) support Configurations such as read only access, data model mutation, Table 6 shows most important DPL API characteristics and creating in-memory databases, transaction support, etc. are capabilities. Annotation and Object Mapping make rapid provided through StoreConfig. application development possible. The PrimaryIndex class provides the primary storage and Table 6: DPL API Features access methods for the instances of a particular entity class. Type Safe access to persisted objects There are multiple overrides for the PrimaryIndex.put method Updating classes with adding new fields is supported to prevent duplicate entity insertion and provide entity overwrite prevention. Persistent class fields can be private, package-private, protected or public Automatic and extendable data binding between Objects and underlying storage Hot When closing an Environment or Database or when we Index fields can be accessed using a standard java.util collection. Tip commit a Transaction in a multi thread application we Java annotations are used to define metadata like relations between objects should ensure that no thread still has in-progress tasks. Field refactoring is supported without changing the stored date.(Called mutation)
Table 6 and Table 7 list the features that mostly determine BDB Java edition environment anatomy when we should use which API. Table 7 lists possible use cases for each API. A BDB JE database consists of one or more log files which are placed inside the environment directory. Table 7: Which API is suitable for your case Use case characteristics Suitable API The log files are named NNNNNNNN.jdb where NNNNNNNN Data model is highly dynamic and changing Base API is an 8-digit hexadecimal number that increases by 1 (starting Data model has complex object model and relationships DPL from 00000000) for each log file written to disk. BDB JE rolls Need application portability between Java Edition and Core Edition DPL, Base API to next file when the current file size reaches the predefined configurable size. The predefined size is 10MB. Transaction Support A BDB database can be considered like a relational table in an RDBMS. In Base API we directly use the database we need to Transaction Support is an inseparable part of enterprise access, while in DPL we use an EntityStore which may interact software development. BDB JE supports transaction and with multiple databases under the hood. provides concurrency and record level locking. To add Each BDB environment can contain tens of databases and all of transaction support to DPL in our DPL sample code we can these databases will be stored in a single row of log files. (No introduce the following changes: separate log files per-database). Figure 1 shows the concept ... visually. envConfig.setTransactional(true); stConf.setTransactional(true); To create in-memory database we can use DatabaseConfig. TransactionConfig txConf = new TransactionConfig(); stConf.setTransactional(true); Hot setTemporary(true) and StoreConfig.setTemporary(true) txConf.setReadCommitted(true); Tip to get an in-memory instance with no data persisted Transaction tx= dbEnv.getThreadTransaction(); dbEnv.beginTransaction(tx, txConf); beyond the current session. ... userIndex.putNoReturn(tx, new Employee(“u180”, “Doe”, null));// insert ... tx.commit(); ... The simplicity of BDB JE Transaction Support makes it very suitable for transactional cache systems. The isolation level, deferrable and manual synchronization of transactional data with hard disk (Durability), replication policy, and transaction lock request and transaction lifetime timeout can Figure 1: BDB JE environment and log files. be configured using the Transaction and TransactionConfig The environment path should point to an already objects. Hot existing directory, otherwise the application will face and Tip exception. When we create an environmnt object for the Hot Environment, Database, and EntityStore are thread safe first time, necessary files are created inside that direcory. Tip meaning that we can use them in multiple threads without manual synchronization
BDB Java edition environment anatomy Transaction support in Base API is a bit different, as in Base API we directly deal with databases while in DPL we deal with Table 5 shows Base API characteristics and benefits. The key- environment and EntityStore objects. The following changes value access model provides the most flexibility. will allow transaction support in Base API.
DZone, Inc. | www.dzone.com 4 Oracle Berkeley DB
... numerical and a named sequence can be used for multiple envConfig.setTransactional(true); entities. dbconf.setTransactional(true); TransactionConfig txConf = new TransactionConfig(); txConf.setSerializableIsolation(true); Now let’s see how we can store and retrieve an employee with txConf.setNoSync(true); its related project objects. Transaction tx = dbEnv.getThreadTransaction(); tx.setLockTimeout(1000); tx.setTxnTimeout(5000); ... Database db = dbEnv.openDatabase(tx, «SmpleDB», dbconf); PrimaryIndex
EntityCursor
@PrimaryKey Defines the class primary key and must be used one and only one time for The Environment and EntityStore definitions are omitted. every entity class. The SecondaryIndex provides primary methods for retrieving @SecondaryKey Declares a specific data member in an entity class to be a secondary key for that object. This annotation is optional, and can be used multiple times for objects related to the Secondary Key of a particular object. an entity class. The SecondaryIndex can be used to retrieve the related objects @Persistent Declares a persistent class which lives in relation to an entity class. through a traversable cursor. We can also use SecondaryIndex @NotTransient Defines a field as being persistent even when it is declared with the transient to query for a specific range of objects in a given range for its keyword. primary key. @NotPersistent Defines a field as being non-persistent even when it is not declared with the transient keyword. Table 9: Supported Object Relation @KeyField Indicates the sorting position of a key field in a composite key class when the Comparable interface is not implemented. The KeyField integer Relation Description element specifies the sort order of this field within the set of fields in the composite key. ONE_TO_ONE A single entity is related to a single secondary key value. ONE_TO_MANY A single entity is related to one or more secondary key values. We used two of these annotations in practice and you saw MANY_TO_ONE One or more entities are related to a single secondary key value. @SecondaryKey in the Employee class. Now we are going to see MANY_TO_MANY One or more entities are related to one or more secondary key values. how the @SecondaryKey annotation can be used. Let’s create the Project entity which the Employee class has a many-to-many A SecondaryIndex can be used to traverse over the collection relation with. of secondary key’s values to retrieve the secondary objects.
@Entity public class Project { Multiple processes can open a database as long as only one process opens it in read-write mode and other public String projName; Hot @PrimaryKey(sequence = «ID») processes open the database in read-only mode. The read- public long projID; Tip only processes get an open-time snapshot of the database public Project() { and won’t see any changes coming from other process. } public Project(String projName) { this.projName = projName; } BDB JE Collections API }
The @PrimaryKey annotation has a string element to define The only different between using BDB JE collections API and the name of a sequence from which we can assign primary classic collections is the fact that when we use BDB JE Collections key values automatically. The primary key field type must be API we are accessing persisted objects instead of in-memory
DZone, Inc. | www.dzone.com 5 Oracle Berkeley DB
objects which we usually access in classic collection APIs. Now let’s see the driver code which put this class in action.
The Collections API Characteristics public class CollectionSample { public static void main(String[] argv) An implementation Map, SortedMap, Set, SortedSet, and Iterator. throws Exception { // Creating the environment To stay compatible with Java Collections, Transaction is supported using TransactionWorker EnvironmentConfig envConfig = new EnvironmentConfig(); and TransactionRunner which the former one is the interface which we can implement to envConfig.setTransactional(true); execute our code in a transaction and later one process the transaction. envConfig.setAllowCreate(true); Environment dbEnv = new Environment(new File(“/home/ Keys and values are represented as Java objects. Custom binding can be defined to bind the masoud/dben-col”), envConfig); stored bytes to any type or format like XML, for example. // creating an instance of our TransactionWorker TransWorker worker = new TransWorker(dbEnv); Data binding should be defined to instruct the Collections API about how keys and values are TransactionRunner runner = new TransactionRunner(dbEnv); represented as stored data and how stored data is converted to and from Java objects. We can runner.run(worker); use one of the two (SerialBinding, TupleBinding) default data bindings or a custom data binding. } Environment, EnvironmentConfig, Database and DatabaseConfig stay the same as it was for } Base API. CollectionSample Collections API extends Java serialization to store class description separately to make data The steps demonstrated in the are records much more compact. self describing. The only new object in this snippet is the TransactionRunner object which we used to run the To get a real sense about BDB JE Collections API think of it as TransWorker object. I omit many of the safe programming we can persist and retrieve objects using a collection class like portions to keep the code simple and conscious. we need SortedMap’s methods like tailMap, subMap or put, putall, get, exception handling and properly closure of all BDB JE objects and so on. to ensure data integrity But before we use the SortedMap object to access the stored data, we need to initialize the base objects like Database and BDB JE Backup/Recovery and Tuning Environment; we should create the ClassCatalog object, and finally we should define bindings for our key and value types. Backup and Recovery Collections API sample We can simply backup the BDB databases by creating an Now let’s see how we can store and retrieve our our objects operating system level copy of all jdb files. When required we using Collections API. In this sample we are persisting a pair of can put the archived files back into the environment directory Integer key and String value using SortedMap. to get a database back to the state it was at. The best option is First lets analyze the TransactionWorker implementation. to make sure all transactions and the write process are finished
public class TransWorker implements TransactionWorker { to have a consistent backup of the database. private ClassCatalog catalog; private Database db; The BDB JE provides a helper class located at com.sleepycat. private SortedMap map; je.util.DbBackup public TransWorker(Environment env) throws Exception { to perform the backup process from within DatabaseConfig dbConfig = new DatabaseConfig(); a Java application. This utility class can create an incremental dbConfig.setTransactional(true); dbConfig.setAllowCreate(true); backup of a database and later on can restore from that Database catalogDb = env.openDatabase(null, “catalog”, dbConfig); backup. The helper class ideally freezes the BDB JE activities catalog = new StoredClassCatalog(catalogDb); // use Integer tuple binding for key entries during the backup to ensure that the created backup exactly TupleBinding keyBinding = TupleBinding.getPrimitiveBinding(Integer.class); represents the database state when the backup process // use String serial binding for data entries started. SerialBinding dataBinding = new SerialBinding(catalog, String.class); db = env.openDatabase(null, “dben-col”, dbConfig); Tuning map = new StoredSortedMap(db, keyBinding, dataBinding, Berkeley DB JE has 3 daemon threads and configuring these true); } threads affects the overall application performance and /** Performs work within a transaction. */ public void doWork() throws Exception { behavior. These 3 threads are as follow: // check for existing data and writing Integer key = new Integer(0); Cleaner Thread Responsible for cleaning and deleting unused log files. This thread is String val = (String) map.get(key); run only if the environment is opened for write access. if (val == null) { map.put(new Integer(10), “Second”); Checkpointer Thread Basically keeps the BTree shape consistent. Checkpointer thread is } triggered when environment opens, environment closes, and database //Reading Data log file grows by some certain amount. Iterator iter = map.entrySet().iterator(); while (iter.hasNext()) { Compressor Thread For cleaning the BTree structure from unused nodes. Map.Entry entry = (Map.Entry) iter.next(); //Process the entry These threads can be configured through a properties file } } named je.properties or by using the EnvironmentConfig and } EnvironmentMutableConfig objects. The je.properties file, TransWorker implements TransactionWorker which makes which is a simple key-value file, should be placed inside the it necessary to implement the doWork method. This method environment directory and override any further configuration is called by TransactionRunner when we pass an object of which we may make using the EnvironmentConfig and TransWorker to its run method. The TransWorker constructor EnvironmentMutableConfig in the Java code. simply receive an Environment object and construct other The other performance effective factor is cache size. For on- required objects . Then it opens the database in Collection disk instances cache size determines how often the application mode, creates the required binding for the key and values needs to refer to permanent storage in order to retrieve some we want to store in the database and finally it creates the data bucket. When we use in-memory instances cache size SortedMap object which we can use to put and retrieve objects determines whether our database information will be paged using it. into swap space or it will stay in the main memory.
DZone, Inc. | www.dzone.com 6 Oracle Berkeley DB
je.cleaner. Ensures that a minimum amount of space is occupied by live records by structure, and loading the dump into another environment. minUtilization removing obsolete records. Default occupied percentage is 50%. DbDump Dumps a database to a user-readable format. je.cleaner. Determines the cleaner behavior in the event that it is able to remove expunge an entire log file. If “true” the log file will be deleted, otherwise it will be DbLoad Loads a database from the DbDump output. renamed to nnnnnnnn.del DbVerify Verifies the structure of a database. je.checkpointer. Determines how often the Checkpointer should check the BTree structure. If bytesInterval it performs the checks little by little it will ensure a faster application startup but will consume more resources specially IO. To run each of these utilities, switch to BDB JE directory, je.maxMemory Determines what percentage of JVM maximum memory size can be used switch to lib directory and execute as shown in the following Percent for BDB JE cache. To determine the ideal cache size we should put the application in the production environment and monitor its behavior. command:
A complete list of all configurable properties, with java -cp je-3.3.75.jar com.sleepycat.je.util.DbVerify explanations, is available in EnvironmentConfig Javadoc. The list is comprehensive and allows us to configure the BDB JE at The JAR file name may differ depending on your version of granular level. BDB JE. These commands can also be used to port a BDB JE database to BDB Core Edition. All of these parameters can be set from Java code using the EnvironmentConfig object. The properties file overrides the A very good set of tutorials for different set of BDB JE APIs values set by using EnvironmentConfig object. Hot are available inside the docs folder of BDB JE package. Helper Utilities Tip Several examples for different set of functionalities are provided inside the examples directory of the BDB JE Three command line utilities are provided to facilitate dumping package. the databases from one environment, verifying the database
A b o u t the Author r e c o m m eND ED Bo o k Masoud Kalali holds a software engineering degree and has The Berkeley DB Book is a practical guide to the intricacies been working on software development projects since 1998. He of the Berkeley DB. This book covers in-depth the complex has experience with a variety of technologies (.Net, J2EE, CORBA, design issues that are mostly only touched on in terse and COM+) on diverse platforms (Solaris, Linux, and Windows). footnotes within the dense Berkeley DB reference manual. It His experience is in software architecture, design and server side explains the technology at a higher level and also covers the development. Masoud has several articles in Java.net. He is one of internals, providing generous code and design examples. founder members of NetBeans Dream Team. Masoud’s main area of research and interest includes Web Services and Service Oriented Architecture along with large scale and high throughput systems’ development and deployment. BUY NOW Blog: http://weblogs.java.net/blog/kalali/ books.dzone.com/books/berkeley-db Contact: [email protected]
Brought to you by... Professional Cheat Sheets You Can Trust
“Exactly what busy developers need: Patterns simple, short, and to the point.” Design By Jason McDonald Inspired by the James Ward, Adobe Systems #8 GoF ndler doesn’t have to Bestseller e a request with the handler CONTENTS INCLUDE: e potential outcome. Upcoming Titles Most Popular Chain of Responsibility, continued n Chain of Responsibility n Multiple objects may handle a request and the ha be a specific object. ements this pattern. When an n Command Use n A set of objects should be able to handl checks to see if the method n Interpreter When determined at runtime. Expression Web Spring Configuration on or if it should be passed up the n Iterator n A request not being handled is an acceptabl Downloadck the process repeats until code to Now n Mediator until there are no more parent Exception handling in some languages impl Eclipse Plug-In Development jQuery Selectors n Observer exception is thrown in a method the runtime Example n Template Method and more... has a mechanism to handle the excepti Object Behavioral call stack. When passed up the call sta Adobe Live Cycle Windows Powershell handle the exception is encountered or
Visit refcardz.com refcardz.com Visit objects to hand the request to. ABOUT DESIGN PATTERNS Refcardz.com Invoker Adobe Flash Builder 4 Dependency Injection with EJB 3
This Design Patterns refcard provides a quick reference to the Java Performance Tuning Netbeans IDE JavaEditor original 23 GangDesign of Patterns: Four. Each (GoF) patternElements design includes ofpatterns, Reusable class as diagrams,listedObject- in COMMAND the book Client Used to construct objects such ConcreteCommand WPF Getting Started with Eclipse Oriented Software Command +execute( ) explanation, usage information, and a real world example. Creational Patterns: +execute() F# Very First Steps in Flex that they can be decoupledUsed from to formtheir largeimplementing object reated as an object. This allows system. nally object based relationships such Receiver Structural Patterns: Used to manage algorithms, Encapsulates a request allowing it to be t mes or in variant orders. structures between many disparate objects. DZone, Inc. ISBN-13: 978-1-934238-62-2 the request to be handled in traditio Purpose as queuing and callbacks. object handling the invocation. Behavioral Patterns: ISBN-10: 1-934238-62-7 n You need callback functionality. 1251 NW Maynard relationships, and responsibilities between objects. n Requests need to be handled at variantilitate ti theattern asynchronous the functionality processing to be Deals with object relationships that can Use n A history of requests is needed. When ue for processing without any need Prototype n The invoker should be decoupled from the actual implementationmplements its it particular is Object Scope: C Cary, NC 27513 50795 Deals with class relationships that can be Job queues are widely used to fac be changed at runtime. S Proxy of algorithms. By utilizing the commande pinterface the queue is expecting. Example Decorator Observer executed can be given to a job que Class Scope: S B for the queue to have knowledge of the 888.678.0399 changed at compile time. Facade Singleton invoking. The command object that is enqueued i S C algorithm within the confines of th Abstract Factory DZone communitiesC deliver Factory Method over 6State million pages each month to C B 919.678.0300 Adapter Flyweight Strategy S S B Bridge Interpreter Template Method more than 3.3 millionS softwareB developers,B architects and decision Refcardz Feedback Welcome Builder Iterator Visitor
C B B www.dzone.com Get More Refcarz! Refcarz! More Get www.dzone.com Chain of Mediator B B makers. DZone offersResponsibility something for everyone, including news, Memento [email protected] B Object Behavioral Command B
e $7.95 tutorials, cheatsheets,Composit blogs, feature articles, source code and more. S Sponsorship Opportunities 9 781934 238622 successor
“DZone is a developer’sCHAIN OF RESPONSIBILITY dream,” says PC Magazine. [email protected] <
Copyright © 2009 DZone, Inc. All rights reserved. No+handlerequest() part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, Version 1.0
Client photocopying, or otherwise, without prior written permission of the publisher. Reference: ConcreteHandler 2 +handlerequest()
e a request by linking ConcreteHandler 1 | www.dzone.com
+handlerequest()
DZone, Inc. Patterns
Gives more than one object an opportunity to handl receiving objects together.
Purpose Design Design