DB2 and NoSQL - The end of relaonal? George Baklarz IBM Canada Lab Session Code: B08 Thursday, September 12 (11:15 am – 12:15 pm) | DB2 for Linux, Unix, and Windows 2 Agenda • Database Market • Databases • Applicaon APIs • NoSQL • Types of NoSQL Databases • Current Market Trends with NoSQL • JavaScript Object Notaon • What is it and Why do developers Like it • JSON Datastore in DB2 • MongoDB Query API • Java Programming API • JSON Wire-listener 3 The Database Market and Choice You are here 4 Database API Support Javascript Java PhP Python/Jython Ruby/JRuby Scala node.js JDBC API pureQuery node.js Zend node-odbc SqlAlchemy / Rails Lift API framework JSON Django Adapter Adapter adapters driver JSON API Python interpreter Ruby interpreter Hibernate c java c java OpenJPA DB2 CLI and ODBC driver DB2 JCC JDBC driver DB2 5 ADO.NET Support Application (ASP.NET) ADO.NET ODBC .NET OLE DB .NET Data Provider Data Provider Other .NET DB2 .NET Data Data Providers Provider IDS/DB2 ODBC IDS/DB2 OLE Driver Provider Informix DB2 DBMS DB DB Now supporting: Visual Studio 2012, .NET Framework 4.5, Entity Framework 4.3, Entity Framework 5.0 is delivered with DB2 10.5 6 What is NoSQL? Dominant Flavors • Key Value Stores 122+ NoSQL Database Hash table of keys, where the data part of Offerings Today! • key-value is in a binary object • Examples pure key-value stores : MemcacheD, REDIS, WebSphere eXtreme Scale • Document Stores • Stores documents made up of tagged elements, which have keys and document-like objects • Examples : MongoDB, couchDB • Column Family • Each storage block contains data from only one column/column set • Examples : HBase, Cassandra Motivation • Graph Store § Many apps need fewer database features (simplicity) • Key-values are related through graph structure § Need rapid application evolution/deployment, with minimal interaction with DBA • Common Model : RDF § Some apps need extremely high scale (e.g. Twitter) • Examples : Jena, Sesame § Need for a low-latency, low-overhead API to access data § Increasing use of distributed analytics 7 No SQL Market Data • 2011: Market Research Media noted worldwide NoSQL market was 2014 expected to reach ~$3.4B by 2018, generang $14B from 2013-2018 % NoSQL Enterprise Adoption with a CAGR of 21% 2010 2015 • Comparison: data implies NoSQL market ~$895M "MongoDB is the new MySQL." • MySQL in 2011 ~$100M "Current adoption of NoSQL in enterprises is 4% in 2010. expected to double by 2014, and grow to 20% by 2015. 10% of existing apps could move to NoSQL. [Sept 2011] "NoSQL is in its infancy, many immature with only community support and limited tool capability; however, expect to see a few used more widely during the next 5 years." 8 Hacker News Trends 9 The JSON-XML ShiN • Developers find it easier to move data back and forth without losing informaon in JSON vs. XML • XML is more powerful and more sophisMcated than JSON • BUT JSON found to be 'good enough' à It makes programming tasks easier • By the Mme RDBMS world got very sophisMcated with XML, developers had chosen JSON • Applicaon shid lead to emergence of database that store data in JSON (i.e. MongoDB) • JSON on the server side is appealing for developers using JSON on the client Mer side 10 Open APIs State of the Market • JSON is the new cool • XML declining: 5 years ago hardly any JSON • Why? JSON is: • Less verbose and smaller docs size • <Mytag>value</Mytag> vs. Mytag:value • Tightly integrated with JavaScript which has a lot of focus • Most new development tools support JSON and not XML • 50% of NoSQL Engines are for JSON Documents 11 New Era Applica4on Characteris4cs • Applicaons evolve rapidly as the needs for mobile and Web presence try to keep pace with internet user needs • Applicaon developers are increasingly looking for soluMons that allow nearly conMnuous integraon of applicaon changes • Amazon.com allows 1000’s of their developers to check in product code changes daily… • Developers resist soluMons that require delays to sync up with DBA change windows 12 NoSQL JSON Stores are Appealing to Developers • JSON schema can be evolved rapidly without intervenMon by DBAs or data modelers. • Objects like "shopping cart" in these applicaons really aren’t used outside the Web applicaon, so there is no need to interlock closely with the rest of the enterprise data model • JSON offers a very simple and elegant model for persisMng Java or JavaScript objects, without needing a heavy-weight persistence soluon like OpenJPA or Hibernate • Performance and scalability is very good for JSON • Store a single JSON document represenMng the object versus • Store "n" rows in relaonal as a "normalized" object 13 Typical JSON Open Source Datastore Aributes • Logging is oden turned off to improve performance • By default, no return code on insert (a.k.a. "fire and forget") • App must verify update was performed • Data is sharded for scalability • Shards are replicated asynchronously for availability • Queries to replica nodes can return back-level data someMmes… • No concept of commit or rollback • Each JSON update is independent • No document-level locking • App must manage a "revision" tag to detect document update conflicts • Applicaons have to implement compensaon logic to update mulMple documents with ACID properMes • JSON documents are stored in collecMons • But no "join" across collecMons • No document-level or tag-level security • No built-in temporal or geo-spaal query support 14 Data Access Example Using JavaScript and JSON • Relaonal representaon Lastname Firstname Street Jones Billy 123 Maple Drive • JSON representaon JSON_string = ‘{"Lastname":"Jones", "Firstname":"Billy", "Street":"123 Maple Drive"}’; " • Javascript data access var JSONrecord = JSON.parse(JSON_string); " l_name = JSONrecord.Lastname;" f_name = JSONrecord.Firstname;" l_street = JSONrecord.Street;" 15 Simple Database API for JSON • Insert a record, a blog post by Joe: "db.posts.insert({author:"Joe", date:"2012-04-20", post:"…"}) • Find all posts by Joe: "db.posts.find({author:"Joe"})" • Update date of Joe's post "db.posts.update({author:"Joe",{$set:{date:"2013-08-22"}})" • Delete all posts of Joe: "db.posts.remove({author:"Joe"})" • Remove the posts collecMon "db.posts.drop()" 16 The Guardian Move from Oracle to MongoDB • ExisMng system: Modern Java Applicaon using Load became an issue Spring, Hibernate, and Oracle RDBMS was difficult to scale • Problems: • Schema upgrades caused downMme • Complexity • Found 300 tables complex (imagine SAP) • 10,000 lines of Hibernate configuraon code found to be buggy • 1,000 domain objects for database mapping • 70,000 lines of domain object code, Mght binding to database • ORM did not really mask complexity • Database had strong influence on domain model, complex mapping joins Introduced memcached to patch up load problems • Required complex hibernate features, complex caching strategy, lots of opMmizaons, hand code complex queries in SQL • Load problems: introduced caching, and SOLR/ API to opMmize reads • Solved load problems, but increased complexity to 3 models: tables, objects, JSON API 17 The Guardian Move to MongoDB Found it a simple key store… too simple • First project: Migrate Oracle system with 3M users, complex PL-SQL • Embraced MongoDB: • Simple flexible schema with similar query and indexing to RDBMS • MulMple domain concepts expressed in single document • Can be designed in forwardly extensible way • Document oriented • Stores parsed JSON documents • Can express complex queries • Can be flexible about consistency • Malleable schema, can change at runMme • Can work at both large and small scales • Easy for developers to get going • Commercial support available (10Gen) • Produces a net reducMon in lines of code and complexity 18 DB2 JSON Support: Agility With a Trusted Founda4on • Interoperate seamlessly with modern applicaons • Flexible schemas allow rapid delivery of applicaons • Preserve tradiMonal DBMS capabiliMes, leverage exisMng skills and tools • MulM-statement TransacMons • Management / Operaons • Security • Scale, performance and high availability • Extend with advanced features (future) • Temporal semanMcs • Full Text search • MulM-collecMon joins • Combine with Enterprise RDBMS data 19 DB2 JSON API Technology Details • IBM provided Java Driver for JSON API • Java Driver supporMng JSON API for data access layer • TransacMons • Parametric SQL statements (Delete, select) • Temporal tables • CLP-Like Command Shell • Ad-hoc updates / queries • Administraon commands • Open Source Driver Wire Listener • Leverage NoSQL community drivers for data access layer 20 DB2 JSON Java API • Java Driver that translates API calls to SQL + funcon invoca:ons • Supports Transactions • Batches insertions Java Apps • Fire-forget inserts (fast) JSON API JSON • Indexing Command Shell • Time travel query JDBC Driver DRDA • Smart Query re-write • Java command line JSON_TABLE() JSON JSON_UPDATE() UDFs … JSON_VAL() - builtin, supports extraction of (SQL) DB2 Engine IoE w/ BLOB in the values from BSON expression 21 DB2 NoSQL/JSON API from Java /*Set up Conn. and Database handle*/ /* Use cursor to fetch back the JSON */ Context ctx = new InitialContext(); DataSource ds = DBCursor cursor = shop.find(new (DataSource)ctx.lookup("jdbc/myDB2"); BasicDBObject("customer", "Bill")); Connection conn = ds.getConnection(); try { Database db = new Database(conn); DBCollection shop = while(cursor.hasNext()) { db.getCollection("shop"); DBObject obj = cursor.next(); doSomething(obj); /*Create JSON objects and insert*/ } BasicDBObject cart = new } finally{ BasicDBObject();