Introduction NoSQL – purpose-built data management NoSQL systems are purpose-built solutions, designed to address specific technical requirements. NoSQL systems originated to provide high throughput, fault-tolerant horizontally scalable simple data storage and retrieval with a bare minimum of additional functionality. Specifically, NoSQL systems were created in order to provide: Horizontally distributed data management of simple structured and unstructured data across a large cluster of commodity storage systems, Highly fault-tolerant data management and ability to continue operating even after multiple hardware and system failures, Very high throughput for simple read/write operations, with limited or no transaction semantics, Schema-less or flexible schema definitions allowing highly variable data and record structures, Application and application developer-centric special purpose data models and APIs. On the other hand, RDBMS systems, like Oracle Database, are designed to provide general purpose data management capabilities and standard APIs for a very wide variety of requirements. Hence, they incorporate a lot of features and functionality. Not all applications require the full set of RDBMS functionality. If the application doesn’t require all of the functionality in a typical RDBMS, why would the customer pay for the hundreds of features that they don’t need? For example, web-centric customer service, loyalty card programs and customer profile management applications primarily require fast, scalable key-based access to data. In such scenarios, a cost-effective, purpose-built NoSQL database is an attractive alternative to a relational system. Enterprises, ISVs and SIs are actively identifying applications and data management processes which can be implemented and managed more effectively using special-purpose NoSQL systems. NoSQL Application Example – simple data, simple queries Customer Profile Management Let’s look at a specific example of an application that matches the capabilities of a NoSQL system – customer profile management. In the past, customer profiles were typically financial transaction- oriented data structures/repositories. Today, a customer profile includes a much richer data set that includes information from a wide variety of customer interaction points that include both structured and unstructured data. Capturing and managing this new class of data is crucial to the enterprise, in order to more effectively obtain a 360 degree view of a customer and to optimize customer interactions. This information enables a broad category of Line-of-Business applications from Marketing, Advertising, Customer Service, Risk Analysis, Fraud Detection, Personalized content, Promotional Campaigns, Loyalty Programs, Inventory Management, etc. These Line-of-Business applications leverage the richer user-profile data in order to provide a. More personalized customer experiences via targeted product offerings, special promotions, loyalty rewards, more informed/context sensitive interactions, etc., b. Better operational insight into how their customers interact with them, how they perceive the company and its products and services, with a longer and more complete historical perspective, c. Better competitive insight into how customers perceive their competition, d. Better operational decision making, based on a more detailed understanding of their customers. Modern customer profile management applications benefit directly from the capabilities of NoSQL systems because they provide a. A horizontally scalable, distributed data management system to manage the large volume of data that is part of a rich customer profile and that can grow with little to no maintenance as the customer base grows, b. A highly fault tolerant system, ensuring that the customer profile data is always available to the applications that need to access and update it, c. A flexible or no-schema data store, which facilitates a wide variety of data record formats to be stored in a given customer profile and for those record formats to change over time, d. Specialized, high throughput application-specific read and write access to the portions of the customer profile that are important for that particular application1. A great example of this is our recent work with well known Entertainment company on their Next Generation Experience (NGE) application. This company wants to create a centralized repository for each of its customer’s experience across all of its entertainment properties (theme parks, restaurants, cruises, merchandizing outlets, etc.) over a span of decades. The value of this solution is that a customer’s visits as a child, then as a parent, then as a grandparent are all available in the system, regardless of which (other) system the interaction originated from. They want the customer profile repository to contain both structured and unstructured data, with a schema-flexible format so that it can evolve over time. Each customer’s profile will include URL-style pointers to records of activities at the various properties, as well as photos, privileges, loyalty program links, associated groups, other related customers, customer feedback, etc. For this company the concept of “a customer profile” includes many different sources and types of data, over an extended period of time, resulting in a very large and varied set of data – centralized and served from one data management infrastructure. This customer profile will be used to drive existing and future applications that provide the customer with a personalized, enhanced experience, utilizing both real- time and Business Intelligence Advanced Analytics access to the customer profile. Originally this centralized repository was designed to be managed in a relational database. However, last year they decided to re-design the NGE repository to use a NoSQL database. They made this decision because a. the NGE customer profile repository primarily required simple queries, simple data, flexible schemas and horizontally distributed storage, which was more efficiently addressed using a special purpose NoSQL database, b. of specific application developer technical synergies with NoSQL technology. In particular, their applications didn’t manage data using SQL nor did it represent the data as SQL tables. From the application point of view, records were just JSON objects consisting of simple, 1 It is important to note that these applications often require support for transactions and concurrency. Not all NoSQL databases provide this functionality. Oracle NoSQL Database does. For more information see section on Oracle NoSQL Database on page 9. but variable data elements and pointers to other records. Their natural application- developer-centric technical inclination was to eschew using SQL and focus on a simpler, more special purpose database and API that was closer to their application’s view of the data. They are now also looking at a NoSQL Database. This is great example of a common use case for a NoSQL Database. RDBMS vs NoSQL – The right tool for the right job Sophisticated RDBMS systems, for example the Oracle Database, encompass a very rich set of features and functionality, primarily focused on general purpose OLTP and Data Warehousing use by many different types of applications. NoSQL systems, on the other hand, encompass a very limited set of features and functionality while providing horizontal scalability, availability and data modeling flexibility for specific applications that manage simple data and simple queries. NoSQL systems typically include a set of software packages running across dozens, hundreds or even thousands of smaller systems. Efficient horizontal scalability, high availability and concurrent data processing are provided out of the box, but the integration effort to weave the software components into an integrated solution is left up to the customer. Many of the advanced features that are common with mature RDBMS systems are not present in NoSQL systems. This requires NoSQL technology users to integrate their NoSQL data with the RDBMS systems in order to make use of the advanced features that they need. NoSQL systems can do lots of specific-purpose, simple operations extremely quickly, but are not designed with the features to perform complex, general purpose operations in a integrated way. Let’s look at an example of how this plays out in terms of understanding what types of application characteristics are best suited for NoSQL and which ones are not. Simple Data, Simple Queries Let’s take the example of customer profile management discussed earlier. The functionality that is required is very simple – the application needs to read and write a few records based on the primary key -- customer ID. These records combine structured and unstructured data, stored in a way (often de-normalized) that allows the application to only operate on the subset of data that it needs for that specific type of transaction. Customer profile management applications typically have to perform an extremely high number of these simple operations of lookup based on a customer ID with minimal latency. The latency component is critical because these applications often implement the “last mile” interface with the customer. Customers and other downstream systems interact directly with the content managed in the NoSQL Database, and real-world studies have shown that any increase in latency can be linked directly to reduced revenue and loss of business.2 2 Amazon.com conducted an extensive study that demonstrated