Rocket U2 Databases and the Multivalue Model Dan Mcgrath, Product Manager Previously – Developer & Systems Architect

Rocket U2 Databases and the MultiValue model Dan McGrath, Product Manager Previously – Developer & Systems Architect

Today we will cover…

3 Today we will cover…

4 Rocket Software

Rocket Software (www.rocketsoftware.com) is a global software development firm that builds enterprise products and delivers enterprise solutions in the following segments:

Business Intelligence and Analytics

Storage, Networks, and Compliance

Application Development, Integration, and Modernization

Database Servers and Tools.

Rocket is engaged in business and technology partnerships with IBM, EMC, Fujitsu, HP Enterprise Services, Avaya, Motorola, Epicor, and many others.

The company is headquartered in Boston, Massachusetts, USA.

22 years

1,000+ employees

600+ engineers

100+ products

15+ R&D labs

10+ OEM partners

1,000+ ISV partners

10,000+ customers

 Rocket U2 is installed at 30,000 sites around the world.

 Users include:

40% • US Higher Education 50% • Top 400 North America energy product companies 55% • Fortune 100 companies 80% • Automobile dealerships in North America 80% • Mexico’s emergency response systems 100% • Airlines checking luggage in Australia/New Zealand

11 U2 Database - Platforms

Windows AIX HP-UX

Solaris RHEL SuSE

12 UniData & UniVerse

MultiValue (MV) DB Engine

• Efficient disk I/O • Table joining not required – nested data • Flexible storage structure

Relational DB Engine

• Map MV to Relational Tables • ODBC/JDBC access • Optional Data-type Enforcement • Federated Database System

13 UniData & UniVerse

Built-in JSON & XML support

Customizable security framework

Customizable DB commands

Automatic Data Encryption

Federated Database System

Replication

14 Database Structure

Instance Localhost

Accounts Production Staging Dev

Files VOC [Files…]

15 Database Structure

Keywords

Commands

VOC Files

Scripts

16 File types

Hash Tables OS Directories

Static

Dynamic

Locally Distributed

17 File Structures

Settings Hash table • # Groups • Block size • Hash Algorithm

Example file: 3 groups

Header Group 0 Group 1 Group 2 Overflow

• # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers (2) • Block Size • Data • Data • Data • Data (1) • Hash Algo

Block size Block size Block size Limited by OS

18 File Structures

Settings Dynamic Hash table • Hash Table’s settings • Split/Merge %

Example file: Initial 3 groups & first split

Header Group 0 Group 1 Group 2 Group 3

• # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers • Block Size • Data • Data • Data • Data • Hash Algo • Split/Merge

Separate Overflow file Linear extension; buddy system • IDs/Pointers (2) • Data (1)

19 File Structures

Locally Settings Distributed Hash • Hash Table’s settings table • Distribution Method

Part 0 Part … Part n Header Header Header Groups Groups Groups Overflow Overflow Overflow

20 File Structures

Directory

Inherits OS & File system ID: OS Data: OS file limits filename contents

21 Data Structures - MultiValue

Records Column types:

•Key/Value pairs •Single value •1-n columns •List (ordered) • Can be 0-bytes •List of lists (ordered)

Features

•Dynamic columns & lists •Indexable ‘Virtual’ Columns •‘Virtual’ Lookups

22 Data Structures – ‘people’

<0> <1> <2> <3>

goo.gl/C2z9f Jon Skeet goo.gl/n44vS

goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW

23 Data Structures

<0> <1> <2> <3>

goo.gl/C2z9f Jon Skeet goo.gl/n44vS

goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW

24 Data Structures

<0> <1> <2> <3>

goo.gl/C2z9f Jon Skeet goo.gl/n44vS

goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW

25 Data Structures

<0> <1> <2> <3>

goo.gl/C2z9f Jon Skeet goo.gl/n44vS

goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW

26 Metadata - MultiValue

Terminology

• @ID = Key • Dictionary ≈ Schema • Dictionary Item = Column Definition • Field/Attribute = Column • File = Table/Collection

27 Metadata - MultiValue

• Logical file = VOC entry Files • Physical file = OS entry • Multiple logical per physical

• Logical file / primary dictionary Sharing • View with other dictionaries

• Dictionaries optional Schema-less • Not enforced; ‘views’

28 Metadata - MultiValue

Dictionary Item Types

• Direct (indexable) • Virtual - Local (indexable) • Virtual - Lookup (indexable-ish) • Associations

29 Metadata – Direct @ID Type: Direct (Data Descriptor) Location: 0 Column: Single

@ID <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u

30 Metadata – Direct Name Topics URLs Type: Direct Type: Direct Type: Direct Location: 1 Location: 2 Location: 3 Column: Single Column: List Column: List of Lists

@ID Name Topics URLs goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u

31 Metadata – Associations Topics URLs Interests

Location: 2 Location: 3 Type: Association Column: List Column: List of Lists Items: Topics, URLs Association: Interests Association: Interests

@ID Name Topics URLs goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u

32 Metadata – Virtual (Local) NumTopics Soundex

Type: Virtual Type: Virtual

Location: DCOUNT(Topics, @VM) Location: SUBR(‘soundex’, Topics)

Column: Single Column: List

@ID Name Topics URLs NumTopics Soundex goo.gl/C2z9f Jon Skeet J523 goo.gl/n44vS goo.gl/0GPMb Unicorns U526 a45fe2 Tony the Pony goo.gl/MimWd 4 Boat Programming B316 jQuery goo.gl/iWs1u j260

33 Metadata – Virtual (Lookup) CreditCard Type: Virtual Location: TRANS(‘creditCards’,@ID, 1, ‘X’) Column: Single

@ID Name Topics URLs CreditCard goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood 3787-344936-71000 Estimates goo.gl/PCrfW

34 Metadata – Sharing

@ID byteLen

Type: Direct Type: Virtual

Location: 0 Location: LEN(@RECORD)

Column: Single Column: Single

JSON Type: Virtual

Location: SUBR(‘toJSON’, @RECORD, @FILENAME)

Column: Single

35 Locking

Read: Shared Group Lock Example file: 3 groups

Header Group 0 Group 1 Group 2 Overflow

• # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers (2) • Block Size • Data • Data • Data • Data (1) • Hash Algo

Write: Exclusive Group Lock

36 SQL Engine & Nested Tables

SQL Engine • SQL at the command-line embedded • Schema management tools

‘Virtual’ • Explicitly by schema generation tables for SQL • On-the-fly using ‘UNNEST’ keyword

JDBC & ODBC • Able to run native commands too access

Data-type • Ignore completely • Log violations to disk enforcement • Raise errors and prevent action

37 Federated Database System

Driver API exposed Drivers for: • We use this • SQL Server API to write Access as • DB2 our drivers if it was • Oracle Store data local in an external DB

38 Automatic Data Encryption

File level encryption

Column level encryption

Record key encryption

Index encryption

Multiple password protection of ‘Master Key’

‘Wallets’ Grouping / Temporary access

39 Replication

Publisher  1-n Subscribers

Account (full environment) replication

File replication

Organize into ‘groups’

Can Publish & Subscribe from same machine

3 modes:

Deferred – Sync every ‘x’ time

Immediate – Commits once staged

Real-time – Commits once sync’d

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

42 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

43 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

44 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

45 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

46 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

47 Replication

Publishing System Subscribing System

xadmin uvrepmanager uvrepmanager xadmin

Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA)

Subscriber Packet File (SPF)

uvsh Replication Buffer Replication Buffer uvrw

Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF)

48 Application Server “We need SMART Models, THIN Controllers, and DUMB Views”

50 Application Server

U2 Database •Model

View Controller

51 Application Server

U2 Database •Model

View Controller

52 Application Server

U2 Database

24/7 Phantoms

Callable Subroutines for Business Logic

Session Memory

Customizable DB Commands & Keywords

Customizable Security Framework

Select result ‘snapshots’

VOC

SELECT

PersonModel

CreditCardModel

stdSELECT

VOC

SELECT • V  SELECT

PersonModel • C /byteCode/PersonModel

CreditCardModel • C  /byteCode/CreditCardModel

stdSELECT • V  SELECT

VOC

SELECT • C  /byteCode/CustSelect

PersonModel • C /byteCode/PersonModel

CreditCardModel • C  /byteCode/CreditCardModel

stdSELECT • V  SELECT

VOC

SELECT • V  SELECT • Security SecControl

PersonModel • C /byteCode/PersonModel • Security SecControl

SecControl • C  /byteCode/SecControl

VOC

SELECT • V SELECT • Security SecControl

EVAL • K  EVAL • Security SecControl

SELECT people WITH name EVAL “TRANS(‘users’, @ID, 1, ‘X’)”

Sharing Sharing Save to disk within a between • Resume same processing after session sessions outage

SOAP Web Services Expose U2 • Automatic WSDL generation as a

Web Service RESTful Web Services • JSON-based

62 Business Intelligence & Analytics

63 Business Intelligence & Analytics

64 Network/Systems Monitoring

Rocket Software; enterprise solutions

Rocket U2 DBs; Airlines to 911

Flexible data structure & metadata

Plays nice with Relational DBs

www.rocketsoftware.com/u2  Free personal editions

Email: [email protected] Twitter: @itcmcgrath

Tech Conferences: U2 University  UK: 26-28 June, Australia: 14-16 August  u2u.rocketsoftware.com

Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWARE WITHOUT NOTICE. ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR • ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF ROCKET SOFTWARE.

©2012 Rocket Software, Inc. All Rights Reserved. 77 Trademarks and Acknowledgements The following are trademarks or registered trademarks of Rocket Software, Inc.: Dynamic Connect, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and wIntegrate. IBM, the IBM logo, AIX, and DB2 are trademarks of IBM in the United States and other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Microsoft, SQL Server, Windows, and Excel are trademarks of the Microsoft group of companies. UNIX is a registered trademark of The Open Group. Other company, product, and service names mentioned herein may be trademarks or service marks of others.