Rocket U2 Databases and the MultiValue model Dan McGrath, Product Manager Previously – Developer & Systems Architect Today we will cover… 3 Today we will cover… 4 Rocket Software Rocket Software (www.rocketsoftware.com) is a global software development firm that builds enterprise products and delivers enterprise solutions in the following segments: Business Intelligence and Analytics Storage, Networks, and Compliance Application Development, Integration, and Modernization Database Servers and Tools. Rocket is engaged in business and technology partnerships with IBM, EMC, Fujitsu, HP Enterprise Services, Avaya, Motorola, Epicor, and many others. The company is headquartered in Boston, Massachusetts, USA. ©2012 Rocket Software, Inc. All Rights Reserved. 5 Rocket Worldwide Locations ©2012 Rocket Software, Inc. All Rights Reserved. 6 Rocket by the Numbers 22 years 1,000+ employees 600+ engineers 100+ products 15+ R&D labs 10+ OEM partners 1,000+ ISV partners 10,000+ customers ©2012 Rocket Software, Inc. All Rights Reserved. 7 Rocket Brands ©2012 Rocket Software, Inc. All Rights Reserved. 8 Who Uses Rocket U2? Rocket U2 is installed at 30,000 sites around the world. Users include: 40% • US Higher Education 50% • Top 400 North America energy product companies 55% • Fortune 100 companies 80% • Automobile dealerships in North America 80% • Mexico’s emergency response systems 100% • Airlines checking luggage in Australia/New Zealand ©2012 Rocket Software, Inc. All Rights Reserved. 9 Now, the real stuff… UniData & UniVerse 11 U2 Database - Platforms Windows AIX HP-UX Solaris RHEL SuSE 12 UniData & UniVerse MultiValue (MV) DB Engine • Efficient disk I/O • Table joining not required – nested data • Flexible storage structure Relational DB Engine • Map MV to Relational Tables • ODBC/JDBC access • Optional Data-type Enforcement • Federated Database System 13 UniData & UniVerse Built-in JSON & XML support Customizable security framework Customizable DB commands Automatic Data Encryption Federated Database System Replication 14 Database Structure Instance Localhost Accounts Production Staging Dev Files VOC [Files…] 15 Database Structure Keywords Commands VOC Files Scripts 16 File types Hash Tables OS Directories Static Dynamic Locally Distributed 17 File Structures Settings Hash table • # Groups • Block size • Hash Algorithm Example file: 3 groups Header Group 0 Group 1 Group 2 Overflow • # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers (2) • Block Size • Data • Data • Data • Data (1) • Hash Algo Block size Block size Block size Limited by OS 18 File Structures Settings Dynamic Hash table • Hash Table’s settings • Split/Merge % Example file: Initial 3 groups & first split Header Group 0 Group 1 Group 2 Group 3 • # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers • Block Size • Data • Data • Data • Data • Hash Algo • Split/Merge Separate Overflow file Linear extension; buddy system • IDs/Pointers (2) • Data (1) 19 File Structures Locally Settings Distributed Hash • Hash Table’s settings table • Distribution Method Part 0 Part … Part n Header Header Header Groups Groups Groups Overflow Overflow Overflow 20 File Structures Directory Inherits OS & File system ID: OS Data: OS file limits filename contents 21 Data Structures - MultiValue Records Column types: •Key/Value pairs •Single value •1-n columns •List (ordered) • Can be 0-bytes •List of lists (ordered) Features •Dynamic columns & lists •Indexable ‘Virtual’ Columns •‘Virtual’ Lookups 22 Data Structures – ‘people’ <0> <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW 23 Data Structures <0> <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW 24 Data Structures <0> <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW 25 Data Structures <0> <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood Estimates goo.gl/PCrfW 26 Metadata - MultiValue Terminology • @ID = Key • Dictionary ≈ Schema • Dictionary Item = Column Definition • Field/Attribute = Column • File = Table/Collection 27 Metadata - MultiValue • Logical file = VOC entry Files • Physical file = OS entry • Multiple logical per physical • Logical file / primary dictionary Sharing • View with other dictionaries • Dictionaries optional Schema-less • Not enforced; ‘views’ 28 Metadata - MultiValue Dictionary Item Types • Direct (indexable) • Virtual - Local (indexable) • Virtual - Lookup (indexable-ish) • Associations 29 Metadata – Direct @ID Type: Direct (Data Descriptor) Location: 0 Column: Single @ID <1> <2> <3> goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u 30 Metadata – Direct Name Topics URLs Type: Direct Type: Direct Type: Direct Location: 1 Location: 2 Location: 3 Column: Single Column: List Column: List of Lists @ID Name Topics URLs goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u 31 Metadata – Associations Topics URLs Interests Location: 2 Location: 3 Type: Association Column: List Column: List of Lists Items: Topics, URLs Association: Interests Association: Interests @ID Name Topics URLs goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u 32 Metadata – Virtual (Local) NumTopics Soundex Type: Virtual Type: Virtual Location: DCOUNT(Topics, @VM) Location: SUBR(‘soundex’, Topics) Column: Single Column: List @ID Name Topics URLs NumTopics Soundex goo.gl/C2z9f Jon Skeet J523 goo.gl/n44vS goo.gl/0GPMb Unicorns U526 a45fe2 Tony the Pony goo.gl/MimWd 4 Boat Programming B316 jQuery goo.gl/iWs1u j260 33 Metadata – Virtual (Lookup) CreditCard Type: Virtual Location: TRANS(‘creditCards’,@ID, 1, ‘X’) Column: Single @ID Name Topics URLs CreditCard goo.gl/C2z9f Jon Skeet goo.gl/n44vS goo.gl/0GPMb a45fe2 Tony the Pony Unicorns goo.gl/MimWd Boat Programming jQuery goo.gl/iWs1u Unicorns goo.gl/MimWd 5bd4e6 Jeff Atwood 3787-344936-71000 Estimates goo.gl/PCrfW 34 Metadata – Sharing @ID byteLen Type: Direct Type: Virtual Location: 0 Location: LEN(@RECORD) Column: Single Column: Single JSON Type: Virtual Location: SUBR(‘toJSON’, @RECORD, @FILENAME) Column: Single 35 Locking Read: Shared Group Lock Example file: 3 groups Header Group 0 Group 1 Group 2 Overflow • # Groups • IDs/Pointers • IDs/Pointers • IDs/Pointers • IDs/Pointers (2) • Block Size • Data • Data • Data • Data (1) • Hash Algo Write: Exclusive Group Lock 36 SQL Engine & Nested Tables SQL Engine • SQL at the command-line embedded • Schema management tools ‘Virtual’ • Explicitly by schema generation tables for SQL • On-the-fly using ‘UNNEST’ keyword JDBC & ODBC • Able to run native commands too access Data-type • Ignore completely • Log violations to disk enforcement • Raise errors and prevent action 37 Federated Database System Driver API exposed Drivers for: • We use this • SQL Server API to write Access as • DB2 our drivers if it was • Oracle Store data local in an external DB 38 Automatic Data Encryption File level encryption Column level encryption Record key encryption Index encryption Multiple password protection of ‘Master Key’ ‘Wallets’ Grouping / Temporary access 39 Replication Publisher 1-n Subscribers Account (full environment) replication File replication Organize into ‘groups’ Can Publish & Subscribe from same machine ©2012 Rocket Software, Inc. All Rights Reserved. 40 Replication 3 modes: Deferred – Sync every ‘x’ time Immediate – Commits once staged Real-time – Commits once sync’d ©2012 Rocket Software, Inc. All Rights Reserved. 41 Replication Publishing System Subscribing System xadmin uvrepmanager uvrepmanager xadmin Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA) Subscriber Packet File (SPF) uvsh Replication Buffer Replication Buffer uvrw Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF) 42 Replication Publishing System Subscribing System xadmin uvrepmanager uvrepmanager xadmin Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA) Subscriber Packet File (SPF) uvsh Replication Buffer Replication Buffer uvrw Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF) 43 Replication Publishing System Subscribing System xadmin uvrepmanager uvrepmanager xadmin Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA) Subscriber Packet File (SPF) uvsh Replication Buffer Replication Buffer uvrw Log Extension Log Extension Publisher Files Subscriber Files Files (LEF) Files (LEF) 44 Replication Publishing System Subscribing System xadmin uvrepmanager uvrepmanager xadmin Transaction Control uvpub uvsub Transaction Control Area (TCA) uvpublistener sublistener submain Area (TCA) Subscriber Packet File (SPF) uvsh Replication Buffer Replication Buffer uvrw Log Extension Log Extension Publisher Files Subscriber Files