Oracle TimesTen In-Memory Database Architecture, Performance Tips, Use Cases Chris Jenkins ([email protected]) Senior Director, In-Memory Technology TimesTen Product Management Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 3 Best In-Memory Databases: For Both OLTP and Analytics In-Memory for OLTP Oracle TimesTen In-Memory Database Application Application Application • Lightweight, highly-available IMDB • Primary use case: Extreme OLTP Application Application Application • Microsecond response time • Millions of TPS on commodity hardware In-Memory for Analytics Oracle Database In-Memory Option • Dual-Format In-Memory Database • Primary use case: Real Time Analytics • Billions of Rows/Sec scan rate • Faster mixed-workload enterprise OLTP Fewer indexes needed to support analytics Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 4 Agenda 1 What is TimesTen? 2 TimesTen Classic Architecture 3 TimesTen Scaleout Architecture 4 Performance Tips & Tricks 5 When to use TimesTen? 6 Summary Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Agenda 1 What is TimesTen? 2 TimesTen Classic Architecture 3 TimesTen Scaleout Architecture 4 Performance Tips & Tricks 5 When to use TimesTen? 6 Summary Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Oracle TimesTen In-Memory Database One product, two deployment modes TimesTen Classic TimesTen Scaleout Application Application Reads Application Reads read/writes from Standby from Subscribers Subscriber Active Standby Subscriber Single System Image Subscriber In-Memory Database • Replicated In-Memory Relational Database • Scale-Out In-Memory Relational Database • Highly Available • Highly Available • Extremely low latency reads and writes • Extremely high throughput reads and • Read scaling across multiple hosts writes • Cache Groups (cache for Oracle DB EE) • Scales both reads and writes Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 7 Agenda 1 What is TimesTen? 2 TimesTen Classic Architecture 3 TimesTen Scaleout Architecture 4 Performance Tips & Tricks 5 When to use TimesTen? 6 Summary Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | TimesTen Classic In-memory RDBMS Persistent and Recoverable – Pure in-memory – Database persisted on local storage – Entire database in RAM – Automatic recovery after failure – ACID compliant – Standard SQL and APIs Extremely Fast Highly Available – Low, consistent response time – Active-standby and multi-master – Very high throughput replication – Excellent scalability – Highly parallel, high throughput – Async and Sync – HA and DR Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 9 Architecture: Classic Instance • Installation Client-Server Application TimesTen Oracle – An unzipped copy of the Applications Replica DB RDBMS TimesTen software package and Tools TT Client – Immutable Network • Instance Server Replication daemon Replication agent(s)agent(s) ServerServer – Created using ttInstanceCreate proxies MillionthsCache Application proxies agent(s) CodeApplication of a – A runnable copy of the Code Admin/Utility TimesTen Data Second software TimesTen Data programs Manager Library In-Memory Manager Library – Linked to an installation Database(s) Data Store subdaemon(s) – Direct mode Includes configuration files Applications Checkpoint – Identified by TIMESTEN_HOME and Tools Millionths files – Set of processes of a Log files Second Log TimesTen Buffer – Supports one or more daemon databases Metadata, Tables, Data Tables, Indexes,Indexes, Views, Instance SystemSequences, Tables … Locks, Cursors, Locks, Cursors, Command cache, cache, Command Temp Indexes, … Temp Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 10 Application Connectivity Two modes, functionally equivalent , supported for all APIs (ODBC, OCI, JDBC, ODP.NET, …) Host 1 Host 2 App 1 TT Server 1 App 1 You can mix Database and match App 2 TT Server 2 App 2 these modes as Shared desired based … Memory … … on your requirements. TT Server X App N App N Network Direct mode Client/server mode • Apps run on same host as database • TCP/IP connections between apps and TT server processes • Apps directly map database shared memory (via TT engine) • TT server process is a multi-threaded direct mode app • No context switches, no IPC for database access • Each interaction involves 1 or more n/w round trips • Ultra low latency (in process direct memory access) • More overhead, higher latency than direct mode Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 11 Memory Layout Database is a single shared memory segment (separate much smaller segment for PL/SQL) ~20 MB PermSize TempSize LogBufMB Perm Region Temp Region Log Buffer Metadata Temp objects Strand 1 TimesTen is a row store • System catalog tables • Tables • At least for now… D • System views • Indexes Use of huge pages • Sequences • Recommended unless database is b Sort space Strand 2 H • System Databasedata Shared Memory Segment very small Tables Locks • Mandatory on Linux if segment size d >= 256 GB r • Logical Tuple Pages Connections • Physical Tuple Pages … If not using huge pages • Page directories Commit • Lock segment in memory buffers Strand N • MemoryLock=4 Indexes … Consider NUMA effects… • Hash buckets … • B-Tree nodes Persistent Not to scale Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 12 Persistence - Checkpointing Dual checkpoint files, automatic fuzzy checkpoints Two checkpoint files, dbname.ds0 and dbname.ds1 • Each is a full image of DbHdr + Perm • Written to alternately by checkpoint operations Perm Region Perm region is divided into pages • Variable size based on contents 10 01 01 010 • Two bits to track ‘dirty’ state wrt to checkpoint files Last checkpoint was to .ds1 at some earlier time T0 Pg0 Pg1 At time T1 D • Page 0 is dirty wrt .ds0 and .ds1 0 • Page 2 is dirty wrt .ds0 b 1 0 Pg2 Checkpoint occurs at time T2 H • .ds0 is oldest file (previous checkpoint was to .ds1) d • DbHdr + pages 0 & 2 are written to .ds0 (in place update) r • Pages 0 & 2 dirty bits for .ds0 are cleared … At time T3 .ds1 • Page 1 is modified (e.g. by a DML operation) .ds0 • Both dirty bits are set in the page 0 0 • .ds0 dirty: Pg1 • .ds1 dirty: Pg0, Pg1 PgN Checkpoint occurs at time T4 • .ds1 is oldest file • DbHdr + pages 0 and 1 are written to .ds1 • Dirty bit for .ds1 cleared in both pages Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13 Persistence – Transaction Logging Parallel log manager High throughput and concurrency • Multi-strand buffer (multiple ring buffers) • Behaves as a single shared logical buffer Log files In-memory • Configurable maximum size Database • Sequence number (64-bit) • Old files purged by checkpointing Highly concurrent Log Buffer • Concurrent record post to each strand AND Strand 1 Strand N • Flush buffer to disk … Asynchronous and synchronous operation • Asynchronous (DurableCommits=0) is the default Async Sync • Synchronous (DurableCommits=1) is an option • Many records • Less records written Async/sync configurable at written at once at once • Database level • Outside transaction • Part of commit • Connection level path • Write through to • Transaction level (application API) • Decoupled from media Log usage commit • Commit blocks until • Undo and Redo I/O completion • Replication, XLA, AWT caching & incremental backups • Group commit Log file purge criteria Current optimisation • No records in file belong to an open transaction Log File • Changes for all records written to both checkpoint files • Not required by replication, XLA, AWT or backups Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 14 Persistence – Recovery Occurs on every ramLoad operation Most recent checkpoint loaded into memory Database A • Physical operation Shared p • Segment p Can use multiple reader threads (configurable) s Log replay • From the corresponding checkpoint log record… Rollback any • …to end of log open txns • Mark any indexes that would be modified Rollback any open transactions Drop marked indexes • No commit marker seen Rebuild them Drop and re-build marked indexes Ckpt Log • Index modifications are not redo logged Files files • Done in parallel (configurable) Static checkpoint to oldest checkpoint file Allow application connections Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 15 Concurrency Single level versioning for READ COMMITTED isolation READ COMMITTED isolation (default) • Readers don’t block writers and vice versa MYTABLE READER • Readers don’t place any locks pKey columnX 0 This is row 0 INSERT/UPDATE/DELETE 1 This is row 1 • Creates a new (uncommitted) version of affected rows 2 This is row 2 • eXclusively locks the row(s) 3 This is row 3 Reader • UPDATE 44 ThisNew is row row 4 4 4 This is row 4 Reads old (committed) version(s) 5 This is row 5 Old version(s) deleted on COMMIT or ROLLBACK • In reclaim phase 6 This