H-Store And VoltDB Yiqun (Ethan) Zhang December 3, 2018 Carnegie Mellon University [email protected] https://www.linkedin.com/in/yzhang1991 2 fast transactions 3 4 AGENDA • History 5 AGENDA • Architectural Overview 6 AGENDA • How VoltDB diverged from H-Store 7 AGENDA • New research followed H-Store 8 read-only long-running complex joins exploratory queriesOLAP 9 compression 10 What if state fits in memory? • A lot of data sets do fit in memory • 100 MB per warehouse in TPC-C • Even data for 1,000 such warehouses can still fit! 11 12 Where did we spend our time? 100% CPU Cycle Breakdown for Shore on TPC-C New Order Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 13 Where did we spend our time? 100% CPU Cycle Breakdown for Shore on TPC-C New Order Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 13 Where did we spend our time? 70.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 13 Where did we spend our time? 70.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 13 Where did we spend our time? 70.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 14 OLTP transactions are short-lived • The heaviest TPC-C transaction: • reads/writes ~200 records; • can be finished in less than 1 millisecond; • CPU is not the bottleneck. 15 Where did we spend our time? run single-threaded 33.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 16 Single-threaded problems • Waiting on users leaves CPU idle. • Single-threaded does not jive with the multicore world. 17 Transactions are repetitive • Queries are known in advance; • Control flows are settled in advance too. • External transaction control can be converted into pre- compiled stored procedures with structured control code intermixed with parameterized SQL commands on the server. 18 Waiting on users external transaction control • Don’t • External transaction control and performance are not friends; • Use server-side transactional logic; • Move the logic to data, not the other way around; 19 Using ALL the cores • Partitioning data is a requirement for scale-out. • Single-threaded is desired for efficiency. Why not partition to the core instead of the node? • Concurrency via scheduling, not shared memory. 20 Where did we spend our time? run single-threaded 33.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 21 Where did we spend our time? durability through replication run single-threaded 33.4% CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 21 Where did we spend our time? undo logging durability through replication run single-threaded CPU Cycle Breakdown for Shore on TPC-C New Order in-memory data storage Source: Harizopoulos, Abadi, Madden and 19.6% Stonebraker, “OLTP Under the Looking Glass”, SIGMOD 2008 21 What did we end up building? • In-memory relational SQL database. • No external transaction control – Stored Procedures • Single-threaded engines run in parallel. • Partitioned to the core. • Concurrency via Scheduling, not shared memory. • Serializable ACID. • Durability through Replication 22 Architecture 23 24 Run in parallel In-memory store 25 Single-threaded engine Node #1 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME CREDITS 26 Node #1 Replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME CREDITS 26 Node #1 Read from a replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS Command Router 27 Node #1 Read from a replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS Command Router 27 Node #1 Write to a replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS Command Router 28 Node #1 Write to a replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS Command Router 28 Node #1 Write to a replicated table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS Command Router 28 Node #1 Partitioned Table PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME CREDITS 29 Node #1 Single Partition Read 123, Brown, Joe, 100 456, Silvers, Phil, 77 select * from PLAYER where PLAYER_ID = 687 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 30 Node #1 Single Partition Read 123, Brown, Joe, 100 456, Silvers, Phil, 77 select * from PLAYER where PLAYER_ID = 687 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS 345, White, Betty, 94 Command Router 687687,, BlackBlack,, Mark,Mark, 5555 525, Snow, Ann, 73 30 Node #1 Single Partition Read 123, Brown, Joe, 100 456, Silvers, Phil, 77 select * from PLAYER where PLAYER_ID = 687 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS 345, White, Betty, 94 Command Router 687687,, BlackBlack,, Mark,Mark, 5555 525, Snow, Ann, 73 30 Node #1 Single Partition Write 123, Brown, Joe, 100 456, Silvers, Phil, 77 update PLAYER set CREDITS = 50 where PLAYER_ID = 123 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 31 Node #1 Single Partition Write 123, Brown, Joe, 50100 456, Silvers, Phil, 77 update PLAYER set CREDITS = 50 where PLAYER_ID = 123 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 31 Node #1 Single Partition Write 123, Brown, Joe, 50100 456, Silvers, Phil, 77 update PLAYER set CREDITS = 50 where PLAYER_ID = 123 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 31 Node #1 Multi Partition Read 123, Brown, Joe, 50 456, Silvers, Phil, 77 select * from PLAYER where CREDITS > 75 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 32 Node #1 Multi Partition Read 123, Brown, Joe, 50 456, Silvers, Phil, 77 select * from PLAYER where CREDITS > 75 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 32 Node #1 Multi Partition Read 123, Brown, Joe, 50 456, Silvers, Phil, 77 select * from PLAYER where CREDITS > 75 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 32 Node #1 Multi Partition Write 123, Brown, Joe, 50 456, Silvers, Phil, 77 update PLAYER set CREDITS = 0 where CREDITS < 60; 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 33 Node #1 Multi Partition Write 123, Brown, Joe, 500 456, Silvers, Phil, 77 update PLAYER set CREDITS = 0 where CREDITS < 60; 234, Green, Peter, 410 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 550 525, Snow, Ann, 73 33 Node #1 Multi Partition Write 123, Brown, Joe, 500 456, Silvers, Phil, 77 update PLAYER set CREDITS = 0 where CREDITS < 60; 234, Green, Peter, 410 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME WRITE CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 550 525, Snow, Ann, 73 33 Multi Partition Writes • Need two-phase commit. • Simple solution – block until the transaction finishes. • Introduces network stall - BAD. 34 Single Partition case Client Coordinator Partition 1 execute execute execute execute 35 Multi Partition case Client Coordinator Partition 1 execute execute execute ...... Partition 2 execute execute execute ...... network stall network stall 36 Multi Partition case Client CoordinatorEarly Acknowledgement Partition 1 execute execute execute ...... Partition 2 execute execute execute ...... network stall network stall 36 Node #1 Durability not guaranteed 123, Brown, Joe, 100 456, Silvers, Phil, 77 select * from PLAYER where PLAYER_ID = 687 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 37 Node #1 Durability not guaranteed 123, Brown, Joe, 100 456, Silvers, Phil, 77 select * from PLAYER where PLAYER_ID = 687 234, Green, Peter, 41 567, Brown, Mary, 68 PLAYER PLAYER_ID LAST_NAME Node #2 FIRST_NAME READ CREDITS 345, White, Betty, 94 Command Router 687, Black, Mark, 55 525, Snow, Ann, 73 37 Node #1 Durability not guaranteed 123, Brown, Joe, 100 456, Silvers,