Deep Dive Dynamo DB
Total Page:16
File Type:pdf, Size:1020Kb
Deep Dive: Amazon DynamoDB Richard Westby-Nunn – Technical Account Manager, Enterprise Support 28 June 2017 © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda DynamoDB 101 - Tables, Partitioning Indexes Scaling Data Modeling DynamoDB Streams Scenarios and Best Practices Recent Announcements Amazon DynamoDB Fully Managed NoSQL Document or Key-Value Scales to Any Workload Fast and Consistent Access Control Event Driven Programming Tables, Partitioning Table Table Items Attributes All items for a partition key Partition Sort ==, <, >, >=, <= Key Key “begins with” Mandatory “between” Key-value access pattern sorted results Determines data distribution Optional counts Model 1:N relationships Enables rich query capabilities top/bottom N values paged responses Partition table Partition key uniquely identifies an item Partition key is used for building an unordered hash index Table can be partitioned for scale Id = 1 Id = 2 Id = 3 Name = Jim Name = Andy Name = Kim Dept = Engg Dept = Ops Hash (1) = 7B Hash (2) = 48 Hash (3) = CD 0000 54 55 Key Space A9 AA FF Partition-sort key table Partition key and sort key together uniquely identify an Item Within unordered partition key-space, data is sorted by the sort key No limit on the number of items (∞) per partition key • Except if you have local secondary indexes Partition 1 Partition 2 Partition 3 00:0 54:∞ 55 A9:∞ AA FF:∞ Customer# = 2 Customer# = 1 Customer# = 3 Order# = 10 Order# = 10 Order# = 10 Item = Pen Item = Toy Item = Book Customer# = 2 Customer# = 1 Customer# = 3 Order# = 11 Order# = 11 Order# = 11 Item = Shoes Item = Boots Item = Paper Hash (2) = 48 Hash (1) = 7B Hash (3) = CD Partitions are three-way replicated Id = 2 Id = 1 Id = 3 Name = Andy Name = Jim Name = Kim Dept = Engg Dept = Ops Replica 1 Id = 2 Id = 1 Id = 3 Name = Andy Name = Jim Name = Kim Dept = Engg Dept = Ops Replica 2 Id = 2 Id = 1 Id = 3 Name = Andy Name = Jim Name = Kim Dept = Engg Dept = Ops Replica 3 Partition 1 Partition 2 Partition N Eventually Consistent Reads Rand(1,3) Replica 1 Replica 2 Replica 3 Partition A Partition A Partition A 1000 RCUs 1000 RCUs 1000 RCUs Strongly Consistent Reads Locate Primary Replica 1 Replica 2 Replica 3 Partition A Partition A Partition A 1000 RCUs 1000 RCUs 1000 RCUs Primary Indexes Local secondary index (LSI) Alternate sort key attribute Index is local to a partition key 10 GB max per partition Table A1 A2 A3 A4 A5 (partition) (sort) key, i.e. LSIs limit the # of sort keys! A1 A3 A2 KEYS_ONLY (partition) (sort) (table key) A1 A4 A2 A3 LSIs (partition) (sort) (table key) (projected) INCLUDE A3 A1 A5 A2 A3 A4 ALL (partition) (sort) (table key) (projected) (projected) Global secondary index (GSI) Online Indexing Alternate partition (+sort) key Index is across all table partition keys A1 A2 A3 A4 A5 RCUs/WCUs Table (partition) provisioned separately for GSIs A2 A1 KEYS_ONLY (part.) (table key) A5 A4 A1 A3 INCLUDE A3 GSIs (part.) (sort) (table key) (projected) A4 A5 A1 A2 A3 (part.) (sort) (table key) (projected) (projected) ALL LSI or GSI? LSI can be modeled as a GSI If data size in an item collection > 10 GB, use GSI If eventual consistency is okay for your scenario, use GSI! Scaling Scaling Throughput § Provision any amount of throughput to a table Size § Add any number of items to a table - Max item size is 400 KB - LSIs limit the number of range keys due to 10 GB limit Scaling is achieved through partitioning Throughput Provisioned at the table level • Write capacity units (WCUs) are measured in 1 KB per second • Read capacity units (RCUs) are measured in 4 KB per second • RCUs measure strictly consistent reads • Eventually consistent reads cost 1/2 of consistent reads Read and write throughput limits are independent RCU WCU Partitioning Math Number of Partitions By Capacity (Total RCU / 3000) + (Total WCU / 1000) By Size Total Size / 10 GB Total Partitions CEILING(MAX (Capacity, Size)) Partitioning Example Table size = 8 GB, RCUs = 5000, WCUs = 500 Number of Partitions By Capacity (5000 / 3000) + (500 / 1000) = 2.17 By Size 8 / 10 = 0.8 Total Partitions CEILING(MAX (2.17, 0.8)) = 3 RCUs per partition = 5000/3 = 1666.67 RCUs and WCUs are uniformly WCUs per partition = 500/3 = 166.67 spread across partitions Data/partition = 10/3 = 3.33 GB Burst capacity is built-in Consume saved up capacity 1600 1200 Burst: 300 seconds (1200 × 300 = 360k CU) 800 Capacity Units 400 “Save up” unused capacity 0 Time Provisioned Consumed Burst capacity may not be sufficient 1600 1200 Burst: 300 seconds (1200 × 300 = 360k CU) 800 Capacity Units Throttled requests 400 0 Time Provisioned Consumed Attempted Don’t completely depend on burst capacity… provision sufficient throughput What causes throttling? Non-uniform workloads • Hot keys/hot partitions • Very large bursts Dilution of throughput across partitions caused by mixing hot data with cold data • Use a table per time period for storing time series data so WCUs and RCUs are applied to the hot data set Example: Key Choice or Uniform Access Heat Partition Time Getting the most out of DynamoDB throughput “To get the most out of DynamoDB 1. Key Choice: High key throughput, create tables where cardinality (“uniqueness”) the partition key has a large number of distinct values, and 2. Uniform Access: access is evenly spread over the key- values are requested fairly space uniformly, as randomly as possible.” 3. Time: requests arrive evenly spaced in time —DynamoDB Developer Guide Example: Time-based Access Example: Uniform access Data Modeling Store data based on how you will access it! 1:1 relationships or key-values Use a table or GSI with a partition key Use GetItem or BatchGetItem API Example: Given a user or email, get attributes Users Table Partition key Attributes UserId = bob Email = [email protected], JoinDate = 2011-11-15 UserId = fred Email = [email protected], JoinDate = 2011-12-01 Users-Email-GSI Partition key Attributes Email = [email protected] UserId = bob, JoinDate = 2011-11-15 Email = [email protected] UserId = fred, JoinDate = 2011-12-01 1:N relationships or parent-children Use a table or GSI with partition and sort key Use Query API Example: Given a device, find all readings between epoch X, Y Device-measurements Part. Key Sort key Attributes DeviceId = 1 epoch = 5513A97C Temperature = 30, pressure = 90 DeviceId = 1 epoch = 5513A9DB Temperature = 30, pressure = 90 N:M relationships Use a table and GSI with partition and sort key elements switched Use Query API Example: Given a user, find all games. Or given a game, find all users. User-Games-Table Game-Users-GSI Part. Key Sort key Part. Key Sort key UserId = bob GameId = Game1 GameId = Game1 UserId = bob UserId = fred GameId = Game2 GameId = Game2 UserId = fred UserId = bob GameId = Game3 GameId = Game3 UserId = bob Documents (JSON) Javascript DynamoDB string S Data types (M, L, BOOL, NULL) number N introduced to support JSON boolean BOOL Document SDKs • Simple programming model null NULL • Conversion to/from JSON array L • Java, JavaScript, Ruby, .NET Cannot create an Index on object M elements of a JSON object stored in Map • They need to be modeled as top- level table attributes to be used in LSIs and GSIs Set, Map, and List have no element limit but depth is 32 levels Rich expressions Projection expression • Query/Get/Scan: ProductReviews.FiveStar[0] Filter expression • Query/Scan: #V > :num (#V is a place holder for keyword VIEWS) Conditional expression • Put/Update/DeleteItem: attribute_not_exists (#pr.FiveStar) Update expression • UpdateItem: set Replies = Replies + :num DynamoDB Streams DynamoDB Streams Stream of updates to a table Highly durable Asynchronous • Scale with table Exactly once 24-hour lifetime Strictly ordered Sub-second latency • Per item DynamoDB Streams and AWS Lambda Scenarios and Best Practices Event Logging Storing time series data Time series tables Events_table_2016_April Event_id Timestamp Attribute1 …. Attribute N RCUs = 10000 Current table (Partition key) (sort key) WCUs = 10000 Hot data Hot Events_table_2016_March RCUs = 1000 Event_id Timestamp Attribute1 …. Attribute N (Partition key) (sort key) WCUs = 100 Events_table_2016_Feburary Older tables RCUs = 100 Event_id Timestamp Attribute1 …. Attribute N (Partition key) (sort key) WCUs = 1 Cold data Cold Events_table_2016_January RCUs = 10 Event_id Timestamp Attribute1 …. Attribute N (Partition key) (sort key) WCUs = 1 Don’t mix hot and cold data; archive cold data to Amazon S3 DynamoDB TTL Events_table_2016_April Event_id Timestamp myTTL …. Attribute N RCUs = 10000 Current table (Partition key) (sort key) 1489188093 WCUs = 10000 Hot data Hot Events_Archive Event_id Timestamp Attribute1 …. Attribute N RCUs = 100 (Partition key) (sort key) WCUs = 1 Cold data Cold Use DynamoDB TTL and Streams to archive Isolate cold data from hot data Pre-create daily, weekly, monthly tables Provision required throughput for current table Writes go to the current table Turn off (or reduce) throughput for older tables OR move items to separate table with TTL Dealing with time series data Real-Time Voting Write-heavy items Scaling bottlenecks Voters Provision 200,000 WCUs Partition 1 Partition K Partition M Partition N 1000 WCUs 1000 WCUs 1000 WCUs 1000 WCUs Candidate A Candidate B Votes Table Write sharding Voter UpdateItem: “CandidateA_” + rand(0, 10) ADD 1 to Votes Candidate A_7 Candidate B_4 Candidate B_8 Candidate A_1 Candidate A_4 Candidate B_5 Candidate B_1 Candidate A_5 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_2 Candidate A_6 Candidate A_8 Votes Table Candidate B_2 Candidate B_6 Shard aggregation 2. Store Voter Periodic Process 1. Sum Candidate A_7 Candidate B_8 Candidate A Candidate B_4 Candidate A_1 Candidate A_4 Total: 2.5M Candidate B_5 Candidate B_1 Candidate A_5 Candidate B_3 Candidate B_7 Candidate A_3 Candidate A_2 Candidate A_6 Candidate A_8 Votes Table Candidate B_2 Candidate B_6 Shard write-heavy partition keys Trade off read cost for write scalability Consider throughput per partition key and per partition Your write workload is not horizontally scalable Product Catalog Popular items (read) Scaling bottlenecks SELECT Id, Description, ..