Key-Value Graph Document Column-family summary Polygot persistence NoSQL Systems February 15, 2016 NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Data in a key-value store is organized around the associative array (a.k.a. Map, Dictionary, Hash) Key-value stores contain collection of key-value pair with each key being unique in the collection. They store any value in the value field including data structures, e.g., JSON. They operate as caches or data structures. They do not use SQL and have a flexible schema. An important feature of Key-Value stores is the sharding of data that delivers scale-out architecture. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Consistency only for operations on a single key (operations are limited to get, put, or delete on a single key). Optimistic writes can be performed, but are very expensive to implement, because a change in value cannot be determined by the data store. Eventual consistency in Riak ! update conflicts: newest write wins or return values to let client decide. Scaling uses sharding on the key NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Different strategies for KV stores. In general, no guarantees on writes. Riak uses the concept of Quorum (get write tolerance that way, W=3, R=5) Write quorum: W > N=2, the number of nodes participating in the Writes (W) must be more than half the number of nodes involved in replication (N). Ex: Replication factor of 3, only 2 nodes need to confirm on writes. Read quorum (how many nodes do I need to contact to ensure I have the most up to date value): R + W > N so R > N − W . NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Use cases Storing session information, e.g., web sessions User profiles and preferences Shopping cart data Do not use for: joins between keys, multioperation transactions (support for rollback), need for secondary indexes, operations by sets. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Memcached NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence stands for Memory Cache Daemon in-memory indexing Implementation started in 2003 at LiveJournal (social blogging sites) Project Danga Interactive, BSD licence Implemented in C Used by YouTube, Facebook, Twitter, Orange, GAE, AWS, Flickr, Slashdot, etc. Characteristics: distributed memory, caching system. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence It is a server that caches Name Value Pairs (NVPs) in memory. Value can be anything: rows of data, HTML, binary objects. When some data are needed, the system checks if it is available in Memcached and if it is not the case retrieves it from disk and stores it in Memcached for future accesses. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Query model 4 main commands taking over a TCP or UDP connection. SET: add a new item or replace an existing one with new data ADD: only store the data if the key does not exist. REPLACE: only store the data if the key already exists. GET: return the data NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence is not a persistent data store. can not be dumped to disk. has no security mechanism built-in. does not support any fail-over/ high availability mechanisms. Least Recently Used (LRU) algo to remove data when space is needed. Expiration time can be associated to NVP NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Components Server: a NVP server. Basically, stores and retrieves data stored with a key. Limitations: length(key)<250 characters. size(value)<1MB. Server is atomic (does not care about other servers). Client: knows which server contains an NVP. Client libraries for most languages. Enables to compress NVPs with values greater than 1MB. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Redis high performance key-value store flexible schema Supports publish-subscrie messaging Supports many data structures: lists, sets, sorted sets, hashes, hyperloglogs, etc. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Hashes collections of key-value pairs map between key string keys and string values Efficient for representing objects and tabular data Operations: HLEN, HKEYS, HVALS, HDEL, HEXISTS, HINCRBY, HINCRBYFLOAT NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Sets unordered collections of strings Similar to lists but all members are unique Possible to add the same element multiple times without needing to check its existence in the set. Add, remove and test existence of members in constant time. Set-based operations (union, intersection, difference) Operations: SADD, SREM, SISMEMBER, SMEMBERS, SDIFF,SDIFFSTORE, SINTER, SINTERSTORE, SPOP, etc. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Sorted sets Ordered collections of strings. a rank field to determine the order. Members are automatically sorted by rank Members must be unique. Can retrieve elemets by position or rank Set-based operations (union, intersection, difference) Add, remove and updating of members is in O(log(n)). Operations: ZADD, ZREM, ZSCORE, ZRANGE, ... NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Bitmaps Bit operations Can count the number of bits set ot 1. Perform AND, OR, XOR, NOT operations Operations: SETBIT, GETBIT, BITOP AND, BITOP OR, BITOP NOT NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Hyperloglogs A probabilistic data structure to count unique elements ! gives an estimation Do not need to keep a copy of all members. Operations: ZADD, ZREM, ZSCORE, ZRANGE, ... NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence uses the BerkeleyDB storage engine. provides transactional semantics with fined-grained concurrency primary and secondary indexes. high availability NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Amazon DynamoDB automatic sharding Client: AdRoll serves 100 billion ad impressions per day. NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Aerospike Hybrid in-memory: dynamic random access memory (DRAM) and SSD self-healing architecture NoSQL Systems Key-Value Introduction Graph Consistency - Scaling Document Transactions - Use cases Column-family Memcached - Redis summary Oracle NoSQL, DynamoDB, Aerospike, Riak Polygot persistence Riak Riak secondary indexes, full-text search, MapReduce compliant Couchbase: membase NoSQL Systems Key-Value Graph Introduction Document Consistency Column-family Scaling - Use cases summary Neo4J Polygot persistence Stores entities (aka nodes) and relationships (aka edges) between these entities. Nodes and edges can have properties (aka attributes) in the property graph model. Edges may be directed or not. A query on a graph is known as traversing the graph. Traversing is fast because joins are not calculated but persisted. NoSQL Systems Key-Value Graph Introduction Document Consistency Column-family Scaling - Use cases summary Neo4J Polygot persistence Some solutions do not support