Optimize Redis with NextGen NVM Shu Kevin/Si, Peifeng/Li, Zhiming Intel Corporation 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 1 Agenda Redis introduction NVM introduction Scenario 1: Use NVM to increase Redis capacity Scenario 2: Use NVM to improve the performance of Redis persistency Completed features support of Redis on NVM LRU & Defrag Linux Copy-on-Write on NVM Summary 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 2 Redis Introduction Redis DB Redis is an open-source in-memory K-V database that offers high performance, Dictht Dictht replication, and a unique data model with dictEntry dictEntry dictEntry ... ... optional durability. 1 2 N Key value next Redis data in logging Appendonly.aof type encoding ptr DRAM STRING RAW pointer Sdshdr Disk Data change command String List Hash Set Zset Client Fig1. Redis persistency - AOF Fig2. Redis data structures 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 3 Agenda Redis introduction NVM introduction Scenario 1: Use NVM to increase Redis capacity Scenario 2: Use NVM to improve performance of Redis persistency Completed features support of Redis on NVM LRU & Defrag Linux Copy-on-Write on NVM Summary 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 4 NVM Highlight Large capacity Lower $ / GB Close to DRAM throughput and latency Persistency - may or may not use 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 5 Agenda Redis introduction NVM introduction Scenario 1: Use NVM to increase Redis capacity Scenario 2: Use NVM to improve the performance of Redis persistency Completed features support of Redis on NVM LRU & Defrag Linux Copy-on-Write on NVM Summary 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 6 Use NVM to increase Redis capacity Run 24 Redis server Run 24 Redis client Design Option #1 - instances instances Store all heap data in NVM XEON XEON Replace jemalloc by 24 40G Eth 24 CORES CORES libmemkind Minimum Redis code Redis Test Environment: change needed • Single CPU node, 2-2-2 configuration • Stress redis server with same amount of multiple client instances NVM DDR 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 7 Use NVM to increase Redis capacity Design option #1 - Store all heap data in NVM Performance optimization opportunity: Big and sequential data access pattern has better performance than small and random data access pattern in NVM Meanwhile in Redis, a lot of management data structures are small and usually be accessed randomly 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 8 Use NVM to increase Redis capacity Design option #2 - Store most of heap data in NVM Only store large value in NVM (>64 byte by default) Keep Redis management data structures in DRAM Optimize the data placement strategy for each data type Performance: Close to Redis performance run on DRAM URL: https://github.com/pmem/pmem-redis 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 9 Use NVM to increase Redis capacity In DCPMM Implementation details: String In DRAM Encoding to STRING: RAW: Store value (>threshold) in DCPMM and store SDS pointer in type encoding ptr DRAM. String RAW pointer sds type encoding ptr INT: Store in the DRAM String INT Integer EMBSTR: Store in the DRAM type encoding ptr String EMBSTR pointer sds 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 10 Use NVM to increase Redis capacity List type encoding ptr In DCPMM Implementation details: LIST QUICKLIST pointer In DRAM Encoding to LIST: QUICKLIST Quicklist Head tail count len After Redis-3.2.7, quicklist is used to Quicklist Quicklist Quicklist implement the list data type. Each node node node quicklist node contains a ziplist structure. Ziplist Ziplist ziplist is to save memory usage for Ziplist small items like int/embedded string Head tail Len:2 entry1 entry2 end Store value (>threshold) in DCPMM INT/embedded prevlen encoding prevlen encoding pointer and store the pointer in ziplist string sds 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 11 Use NVM to increase Redis capacity Hash In DCPMM Implementation details: type encoding ptr In DRAM Encoding to HASH: Hash ZIPLIST pointer Ziplist type encoding ptr ZIPLIST: Store value (>threshold) in Hash HT pointer DCPMM and store the pointer in ziplist dictEntry dictEntry dictEntry ... ... 1 2 N HASHTABLE: Store value (>threshold) in DCPMM and store the pointer in hashtable field value next typeptr encoding ptr type encoding ptr STRINGAddr RAW pointer STRING RAW pointer sds sds 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 12 Use NVM to increase Redis capacity Set In DCPMM Implementation details: type encoding ptr In DRAM Encoding to SET SET INTSET pointer INTSET type encoding ptr INTSET: It is unnecessary to move SET HT pointer intset into DCPMM due to its small dictEntry dictEntry dictEntry size ... ... 1 2 N HASHTABLE: Store value (>threshold) in DCPMM and store field value next the pointer in hashtable type encoding ptr type encoding ptr STRING RAW pointer STRING RAW pointer sds sds 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 13 Use NVM to increase Redis capacity Zset · In DCPMM Implementation details: type encoding ptr ZSET ZIPLIST pointer Ziplist In DRAM Encoding to ZSET type encoding ptr ZSET SKIPLIST pointer ZIPLIST: Value (>threshold) stored in DCPMM and the pointer Zskip Zskip Zskip Zskip stored in ziplist Head node node node node SKIPLIST: Value (>threshold) stored in DCPMM and the pointer stored in skiplist score 1 2 3 4 robj pointer pointer pointer pointer sds sds sds sds 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 14 Agenda Redis introduction NVM introduction Scenario 1: Use NVM to increase Redis capacity Scenario 2: Use NVM to improve the performance of Redis persistency Completed features support of Redis on NVM LRU & Defrag Linux Copy-on-Write on NVM Summary 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 15 Use NVM to improve the performance of Redis persistency Design option #1 – Persist everything in NVM Use libpmemobj to store data and its mgmt. structures in NVM Source: URL: https://github.com/pmem/redis/tree/3.2-nvml 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 16 Use NVM to improve the performance of Redis persistency Design option #2 – Pointer based AOF Store key in DDR and AOF (same to Open Source Redis) Store value in NVM (for persistency) and only store its pointer in AOF(for recover) Leverage AOF to guarantee data integrity Performance: Much better than Open Source Redis AOF (sync=always) URL: https://github.com/pmem/pmem-redis 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 17 Agenda Redis introduction NVM introduction Scenario 1: Use NVM to increase Redis capacity Scenario 2: Use NVM to improve the performance of Redis persistency Completed features support of Redis on NVM LRU & Defrag Linux Copy-on-Write on NVM Summary 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 18 Completed features support of Redis on NVM Besides data movement optimization, also covers below features for NVM adoption: Data retirement support: LRU on NVM Evict the objects only when both DDR and NVM are full Support to evict objects in NVM Defragmentation on NVM In Redis 4.0, leverage jemalloc to support defrag the space in NVM 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 19 Completed features support of Redis on NVM Pitfall – Linux Copy-on-Write is not supported on NVM Problem Statement • Redis leverages CoW to do RBD snapshot, replication, etc. • The NVM space doesn’t support CoW, because it is based on a memory mapped file. Hence the parent and child process share the same NVM address space, which may cause data corrupt during COW. Solution • During CoW, duplicate objects in parent process of Redis • Introduce two hash tables to help the parent process to decide if it needs to duplicate the object or not 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 20 Summary Optimized Redis with Next Gen NVM can achieve: Full compatible API and Functionality Compatible with open source Redis 4.0+ Higher capacity per instance with lower TCO Way to use NVM as a volatile device to provide larger capacity for in- memory database Higher Perf on persistency Novel design to use NVM for high performance persistency on Redis 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 21 Reference Persistent Memory Development Kit http://pmem.io/pmdk/libpmem/ Libmemkind https://github.com/memkind/memkind Libpmem http://pmem.io/pmdk/libpmem/ Open Source Redis download https://redis.io/ Optmized PMEM Redis Repo https://github.com/pmem/pmem- redis 2018 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 22.