Persistent Memory Programming Made Easy with pmemkv Andy Rudoff Intel Corporation 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 1 Agenda . Why pmemkv? . pmemkv Design . Engines . Language bindings . Performance 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 2 Why pmemkv? 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 3 Ways to Use Persistent Memory . Memory Mode . Transparent to application . Transparent to OS . Volatile . …but not a match for every use case . As Storage… 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 4 The SNIA NVM Programming Model file memory Management UI Application Application Application space user Standard Standard Standard Load/ Raw Device File API File API Store Management Library Access “DAX” File System space kernel pmem-Aware MMU File System Mappings Generic NVDIMM Driver Persistent Memory 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 5 With Great Direct Access Comes Great Responsibility Flushing… RAS… Program Initialization Program Initialization DAX mapped file? Dirty Shutdown? (OS provides info) no yes yes no Data set is potentially inconsistent. Use standard API for flushing Recover. (msync/fsync or FlushFileBuffers) CPU caches Known Poison Blocks considered persistent? (ACPI provides info) yes no no yes Repair data set Normal Operation Stores considered persistent CLWB? when globally-visible (CPU_ID provides info) no yes Use CLWB+SFENCE Consistency… CLFLUSHOPT? for flushing (CPU_ID provides info) no yes open(…); mmap(…); Use CLFLUSHOPT+SFENCE Use CLFLUSH for flushing for flushing strcpy(pmem, "Hello, World!"); msync(…); Crash 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 6 PMDK “make pmem programming easier” PCJ – Persistent Collection for PCJ/ C++ C Python pmemkv Java LLPL C+ Java JS Ruby C + Application Load/Store Standard File API User Space PMDK Interface to create a Interface for persistent memory Transaction persistent memory resident allocation, transactions and general Interface to create arrays of facilities Support log file pmem-resident blocks, of same size, atomically updated MMU pmem-Aware Mappings libpmemlog libpmemobj libpmemblk File System Kernel Space Support for volatile memory usage Low level support for NVDIMM Low level support for remote access to memkind local persistent memory persistent memory Low-level support vmemcache libpmem librpmem In Development 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 7 libpmemobj Performance Across Versions B-tree Benchmark 549.66% 511.33% 377.12% 510.59% 167.49% 351.09% 100.00% 199.07% 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 8 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Why pmemkv? . We have Memory Mode and Storage . for legacy use cases . We have libpmem . for raw access . We have libpmemobj . For transactions, allocation, containers . How about a simple put/get API! . Tuned and validated to product quality 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 9 Why pmemkv? • Key-value store can take advantage of access-in-place 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 10 pmemkv design 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 11 pmemkv design goals Technical: Community: • Local key/value store (no networking) • Idiomatic language bindings • Open source, developed in the open and • Simple, familiar, bulletproof API friendly licensing • Easily extended with new engines • https://github.com/pmem/pmemkv • Optimized for persistent memory • Outside contributions are welcome (limit copying to/from DRAM) • Intel provides stewardship, validation on real • Flexible configuration, not limited to a single hardware, and code reviews storage algorithm • Standard/comparable benchmarks • Generic tests 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 12 pmemkv architecture applications • pmemkv core is a frontend for engines C++ JavaScript Java Ruby C applications applications applications applications applications • Core implementation written in C++, Node.js not related to Persistent Memory bindings • Pluggable engines • Some engines are implemented in bindings NAPI Java Ruby bindings bindings pmemkv, some engines are imported C++ API (header from external projects only) JNI FFI • Persistent engines are implemented with libpmemobj (PMDK) C API pmemkv • Native API for pmemkv is written C/C++ pmemkv core (C++) • pmemkv design allows for easy integration with high-level language libpmemobj++ pmemkv memkind TBB “native” bindings libpmemobj engines PMDK pluggable engines 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 13 pmemkv configuration • Flexible configuration API Typical config structure example for libpmemobj-based engines • Works with different kinds of engines • Every engine has documented supported config config cfg; parameters individually • Unordered map status s = cfg.put_string("path", path); assert(s == status::OK); • Takes name configuration value as a k-v pair • Supported configuration types: s = cfg.put_uint64("size", SIZE); • int64/uint64/double assert(s == status::OK); • string s = cfg.put_uint64("force_create", 1); • Arbitrary data (pointer and size) assert(s == status::OK); • Resides on stack • Takes optional destructor as an additional parameter if custom configuration parameter allocates memory 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 14 pmemkv design Application start DRAM Persistent Memory 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 15 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); cfg s = cfg.put_uint64("size", SIZE); assert(s == status::OK); config structure, resides on stack 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 16 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); cfg s = cfg.put_uint64("size", SIZE); assert(s == status::OK); kv db *kv = new db(); db object – volatile object for managing engine 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 17 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); kv s = cfg.put_uint64("size", SIZE); /daxfs/file assert(s == status::OK); k v cfg cmap db *kv = new db(); if (kv->open("cmap", cfg) != status::OK) { std::cerr << db::errormsg() << std::endl; return 1; } kv.open() - creates/opens persistent memory pool - checks consistency and perform recovery - takes ownership for cfg structure 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 18 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); kv s = cfg.put_uint64("size", SIZE); /daxfs/file assert(s == status::OK); k v cfg cmap db *kv = new db(); if (kv->open("cmap", cfg) != status::OK) { std::cerr << db::errormsg() << std::endl; return 1; } // do work here 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 19 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); kv s = cfg.put_uint64("size", SIZE); /daxfs/file assert(s == status::OK); k v cfg cmap db *kv = new db(); if (kv->open("cmap", cfg) != status::OK) { std::cerr << db::errormsg() << std::endl; return 1; } // do work here kv.close() - close database connection kv->close(); - Persistent Memory data remain saved 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 20 pmemkv design config cfg; status s = cfg.put_string("path", "/daxfs/file"); DRAM Persistent Memory assert(s == status::OK); s = cfg.put_uint64("size", SIZE); /daxfs/file assert(s == status::OK); k v cmap db *kv = new db(); if (kv->open("cmap", cfg) != status::OK) { std::cerr << db::errormsg() << std::endl; return 1; } // do work here kv->close(); Safe deletion of volatile data delete kv; 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 21 Engines 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 22 Engines Engine Name Description Experimental? Persistent? Concurrent? Sorted? blackhole Accepts everything, returns nothing No - - - cmap Concurrent hash map No Yes Yes No vsmap Volatile sorted hash map No No No Yes vcmap Volatile concurrent hash map No No Yes No tree3 Persistent B+ tree Yes Yes No No stree Sorted persistent B+ tree Yes Yes No Yes Caching for remote Memcached or caching Yes Yes No - Redis server Sorted concurrent map csmap Yes Yes Yes Yes (under development) 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 23 Engines • Engine contributions are welcome! • Types: • ordered/unordered • persistent/volatile • concurrent/single threaded • Engines are optimized for different workloads & capabilities • All engines work with all language bindings • Generic tests for engines include: • memcheck • helgrind/drd • pmemcheck • pmemreorder 2019 Storage Developer Conference. © Intel Corporation. All Rights Reserved. 24 Multiple Engines Within the Same Memory Pool • pmemkv does not limit you to a single engine to a single memory pool Persistent Memory • Engines are reachable from root object /daxfs/file