Persistent Memory Programming on Conventional Hardware Terence Kelly [email protected] http://ai.eecs.umich.edu/~tpkelly/ SNIA Storage Developer Conference 25 September 2019 c 2019 Terence Kelly, all rights reserved. 1 / 69 Prizes to first three correct outputs Bored? Fidgety? Mini-Hackathon! ACM Queue magazine, July/Aug 2019 (queue.acm.org) Download \famus" library Run \example 01.csh" C-shell script on Linux box E-mail output to [email protected] Subject line \[famus output]" 2 / 69 Bored? Fidgety? Mini-Hackathon! ACM Queue magazine, July/Aug 2019 (queue.acm.org) Download \famus" library Run \example 01.csh" C-shell script on Linux box E-mail output to [email protected] Subject line \[famus output]" Prizes to first three correct outputs 3 / 69 NVM Returns 4 / 69 NVM Programming Style Persistent application data lives in memory only Access/update with LOAD/STORE No separate persistent store e.g., relational database, key-value store 5 / 69 Fewer moving parts: no quirky external store One data format: no serializers/parsers One paradigm: no context switching, \impedance mismatch" Simplicity alone improves cost, correctness, performance NVM Programming Advantage: Simplicity The cheapest, fastest and most reliable components of a computer system are those that aren't there. |Gordon Bell 6 / 69 One data format: no serializers/parsers One paradigm: no context switching, \impedance mismatch" Simplicity alone improves cost, correctness, performance NVM Programming Advantage: Simplicity The cheapest, fastest and most reliable components of a computer system are those that aren't there. |Gordon Bell Fewer moving parts: no quirky external store 7 / 69 One paradigm: no context switching, \impedance mismatch" Simplicity alone improves cost, correctness, performance NVM Programming Advantage: Simplicity The cheapest, fastest and most reliable components of a computer system are those that aren't there. |Gordon Bell Fewer moving parts: no quirky external store One data format: no serializers/parsers 8 / 69 Simplicity alone improves cost, correctness, performance NVM Programming Advantage: Simplicity The cheapest, fastest and most reliable components of a computer system are those that aren't there. |Gordon Bell Fewer moving parts: no quirky external store One data format: no serializers/parsers One paradigm: no context switching, \impedance mismatch" 9 / 69 NVM Programming Advantage: Simplicity The cheapest, fastest and most reliable components of a computer system are those that aren't there. |Gordon Bell Fewer moving parts: no quirky external store One data format: no serializers/parsers One paradigm: no context switching, \impedance mismatch" Simplicity alone improves cost, correctness, performance 10 / 69 Crash-consistency mechanisms Application data in mmap'd files NVM Programming Platforms Research/academic Differences NV-heaps Compiler support Mnemosyne Concurrency/isolation Atlas Industrial Similarities PMDK (Intel) C/C++ NVM Direct (Oracle) 11 / 69 Application data in mmap'd files NVM Programming Platforms Research/academic Differences NV-heaps Compiler support Mnemosyne Concurrency/isolation Atlas Crash-consistency mechanisms Industrial Similarities PMDK (Intel) C/C++ NVM Direct (Oracle) 12 / 69 NVM Programming Platforms Research/academic Differences NV-heaps Compiler support Mnemosyne Concurrency/isolation Atlas Crash-consistency mechanisms Industrial Similarities PMDK (Intel) C/C++ NVM Direct Application data in mmap'd files (Oracle) 13 / 69 Implementable on conventional hardware, without NVM Ye Olde Persistent Memorie Application data in mmap()'d file Sparse backing file: pay-as-you-go storage footprint Pointer cast interprets file as typed application data Persistent heap allocates from mmap()'d file Sliding persistent heap beneath legacy software 14 / 69 Ye Olde Persistent Memorie Application data in mmap()'d file Sparse backing file: pay-as-you-go storage footprint Pointer cast interprets file as typed application data Persistent heap allocates from mmap()'d file Sliding persistent heap beneath legacy software Implementable on conventional hardware, without NVM 15 / 69 Persistent Memory as a Software Abstraction 16 / 69 Outline Part I: Review p-mem programming on conventional hardware (assuming orderly shutdown, i.e., no failures) Part II: Crash consistency (power outages, kernel panics, process crashes) 17 / 69 mmap() Process Virtual in−memory image Address Space: low addr high addr mmap() return value { (start addr) On−Disk Filesystem: backing file { offset length 18 / 69 /* interpret file as byte array */ /* access persistent data... */ /* ... via LOADs ... */ /* ... and STOREs */ /* cf. "sed 's/abc/ABC/g' < input > output" */ Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { if ('a' == * p && 'b' == *(p+1) && 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; *(p+2) = 'C'; } } 19 / 69 /* access persistent data... */ /* ... via LOADs ... */ /* ... and STOREs */ /* cf. "sed 's/abc/ABC/g' < input > output" */ Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; /* interpret file as byte array */ start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { if ('a' == * p && 'b' == *(p+1) && 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; *(p+2) = 'C'; } } 20 / 69 /* ... via LOADs ... */ /* ... and STOREs */ /* cf. "sed 's/abc/ABC/g' < input > output" */ Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; /* interpret file as byte array */ start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { /* access persistent data... */ if ('a' == * p && 'b' == *(p+1) && 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; *(p+2) = 'C'; } } 21 / 69 /* ... and STOREs */ /* cf. "sed 's/abc/ABC/g' < input > output" */ Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; /* interpret file as byte array */ start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { /* access persistent data... */ if ('a' == * p && 'b' == *(p+1) && /* ... via LOADs ... */ 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; *(p+2) = 'C'; } } 22 / 69 /* cf. "sed 's/abc/ABC/g' < input > output" */ Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; /* interpret file as byte array */ start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { /* access persistent data... */ if ('a' == * p && 'b' == *(p+1) && /* ... via LOADs ... */ 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; /* ... and STOREs */ *(p+2) = 'C'; } } 23 / 69 Warm-Up: In-Place \abc" ! \ABC" char *start, *end, *p; /* interpret file as byte array */ start = (char *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); end = start + filesize - 2; for (p = start; p < end; p++) { /* access persistent data... */ if ('a' == * p && 'b' == *(p+1) && /* ... via LOADs ... */ 'c' == *(p+2)) { * p = 'A'; *(p+1) = 'B'; /* ... and STOREs */ *(p+2) = 'C'; } } /* cf. "sed 's/abc/ABC/g' < input > output" */ 24 / 69 /* interpret file contents as application-defined type */ /* access/update persistent data ... */ /* ... via LOADs ... */ /* ... and STOREs, with type checking */ Typed Data Structures struct foo { int bar; double qux; } *foop; foop = (struct foo *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); if (foop->bar) foop->qux *= 2.0; 25 / 69 /* interpret file contents as application-defined type */ /* access/update persistent data ... */ /* ... via LOADs ... */ /* ... and STOREs, with type checking */ Typed Data Structures struct foo { int bar; double qux; } *foop; foop = (struct foo *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); if (foop->bar) foop->qux *= 2.0; 26 / 69 /* access/update persistent data ... */ /* ... via LOADs ... */ /* ... and STOREs, with type checking */ Typed Data Structures struct foo { int bar; double qux; } *foop; /* interpret file contents as application-defined type */ foop = (struct foo *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); if (foop->bar) foop->qux *= 2.0; 27 / 69 /* ... via LOADs ... */ /* ... and STOREs, with type checking */ Typed Data Structures struct foo { int bar; double qux; } *foop; /* interpret file contents as application-defined type */ foop = (struct foo *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); /* access/update persistent data ... */ if (foop->bar) foop->qux *= 2.0; 28 / 69 Typed Data Structures struct foo { int bar; double qux; } *foop; /* interpret file contents as application-defined type */ foop = (struct foo *)mmap(NULL, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor, 0); /* access/update persistent data ... */ if (foop->bar) /* ... via LOADs ... */ foop->qux *= 2.0; /* ... and STOREs, with type checking */ 29 / 69 Persistent Heap, Gordian Knot Edition typedef struct { void *mapaddr, *root, *avail, *end; } *pheap_t; static pheap_t e_pheap; /* persistent heap bookkeeping */ static void *M(size_t s) { /* simple bump-pointer allocator */ void *r = e_pheap->avail; size_t u = sizeof *e_pheap, nu = s / u + (0 == s % u ? 0 : 1); e_pheap->avail = (pheap_t)(e_pheap->avail) + nu; assert(e_pheap->avail