Linux Kernel Crypto API  Crypto Hardware Acceleration  Benchmark and Optimization  Key Management – Linux Keyring, LUKS  Summary

File-system and Block-layer Encryption: Theory, Practice, and Improvement

Weigang Li Wenqian Yu

 Data at-rest encryption  File-system encryption  Layered: eCryptfs, EFS  Native: EXT4, ZFS  Full-disk encryption  dm-crypt  Linux Kernel Crypto API  Crypto Hardware Acceleration  Benchmark and Optimization  Key management – Linux keyring, LUKS  Summary

Application-level • Flexible to define security strategy, algorithm, key Database management, etc. Encryption • Application dependent

Flexibility • File based key management eCryptfs EFS File System • Transparent to application Encryption • Support multiple users / keys EXT4 ZFS

Full Disk Encryption dm-crypt • One key for whole disk (volume) (FDE) software • Encrypt everything on disk LUKS • OS-agnostic, Block-layer Software

Hardware

Transparency Self Encrypting Drive • One Data Encryption Key (DEK) to encrypt the (SED) whole disk, protected by Authentication Key (AK). • Hardware based, secure key is kept in hard-drive

Application-level • Flexible to define security strategy, algorithm, key Database management, etc. Encryption • Application dependent Our focus today

Flexibility • File based key management eCryptfs EFS File System • Transparent to application Encryption • Support multiple users / keys EXT4 ZFS

Full Disk Encryption dm-crypt • One key for whole disk (volume) (FDE) software • Encrypt everything on disk LUKS • OS-agnostic, Block-layer Software

Hardware

Transparency Self-encrypting Drive • One Data Encryption Key (DEK) to encrypt the (SED) whole disk, protected by Authentication Key (AK). • Hardware based, secure key is kept in hard-drive

Application  File-system level encryption can be implemented at different level:  Fuse-based FS in User space syscalls Fuse Enc/Dec user  Layered FS on top of native FS kernel  In native FS, better performance Layered File System  Enc/Dec Transparent to application.  Per-file encryption and key management.

Enc/Dec Native

Block Layer FEK File File

hardware

Disk Enc EFEK Master key

6 2017 Storage Developer Conference. © Intel All Rights Reserved. $ mount -t ecryptfs /secret /secret eCryptfs (upper) eCryptfs Select key type to use for newly created files: 1) tspi 2) openssl IVIV 3) passphrase  eCryptfs = Enterprise 4) pkcs11-helper (page(page based)based) Selection: 3 Cryptographic Filesystem. Passphrase: xxx Select cipher: xxx R  1) aes: blocksize = 16; min keysize = 16; max keysize = 32 Layered file system. 2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56 N FEK Plaintext 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 G Page  In Linux kernel since version 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 Extent 2.6.19. 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 Selection [aes]: 1  Mount eCryptfs on top of a Select key bytes: 1) 16 Per page encryption directory to protect it. 2) 32 3) 24  Selection [16]: 1 Un-mount eCryptfs, the files in Enable plaintext passthrough (y/n) [n]: Enc Enc Enable filename encryption (y/n) [n]: lower FS are encrypted and Attempting to mount with the following options: ecryptfs_unlink_sigs un-readable. ecryptfs_key_bytes=16 ecryptfs_cipher=aes  The Encrypted FEK (EFEK) is ecryptfs_sig=aa20c2d38cf280d5 FEKEK Mounted eCryptfs auth-tok stored in the lower FS. EFEK Ciphertext keyring Page

KDF key source Native FS (lower)

 EFS = Encrypting File System  Filter driver layered on top of NTFS

Source: https://technet.microsoft.com/library/bb457116.aspx#EJAA

Encryption  Combination of public key RNG FEK (RSA) and symmetric key encryption (3DES, AES). Ciphertext  FEK is protected by user’s pub-key RSA public key. Encryption ENC  EFEK is decrypted by user’s EFEK private key to unlock the DEC encrypted file. pri-key RSA Decryption On-disk  FEK can be re-encrypted by other’s pub-key to share the encrypted file with other user FEK Decryption w/o re-encrypting the file content. Plaintext

$ mkfs -t ext4 /dev/nvme1n1 RNG IV (page based) $ tune2fs -O encrypt /dev/nvme1n1 $ mount /dev/nvme1n1 /mnt/ext4 Plaintext $ e4crypt add_key /mnt/ext4 Page FEK Enter passphrase (echo disabled): xxx inode Key with descriptor [775a2062517e439c] applied to /mnt/ext4. nonce $ keyctl list @s 2 keys in keyring: setkey 453787240: --alswrv 0 65534 keyring: _uid.0 Enc 1024073693: --alsw-v 0 0 logon: ext4:775a2062517e439c Enc Ciphertext Page  In Linux kernel since version 4.1.  Master Per-directory Master key. key inode  Per-file encryption key (FEK, derived nonce Key-ring from master key and nonce). On-disk  Support AES-256-XTS.

e4crypt / keyctl (Linux Kernel 4.12)

IV testpool/tank R Enter passphrase for 'tank': xxx N Salt K Plaintext Enter again: xxx G D FEK Block $ zfs get all testpool/tank Master F testpool/tank encryption aes-128-ccm local key

 Enc ZOL = ZFS On Linux Keystore Ciphertext  Block http://zfsonlinux.org/ Wrapping  key Enc ZOL encryption is not in mainline yet IV  PR: https://github.com/zfsonlinux/zfs/pull/5769 KDF Salt  Support AES-CCM, AES-GCM MAC passphrase blkptr_t file E(master key) uri Seal master key IV

 Encrypt everything on the disk – one Application key for whole disk (volume). user syscalls  Hides file and directory information, kernel such as name and size. File system  OS-agnostic.

Block layer Master Encrypt Decrypt key volume volume

hardware Wrapped Enc Disk key Password

13 2017 Storage Developer Conference. © Intel All Rights Reserved. dm-crypt $cryptsetup luksFormat -c aes-xts-plain64 -s 512 / dev/nvme3n1 $cryptsetup luksFormat -c aes-cbc-essiv:sha256 -s 256 / dev/nvme3n1 Are you sure? (Type uppercase yes): YES IV Enter passphrase: xxx (sector(sector based) based) Plaintext Verify passphrase: xxx $cryptsetup open --type luks /dev/nvme3n1 dm-crypt-disk R Sector Master Enter passphrase for /dev/nvme3n1: xxx N (512B) key $ lsblk G nvme3n1 259:1 0 745.2G 0 disk └─dm-crypt-disk 253:3 0 745.2G 0 crypt $ mkfs -t ext4 /dev/mapper/dm-crypt-disk $ mount /dev/mapper/dm-crypt-disk /dm-crypt-disk/ setkey $ df -T Enc Enc /dev/mapper/dm-crypt-disk ext4 1% /dm-crypt-disk $ umount /dm-crypt-disk $ cryptsetup close dm-crypt-disk

passphrase  In Linux kernel since version 2.6. key-file Ciphertext  Per-sector encryption. Key Slot Sector (512B)  Support AES-XTS, AES-CBC. LUKS HDR  Single master key for whole disk.  Using LUKS for key management - 8 key cryptsetup On-disk slots.

Linux Kernel sub-system $ cat /proc/crypto:  Data Transform  Symmetric key ciphers: skcipher eCryptfs Ext4 …  Asymmetric key ciphers: IPsec dm-crypt akcipher  AEAD ciphers API call Callback  Message digest Linux Kernel Crypto API  Random number generation Submit Callback  Compression Memory  Crypto driver is registered and Engine Driver selected based on its priority. sg_list  Support asynchronous operation for Request Response Page hardware acceleration.

DMA

Crypto Hardware

0 RSA 2K IPSec SSL Decrypt 1 Forwarding 3 WebProxy1 (kOps/s) (Gbps) (Gbps)

Software-based OpenSSL with Intel® QAT

Big Data Benchmarks with Compression 2 SW Snappy Compression Intel® QAT Compression 99

Hadoop run time 1. NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. reduced significantly Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2 Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMs Intel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS:

PLYDCRB1.86B.0088.D09.1606011736 87 3. Cloudera* 5.4.2 with Snappy* Software vs. Intel® QuickAssist Technology hardware solution. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. Intel® Xeon® processor E5-2699 v4 (56 cores enabled) 256 GB DDR4 1.6 TB NVMe SSD 1 Intel® C6xxx-based card (24x) 10 Gbps CentOS* 6.7 w/ 2.6.32 kernel Cloudera* 5.4.2 QAT driver 0.9.1 Snappy* 1.1.2 (popular, fast compression codec) One NameNode Eight DataNodes 10 Gbps network 2- 24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total 96 GB), 6 Channels,1 x Intel® CorporationRed Rock Canyon 100GbE EthernetSwitch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO. TERASORT TIME IN MINUTES LOWER IS BETTER Intel® QuickAssist Technology integrates hardware acceleration for compute intensive workloads Such as Bulk Cryptography, Public Key Exchange & Compression on Intel® Architecture Platforms

Without Intel® KPT With Intel® KPT

DRAM DRAM

z LBG KPT Intel® C627 Chipset Intel® with Intel® QAT Intel® QuickAssist Unwrapping (Decrypt) Technology PTT Secure Channel

• Private key exposes in clear text • Private key is wrapped (encrypted) • Key is not protected and unsafe • Key is protected and safe • Subject to attacks • NOT subject to attacks

Private Key (PK) Symmetric Wrapping Key (SWK) Wrapped Private Key (WPK)

Computing Wait Wait IO  Synchronous: CPU  For high performance “single Offload Complete Offload Complete stream” accelerator Hardware  Simple program model Engine sync mode  Asynchronous:  For multi-engines hardware Computing Swap IO  High throughput CPU Offload Complete  Complex control flow Hardware

Engine

Engine async mode

Extent Callback  Synchronous call causes

Sequential Page n wakeup performance drop!

Page 2 • add the encrypted page to bio_vec Throughput(MB/s) Page 1 • encrypt the next page 1000 800 600 all pages are encrypted Linux Kernel Crypto API HW is not 400 fully utilized! 200 Submit BIO 0 Extent off s/w encryption qat-sync Crypto Page vec encryption Engine vec dd if=/dev/zero of=/mnt/ext4/testfile bs=1M count=10000 oflag=sync Page vec Intel® microprocessor. Processor: Intel®Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory bio_vec configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700 NVMe. Intel® QuickAssistTechnology DH8950, in-tree driver in kernel 4.12 Page

Throughput(MB/s) CPU utilization @88 HT cores (%) 5000 45 40 4000 35 30 3000 CPU utilization reduced 25 2000 20 15 1000 10 5 0 0 1 x nvme 2 x nvme 3 x nvme 1 x nvme 2 x nvme 3 x nvme off s/w qat off s/w qat

Intel® microprocessor. Processor: Intel®Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory  configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 3X P3700 Block size (128KB) is “big enough” for one offloading. NVMe. Intel® QuickAssist Technology driver: QAT1.6 v2.6. Cent OS* 7.2 (Kernel 3.10) ZOL 0.7.0 with PR (https://github.com/zfsonlinux/zfs/pull/5769). FIO-2.1.2, sequential write, 16 threads per NVMe disk.  ZIO pipeline drives compression, checksums, data ZOL encryption algorithm: AES-CCM-128 redundancy, and encryption, etc., with a pool of threads (taskq) to improve performance.  Hardware accelerator is fully utilized.

Intel® QAT (DH8950) offloading cost  vs. Software cost (Intel® Xeon® E5-2699 v4 ) Offloading cost: 18000  Create request descriptor 16000  Interrupt handling, response 14000 12000 polling 10000  Hardware* offloading cost is 8000 almost consistent: ~1600-1800 6000 Cost (CPU Cycles) 4000 cycles. 2000  Choose right buffer size to 0 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 get best performance with Buffer Size(Bytes) reasonable offloading cost.

qat_aes_xts xts-aes-aesni  Software cost varies by buffer

Intel® microprocessor. Processor: Intel®Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory size. configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. * Intel® QuickAssist Technology DH8950, in-tree kernel driver. Benchmark code calls Kernel Crypto API (skcipher), compute average CPU time with 1000x API calls. Cycle count as measured at Kernel Crypto (skcipher) API for AES-XTS algorithm.

dm-crypt PagePage (4KB) (4KB)  dm-crypt calls Kernel Crypto API for data (for each page) Sector Sector encryption, it can integrate with a 512B 512B iv iv hardware accelerator seamlessly. When 8 sectors all done  Sector asynchronous call: sending 8x requests 512B for one 4KB page in-parallel, no wait. iv  But: offloading cost is high - API call x8 Callback x8  Encrypting a 4KB page  8x encryption requests. PagePage (4KB)(4KB)  Encrypting 1MB data  2048x Kernel Crypto API Sector Sector encryption requests. 512B 512B Driver Sector Submit 512B request x8 Callback x8

Accelerator Submit BIO

24 2017 Storage Developer Conference. © Intel All Rights Reserved. dm-crypt optimization  Encrypt 8 sectors (sg_list) in a single call.  Offloading cost is reduced greatly.  Kernel Crypto API or accelerator hardware to PagePage (4KB) (4KB) dm-crypt compute IV for each sector. (for each page) Sector 512B Sector iv 512B Throughput (MB/s) iv 900 Sector 512B sg_list 800 iv 700 600 API call x1 Callback x1 500 400 300 PagePage (4KB)(4KB) 200 Kernel Crypto API 100 Sector Sector Driver 512B 512B 0 w/o optimization with optimization

Submit Sector dd if=/dev/zero of=/dev/mapper/dm-c bs=1M count=10000 oflag=sync 512B request x1 Callback x1 Note: experimental result, patch is not in upstream kernel yet. Intel® microprocessor. Processor: Intel®Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700 NVMe. Intel® QuickAssistTechnology DH8950, in-tree driver in kernel 4.12 Accelerator Submit BIO

keyctl  In-kernel key management and user retention facility: kernel http://man7.org/linux/man- pages/man7/keyrings.7.html keyring

request key

setkey Kernel Linux Kernel Crypto API sub-system

27 2017 Storage Developer Conference. © Intel All Rights Reserved. Key management - LUKS # cryptsetup luksDump /dev/nvme3n1 LUKS header information for /dev/nvme3n1 Version: 1 Cipher name: aes Disk Format Cipher mode: xts-plain64 Master Hash spec: sha1 Payload offset: 4096 key MK bits: 512 slot#0 MK digest: ed 36 e3 09 00 72 ea ba 24 04 6e 11 7e 69 1a 87 5a d1 32 57 MK salt: bc 41 51 8e e0 90 65 20 70 6d ef 2f 3f 80 06 a7… MK iterations: 48000 slot#2 UUID: 0f6c5194-b299-4f3e-9f54-17dc67a62f04 Key Slot 0: ENABLED Iterations: 192480 Salt: 62 4b 4d 54 b1 d5 0c 44 c2 a9 a0 d3 03 36 59 78… Key material offset: 8 AF stripes: 4000 slot#7 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Encryption Key Slot 5: DISABLED Encrypted Key Slot 6: DISABLED Data Key Slot 7: DISABLED  Linux Unified Key Setup, it is a specification (on-disk format) for disk encryption. Password  Implemented in cryptsetup utility + dm-crypt (kernel module) for disk encryption.  Master key to encrypt / decrypt data. Plaintext  Password to encrypt / unlock the master key. Data  Support 8 key slots.

 If data encryption / decryption is done in Enc / Dec host CPU, keys (FEK, master key, Library private key…) have to be exposed in memory  Cold boot (memory dump) attack CPU  Heartbleed-like attack Memory  Possible solution  SGX Enclave  Total Memory Encryption (TME)  Debug-register-based or cache- based key storage

Memory  HSM = Hardware Security Module Crypto Library  KEK = Key Encryption Key Wrapped-key  Session key (e.g., FEK) is generated by HSM and wrapped by KEK, the wrapped-key can CPU send back to host memory. Buffer  The wrapped-key is a “key handler” to request HSM service.

 Key storage and data encryption / decryption Crypto Enc / Dec are all kept inside device. Hardware Engine  Clear key material never leaves device. Key hierarchy tree Key Gen KEK Key provision Root Key HSM

30 2017 Storage Developer Conference. © Intel All Rights Reserved. Summary  File-system and full-disk encryption are important software technologies to implement data-at-rest encryption solution.  Data encryption at different layer has its own pros and cons.  Data encryption software shall consider how to best utilize the hardware crypto accelerator underneath in order to get best performance, e.g., buffer size, sync vs. async, offloading cost, etc.  Besides data encryption, secure-key protection is even more important for persistent data storage.  Hardware based key management is a possible solution to protect key leakage.

 http://ecryptfs.sourceforge.net/ecryptfs.pdf  https://zfs.datto.com/slides/caputi.pdf  https://technet.microsoft.com/library/bb457116.aspx#EJAA  http://kernsec.org/files/lss2014/Halcrow_EXT4_Encryption.pdf  www.intel.com/content/www/us/en/embedded/technology/quickassist/overview.html