File-system and Block-layer Encryption: Theory, Practice, and Improvement Weigang Li Wenqian Yu Data Center Group 1 2017 Storage DeveloperIntel Conference. Corporation © Intel All Rights Reserved. Agenda Data at-rest encryption File-system encryption Layered: eCryptfs, EFS Native: EXT4, ZFS Full-disk encryption dm-crypt Linux Kernel Crypto API Crypto Hardware Acceleration Benchmark and Optimization Key management – Linux keyring, LUKS Summary 2 2017 Storage Developer Conference. © Intel All Rights Reserved. Data at-rest Encryption Application-level • Flexible to define security strategy, algorithm, key Database management, etc. Encryption • Application dependent Flexibility • File based key management eCryptfs EFS File System • Transparent to application Encryption • Support multiple users / keys EXT4 ZFS Full Disk Encryption dm-crypt • One key for whole disk (volume) (FDE) software • Encrypt everything on disk LUKS • OS-agnostic, Block-layer Software Hardware Transparency Self Encrypting Drive • One Data Encryption Key (DEK) to encrypt the (SED) whole disk, protected by Authentication Key (AK). • Hardware based, secure key is kept in hard-drive 3 2017 Storage Developer Conference. © Intel All Rights Reserved. Data at-rest Encryption Application-level • Flexible to define security strategy, algorithm, key Database management, etc. Encryption • Application dependent Our focus today Flexibility • File based key management eCryptfs EFS File System • Transparent to application Encryption • Support multiple users / keys EXT4 ZFS Full Disk Encryption dm-crypt • One key for whole disk (volume) (FDE) software • Encrypt everything on disk LUKS • OS-agnostic, Block-layer Software Hardware Transparency Self-encrypting Drive • One Data Encryption Key (DEK) to encrypt the (SED) whole disk, protected by Authentication Key (AK). • Hardware based, secure key is kept in hard-drive 4 2017 Storage Developer Conference. © Intel All Rights Reserved. File System Encryption 5 2017 Storage Developer Conference. © Intel All Rights Reserved. File system encryption Application File-system level encryption can be implemented at different level: Fuse-based FS in User space syscalls Fuse Enc/Dec user Layered FS on top of native FS kernel In native FS, better performance Layered File System Enc/Dec Transparent to application. Per-file encryption and key management. Enc/Dec Native Block Layer FEK File File hardware Disk Enc EFEK Master key 6 2017 Storage Developer Conference. © Intel All Rights Reserved. $ mount -t ecryptfs /secret /secret eCryptfs (upper) eCryptfs Select key type to use for newly created files: 1) tspi 2) openssl IVIV 3) passphrase eCryptfs = Enterprise 4) pkcs11-helper (page(page based)based) Selection: 3 Cryptographic Filesystem. Passphrase: xxx Select cipher: xxx R 1) aes: blocksize = 16; min keysize = 16; max keysize = 32 Layered file system. 2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56 N FEK Plaintext 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 G Page In Linux kernel since version 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 Extent 2.6.19. 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 Selection [aes]: 1 Mount eCryptfs on top of a Select key bytes: 1) 16 Per page encryption directory to protect it. 2) 32 3) 24 Selection [16]: 1 Un-mount eCryptfs, the files in Enable plaintext passthrough (y/n) [n]: Enc Enc Enable filename encryption (y/n) [n]: lower FS are encrypted and Attempting to mount with the following options: ecryptfs_unlink_sigs un-readable. ecryptfs_key_bytes=16 ecryptfs_cipher=aes The Encrypted FEK (EFEK) is ecryptfs_sig=aa20c2d38cf280d5 FEKEK Mounted eCryptfs auth-tok stored in the lower FS. EFEK Ciphertext keyring Page KDF key source Native FS (lower) Ref: http://ecryptfs.sourceforge.net/ecryptfs.pdf 7 2017 Storage Developer Conference. © Intel All Rights Reserved. EFS EFS = Encrypting File System Filter driver layered on top of NTFS Source: https://technet.microsoft.com/library/bb457116.aspx#EJAA 8 2017 Storage Developer Conference. © Intel All Rights Reserved. EFS (Cont.) Plaintext Encryption Combination of public key RNG FEK (RSA) and symmetric key encryption (3DES, AES). Ciphertext FEK is protected by user’s pub-key RSA public key. Encryption ENC EFEK is decrypted by user’s EFEK private key to unlock the DEC encrypted file. pri-key RSA Decryption On-disk FEK can be re-encrypted by other’s pub-key to share the encrypted file with other user FEK Decryption w/o re-encrypting the file content. Plaintext 9 2017 Storage Developer Conference. © Intel All Rights Reserved. EXT4 encryption $ mkfs -t ext4 /dev/nvme1n1 RNG IV (page based) $ tune2fs -O encrypt /dev/nvme1n1 $ mount /dev/nvme1n1 /mnt/ext4 Plaintext $ e4crypt add_key /mnt/ext4 Page FEK Enter passphrase (echo disabled): xxx inode Key with descriptor [775a2062517e439c] applied to /mnt/ext4. nonce $ keyctl list @s 2 keys in keyring: setkey 453787240: --alswrv 0 65534 keyring: _uid.0 Enc 1024073693: --alsw-v 0 0 logon: ext4:775a2062517e439c Enc Ciphertext Page In Linux kernel since version 4.1. Master Per-directory Master key. key inode Per-file encryption key (FEK, derived nonce Key-ring from master key and nonce). On-disk Support AES-256-XTS. e4crypt / keyctl (Linux Kernel 4.12) 10 2017 Storage Developer Conference. © Intel All Rights Reserved. ZOL encryption $ zpool create -o ashift=13 testpool /dev/.. $ zfs create -o encryption=aes-128-ccm IV testpool/tank R Enter passphrase for 'tank': xxx N Salt K Plaintext Enter again: xxx G D FEK Block $ zfs get all testpool/tank Master F testpool/tank encryption aes-128-ccm local key IV Enc ZOL = ZFS On Linux Keystore Ciphertext Block http://zfsonlinux.org/ Wrapping key Enc ZOL encryption is not in mainline yet IV PR: https://github.com/zfsonlinux/zfs/pull/5769 KDF Salt Support AES-CCM, AES-GCM MAC passphrase blkptr_t file E(master key) uri Seal master key IV key source MAC On-disk 11 2017 Storage Developer Conference. © Intel All Rights Reserved. Block Layer Encryption 12 2017 Storage Developer Conference. © Intel All Rights Reserved. Block layer encryption Encrypt everything on the disk – one Application key for whole disk (volume). user syscalls Hides file and directory information, kernel such as name and size. File system OS-agnostic. Block layer Master Encrypt Decrypt key volume volume hardware Wrapped Enc Disk key Password 13 2017 Storage Developer Conference. © Intel All Rights Reserved. dm-crypt $cryptsetup luksFormat -c aes-xts-plain64 -s 512 / dev/nvme3n1 $cryptsetup luksFormat -c aes-cbc-essiv:sha256 -s 256 / dev/nvme3n1 Are you sure? (Type uppercase yes): YES IV Enter passphrase: xxx (sector(sector based) based) Plaintext Verify passphrase: xxx $cryptsetup open --type luks /dev/nvme3n1 dm-crypt-disk R Sector Master Enter passphrase for /dev/nvme3n1: xxx N (512B) key $ lsblk G nvme3n1 259:1 0 745.2G 0 disk └─dm-crypt-disk 253:3 0 745.2G 0 crypt $ mkfs -t ext4 /dev/mapper/dm-crypt-disk $ mount /dev/mapper/dm-crypt-disk /dm-crypt-disk/ setkey $ df -T Enc Enc /dev/mapper/dm-crypt-disk ext4 1% /dm-crypt-disk $ umount /dm-crypt-disk $ cryptsetup close dm-crypt-disk passphrase In Linux kernel since version 2.6. key-file Ciphertext Per-sector encryption. Key Slot Sector (512B) Support AES-XTS, AES-CBC. LUKS HDR Single master key for whole disk. Using LUKS for key management - 8 key cryptsetup On-disk slots. 14 2017 Storage Developer Conference. © Intel All Rights Reserved. Linux Kernel Crypto API 15 2017 Storage Developer Conference. © Intel All Rights Reserved. Linux Kernel Crypto API Linux Kernel sub-system $ cat /proc/crypto: Data Transform Symmetric key ciphers: skcipher eCryptfs Ext4 … Asymmetric key ciphers: IPsec dm-crypt akcipher AEAD ciphers API call Callback Message digest Linux Kernel Crypto API Random number generation Submit Callback Compression Memory Crypto driver is registered and Engine Driver selected based on its priority. sg_list Support asynchronous operation for Request Response Page hardware acceleration. DMA Crypto Hardware 16 2017 Storage Developer Conference. © Intel All Rights Reserved. Cryptographic Hardware Acceleration 17 2017 Storage Developer Conference. © Intel All Rights Reserved. Security Benchmarks 110 Intel® QuickAssist Technology 100 90 80 70 60 50 40 30 20 10 0 RSA 2K IPSec SSL Decrypt 1 Forwarding 3 WebProxy1 (kOps/s) (Gbps) (Gbps) Software-based OpenSSL with Intel® QAT Big Data Benchmarks with Compression 2 SW Snappy Compression Intel® QAT Compression 99 Hadoop run time 1. NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. reduced significantly Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2 Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMs Intel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS: PLYDCRB1.86B.0088.D09.1606011736 87 3. Cloudera* 5.4.2 with Snappy* Software vs. Intel® QuickAssist Technology hardware solution. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. Intel® Xeon® processor E5-2699 v4 (56 cores enabled) 256 GB DDR4 1.6 TB NVMe SSD 1 Intel® C6xxx-based card (24x) 10 Gbps CentOS* 6.7 w/ 2.6.32 kernel Cloudera* 5.4.2 QAT driver 0.9.1 Snappy* 1.1.2 (popular, fast compression codec) One NameNode Eight DataNodes 10 Gbps network 2- 24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total 96 GB), 6 Channels,1 x Intel® CorporationRed Rock Canyon 100GbE EthernetSwitch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO. TERASORT TIME IN MINUTES LOWER IS BETTER Intel® QuickAssist Technology integrates hardware acceleration for compute intensive workloads Such as Bulk Cryptography, Public Key Exchange & Compression on Intel® Architecture Platforms 18 2017 Storage Developer Conference. © Intel All Rights Reserved.