Optimizing eCryptfs for better performance and security Li Wang Yunchuan Wen School of Computer Kylin, Inc. National University of Defense Technology [email protected] [email protected] Jinzhu Kong Xiaodong Yi School of Computer School of Computer National University of Defense Technology National University of Defense Technology [email protected] [email protected] Abstract Cryptfs [5], which is part of the FiST framework [6]. eCryptfs provides transparent per file encryption. Af- This paper describes the improvements we have done to ter mounting a local folder as an eCryptfs folder eCryptfs, a POSIX-compliant enterprise-class stacked with identity authentication passed, the file copied into cryptographic filesystem for Linux. The major improve- the eCryptfs folder will be automatically encrypted. ments are as follows. First, for stacked filesystems, by eCryptfs is widely used, for example, as the basis default, the Linux VFS framework will maintain page for Ubuntu’s encrypted home directory, natively within caches for each level of filesystem in the stack, which Google’s ChromeOS, and transparently embedded in means that the same part of file data will be cached several network attached storage (NAS) devices. multiple times. However, in some situations, multiple eCryptfs is implemented as a stacked filesystem inside caching is not needed and wasteful, which motivates us Linux kernel, it does not write directly into a block de- to perform redundant cache elimination, to reduce ide- vice. Instead, it mounts on top of a directory in a lower ally half of the memory consumption and to avoid un- filesystem. Most POSIX compliant filesystem can act as necessary memory copies between page caches. The a lower filesystem, for example, ext4 [2], XFS [4], even benefits are verified by experiments, and this approach NFS [3]. eCryptfs stores cryptographic metadata in the is applicable to other stacked filesystems. Second, as header of each file written, so that encrypted files can be a filesystem highlighting security, we equip eCryptfs copied between hosts, and no additional information is with HMAC verification, which enables eCryptfs to de- needed to decrypt a file, except the ones in the encrypted tect unauthorized data modification and unexpected data file itself. corruption, and the experiments demonstrate that the de- crease in throughput is modest. Furthermore, two minor The rest of this paper is organized as follows. For back- optimizations are introduced. One is that we introduce ground information, Section1 introduces the eCryptfs a thread pool, working in a pipeline manner to perform cryptographic filesystem. Section2 describes the opti- encryption and write down, to fully exploit parallelism, mizations for eCryptfs performance. Section3 presents with notable performance improvements. The other is the data integrity enforcement for eCryptfs security. a simple but useful and effective write optimization. In Section4 discusses our ongoing and future works on addition, we discuss the ongoing and future works on eCryptfs. Section5 concludes the paper. eCryptfs. 2 Performance Improvements 1 Introduction 2.1 Redundant Cache Elimination eCryptfs is a POSIX-compliant enterprise cryptographic filesystem for Linux, included in the mainline Linux The page cache is a transparent filesystem cache imple- kernel since version 2.6.19. eCryptfs is derived from mented in Linux VFS (Virtual File System) framework. • 137 • 138 • Optimizing eCryptfs for better performance and security The fundamental functionality is to manage the memory pages that buffer file data, avoiding frequent slow disk accesses. For eCryptfs (and other stacked file systems in Linux), there exist (at least) two levels of page caches, as shown in Figure1. The upper level is the eCryptfs page cache, which buffers the plain text data to interact with the user applications. The lower level page cache belongs to the file system on top of which eCryptfs is stacked, and it buffers the cipher text data as well as the eCryptfs file metadata. The eCryptfs file read/write Figure 1: Page caches for eCryptfs on top of ext4 under works as follows, original implementation. • Read operations result in the call of VFS vfs_ read, which searches the eCryptfs page cache. If no matching page is found, vfs_read calls the file system specific readpage call back routine to bring the data in. For eCryptfs this is eCryptfs_ readpage, which calls vfs_read again to cause the lower file system to read the data from the disk into eCryptfs page cache. The data is then de- crypted and copied back to the user application. Figure 2: Page caches for eCryptfs on top of ext4 under • Write operations result in the call of VFS vfs_ read optimization for encrypted file. write, which copies the data from user space buffer into the eCryptfs page cache, marks the cor- responding page dirty, then returns without en- If the eCryptfs file is opened only for reading, the lower cryption (unless the system currently has large page cache is not needed since the cipher data will not amount of dirty pages). Encryption is normally be used after decryption. This motivates us to perform performed asynchronously by the dedicated OS redundant cache elimination optimization, thus reduce kernel thread, during the job of flushing dirty ideally half of the memory consumption. The page pages into lower page cache, by invoking file sys- cache is maintained only at the eCryptfs level. For first tem specific writepage call back routine, here is read, once the data has been read out from disk, de- ecryptfs_writepage. This routine encrypts a crypted and copied up to eCryptfs page cache, we free whole page of data into a temporary page, then in- the corresponding pages in the lower page cache im- vokes vfs_write to copy the encrypted data from mediately by invoking invalidate_inode_pages2_ the temporary page into the lower page cache. range, as shown in Figure2. A better solution is to modify the VFS framework to avoid allocating the lower In real life, eCryptfs is often deployed in archive and page entirely, but that would be much more compli- backup situations. For the former case, people archive cated, and we want to limit our revisions in the scope their private documents into an eCryptfs protected direc- of eCryptfs codes. tory, consequently, the corresponding eCryptfs files are created and opened for writing, and those files are later If the eCryptfs file is opened for writing, and the write opened generally for reading. The latter case is simi- position is monotonic increasing, which guarantees the lar, user copies files out from eCryptfs folder, modifies, same data area will not be repeatedly written, then the and copies the revised files back to replace the original eCryptfs page cache is not needed since the plain data ones. In this case, the eCryptfs files are opened for either will not be used after encryption. It is beneficial to main- reading or writing as well, in other words, in the above tain the page cache only at the lower level. Once the situations, the files are almost never online edited, i.e., data has been encrypted and copied down to lower page opened for both reading and writing. This is also true cache, the corresponding pages in eCryptfs page cache for some other situations. are freed immediately. 2012 Linux Symposium • 139 Figure 3: The data structures for redundant cache elim- ination for encrypted file. Figure 5: The readpage routine for eCryptfs for en- crypted file. writer gets involved. In that case, the state changes from ECRYPTFS_RW_RDOPT to ECRYPTFS_RW_NOOPT. If the file is initially opened for writing, the state becomes ECRYPTFS_RW_WROPT, and remains unchanged until the write position decreases or any reader gets involved, in which case, the state becomes ECRYPTFS_RW_NOOPT. Figure 4: The state machine for redundant cache elimi- After the file is closed by the last user, the state returns nation for encrypted file. to ECRYPTFS_RW_INIT. Figure5 shows the eCryptfs code implementing To achieve this optimization, we maintain a simple state readpage. After the data have been copied into machine as an indicator. There are four states as shown eCryptfs page cache, and redundant cache elimination in Figure3, ECRYPTFS_RW_INIT indicates an initial- is applicable, in line 13, the corresponding lower page ized state, with neither readers nor writers. ECRYPTFS_ index is calculated, then in line 15, the lower page is RW_RDOPT indicates the file is currently opened only for released. reading, and the read optimization, i.e, lower page cache early release applies. ECRYPTFS_RW_WROPT indicates We evaluate redundant cache elimination on a machine the file is currently opened only for writing and the write with a Pentium Dual-Core E5500 2.8GHz CPU, 4G of position is monotonic increasing, in which case, the memory (DDR3), the kernel version 3.1.0-7 i686 with write optimization, i.e, upper page cache early release PAE enabled, the same test system is used for all the ex- applies. ECRYPTFS_RW_NOOPT applies for remaining periments of this paper described. We use the command cases. As shown in Figure4, when a file data struc- ‘iozone -t x -s y -r 4M -i 1’ to measure ‘Re-read’ ture is initialized, the state is set to ECRYPTFS_RW_INIT. throughput. Here x ranges from 1 to 256, increasing by If it is then opened for reading, the state transitions to orders of 2, representing the number of processes, and ECRYPTFS_RW_RDOPT, and remains unchanged until a y ranges from 2G to 8M, representing the file size, such 140 • Optimizing eCryptfs for better performance and security Figure 6: Re-read throughput for redundant cache elim- ination over original implementation. that the total operated data set x ∗ y is always equal to 2G. As shown in Figure6, the optimized eCryptfs achieves a big speedup under this group of tests. This is because the data set is 2G, if two level of page caches are present, it will consume up to 4G memory, thus lead to a poor re-read throughput due to memory page thrashing.