ext4 online defragmentation Takashi Sato NEC Software Tohoku, Ltd. [email protected] Abstract In this paper, an online defragmentation feature for ext4 is proposed, which can solve the fragmentation problem ext4 greatly extends the filesystem size to 1024PB com- on a mounted filesystem. pared to 16TB in ext3, and it is capable of storing many huge files. Previous study has shown that fragmenta- Section 2 describes various features in current Linux tion can cause performance degradation for such large filesystems to reduce fragmentation, and shows how filesystems, and it is important to allocate blocks as con- much fragmentation occurs when multiple processes tiguous as possible to improve I/O performance. writes simultaneously to disk. Section 3 explains the de- sign of the proposed online defragmentation method and In this paper, the design and implementation of an online Section 4 describes its implementation. Section 5 shows defragmentation extension for ext4 is proposed. It is de- the performance benchmark results. Existing work on signed to solve three types of fragmentation, namely sin- defragmentation is shown in Section 6. Section 7 con- gle file fragmentation, relevant file fragmentation, and tains the summary and future work. free space fragmentation. This paper reports its design, implementation, and performance measurement results. 2 Fragmentations in Filesystem 1 Introduction Filesystem on Linux have the following features to re- duce occurrence of fragments when writing a file. When a filesystem has been used for a long time, disk blocks used to store file data are separated into discon- • tiguous areas (fragments). This can be caused by con- Delayed allocation current writes from multiple processes or by the kernel The decision of where to allocate the block on the not being able to find a contiguous area of free blocks disk is delayed until just before when the data is large enough to store the data. stored to disk in order to allocate blocks as con- tiguously as possible (Figure 1). This feature is al- If a file is broken up into many fragments, file read per- ready implemented in XFS and is under discussion formance can suffer greatly, due to the large number of for ext4. disk seeks required to read the data. Ard Biesheuvel has shown the effect of fragmentation on file system perfor- • Block reservation mance [1]. In his paper, file read performance decreases Contiguous blocks are reserved right after the last as the amount of free disk space becomes small because allocated block, in order to use them for successive of fragmentation. This occurs regardless of the filesys- block allocation as shown in Figure 2. This feature tem type being used. is already implemented in ext4. ext3 is currently used as the standard filesystem in Linux, and ext4 is under development within the Linux Although ext4 and XFS already have these features community as the next generation filesystem. ext4 implemented to reduce fragmentation, writing multiple greatly extends the filesystem size to 1024PB compared files in parallel still causes fragmentation and decrease to 16TB in ext3, and it is capable of storing many huge in file read performance. Figure 3 shows the differ- files. Therefore it is important to allocate blocks as con- ence in file read performance between the following two tiguous as possible to improve the I/O performance. cases: • 179 • 180 • ext4 online defragmentation Seconds 1GB file read time(real time) Data 16.5% 22 14.8% degrade 21 degrade write EXT3 XFS 20 19 18 17 16 13 252 2 288 Fragments in 1GB file The environment of the measurement: kernel: 2.6.18-rc4 with Mingming's ext4 patch set CPU: Xeon 3.0GHx This block is freed before the data is Memory: 1GB written to disk, we can write it on the Arch: i386 contiguous blocks Figure 3: The influence of fragments Figure 1: Delayed allocation • Single file fragmentation Data i. Pick up the Single file fragmentation occurs when a single file reserved block is broken into multiple pieces. This decreases the write performance of accessing a single file. • Relevant file fragmentation reserved blocks Relevant file fragmentation occurs when relevant ii. Write to disk files, which are often accessed together by ap- plications are allocated separately on the filesys- tem. This decreases the performance of applica- tions which access many small files. Figure 2: Block reservation • Free space fragmentation Free space fragmentation occurs when the filesys- tem has many small free areas and there is no large • Create the file by writing 32 1GB files sequentially free area consisting of contiguous blocks. This will make the other two types of fragmentation more • Create the file by writing 32 1GB files from 32 likely to occur. threads concurrently Online defragmentation should be able to solve these File read performance decreases about 15% for ext3, three types of fragmentation. and 16.5% for XFS. Currently ext4 does not have any defragmentation fea- 3.2 Single File Fragmentation ture, so fragmentation will not be resolved until the file is removed. Online defragmentation for ext4 is neces- Single file fragmentation could be solved by moving file sary to solve this problem. data to contiguous free blocks as shown in Figure 4. Defragmentation for a single file is done in the following 3 Design of Online Defragmentation steps (Figure 5). 3.1 Types of Fragmentation 1. Create a temporary inode. Fragmentation could be classified into the following 2. Allocate contiguous blocks to temporary file as in three types [2]. Figure 5(a). 2007 Linux Symposium, Volume Two • 181 File ioctl User Kernel Create temporary Move the data to Target Temporary inode and allocate inode inode contiguous blocks contigurous blocks Disk Contiguous free blocks Contiguous blocks (a) Allocate contiguous blocks ioctl Figure 4: Defragment for a single file User Kernel Target page cache Temporary 3. Move file data for each page as described in Figure inode write to inode 5(b). The following sequence of events should be block read from committed to the journal atomically. block (a) Read data from original file to memory page. (b) Swap the blocks between the original inode Contiguous blocks and the temporary inode. (b) Replace data blocks (c) Write data in memory page to new block. Figure 5: Resolving single file fragmentation 3.3 Relevant File Fragmentation Block No = 100 Block No = 100 directory directory Relevant file fragmentation could be solved by moving the files under the specified directory close together with File1 File2 File1 File2 the block containing the directory data as presented in Block No Block No Figure 6. 60000 70000 101 102 Defragmentation for relevant files is done in the follow- Figure 6: Defragment for the relevant files ing steps (Figure 7). 1. The defragmentation program asks the kernel for 5. Repeat steps 2 to 4 for all files under the specified the physical block number of the first block of the directory. specified directory. The kernel retrieves the physi- cal block number of the extent which is pointed by 3.4 Free Space Fragmentation the inode of the directory and returns that value as in Figure 7(a). If the filesystem has insufficient contiguous free blocks, the other files are moved to make sufficient space to allo- 2. For each file under the directory, create a temporary cate contiguous blocks for the target file. Free space de- inode. fragmentation is done in the following steps (Figure 8). 3. The defragmentation command passes the physical block number obtained in step 1 to the kernel. The 1. Find out the block group number to which the tar- kernel searches for the nearest free block after the get file belongs. This could be calculated by using given block number, and allocates that block to the the number of inodes in a group and the inode num- ber as below. temporary inode as described in Figure 7(b). inodeNumber groupNumber = 4. Move file data for each page, by using the same iNodesPerGroup method as for single file defragmentation. 182 • ext4 online defragmentation Specify "0" Return "104" as as the file the physical relative block block number number ioctl() ioctl extent1 extent0 User start=108 start=104 4blocks User 4blocks Kernel inode Kernel extent0 extent1 inode (inode No=201) start = 104 start = 204 5blocks 5blocks exten0 exten1 (a) Get block mapping start=104 start=108 4blos 4blocs ioctl ( goal = 100 ) User (a) Get extens Kernel extent ioctl() original temporary start=104 4blocks inode inode User Allocate a block near goal Kernel Block No = 60000 Block No = 101 inode Register blocks to avoid being used by other files. (b) Specify goal as allocation hint extent start=104 4blocks Figure 7: Resolving relevant files fragmentation reservation window (b) Reserve blocks 2. Get the extent information of all the files belonging ioctl to the target group and choose the combination of User Kernel Search free blocks files to evict for allocating contiguous blocks large Target Temporary from the farthest inode inode group from the enough to hold the target file as in Figure 8(a). target inode. 3. Reserve the chosen blocks so that it will not be used group0 group1 group2 groupN by other processes. This is achieved by the kernel registering the specified blocks to the reservation (c) Move victim file window for each file as presented in Figure 8(b). extent ioctl start=104 14blocks User Kernel Temporary Target inode 4. Move the data in the files chosen in step 2 to other inode block groups. The destination block group could extent start=104 be either specified by the defragmentation com- 14blocks mand or use the block group which is farthest away from the target group as in Figure 8(c).