Deep Compression
Total Page:16
File Type:pdf, Size:1020Kb
COVER STORY Cloop DEEPBlock device compression COMPRESSION with the cloop module KYRO, photocase.com KYRO, The cloop module lets you manage compression at the block device 512 bytes), and they are usually used for random access storage like ramdisks, level. Read on to learn how Knoppix and other Live CDs fit all that CD-ROMs, floppy disks, hard disks, and hard disk partitions. software on a single disc. BY KLAUS KNOPPER Filesystems are a logical representa- tion of ordered data that is often present loop is a kernel block device block-based devices. If you look into the on a block device. A filesystem turns raw module used in Live CDs such output of ls -l /dev, you will easily recog- data into the familiar directory/file view. Cas Knoppix. The cloop module nize these devices by the prefix – c for The mount command is the bridge be- allows the system to read compressed character-based and b for block-based tween a block device partition and its data, usually from a file, thus creating devices – at the beginning of the output projection into a mount point directory. compressed virtual disks. Using cloop, line (see Listing 1). a Linux installation of about 2GB fits on Character-based devices, such as tape Cloop: A Compressed a single 700MB CD-R disc. In this article, drives, mice, and gamepads, provide se- Loopback Block Device I look at how cloop works and provide quential, character-by-character access One block device included in any Linux some insight into general kernel struc- to data. kernel is loop, which maps a plain file to tures of a block device. Block devices instead allow direct ac- a block device node in /dev/loop A Unix system traditionally distin- cess to arbitrary blocks of data, indexed [number]. The loop-attached file can guishes between character-based and by block number or sector (segments of then be mounted like an ordinary hard 32 ISSUE 86 JANUARY 2008 032-036_cloop.indd 32 15.11.2007 10:17:18 Uhr Cloop COVER STORY been successfully port write operations would probably be Plain file read and decom- very inefficient because, at each write or pressed, and it operation, the compressed block struc- Disk Image never terminates ture would have to be reordered, and or interrupts ac- blocks on a block device can never be cess to the under- “deleted,” since the block device has cloop lying device. That no information about whether or not modprobe cloop file=/path/to/file.cloop way, even partly data is actually needed. damaged data can Also, when compressing a block, the still be restored, resulting block size is not really predict- Filesystem/Mount and crashes be- able. In fact, a compressed block can be cause of read er- slightly larger than an uncompressed mount −r −t iso9660 /dev/cloop /mnt rors are unlikely block, in case that the data is already (though, of stored compressed. When exchanging Mountpoint with visible course, an execut- compressed blocks, the block index table directories and files able program that would have to be recalculated (all offsets / has been damaged after the changed block will also /usr /home will most likely change). Therefore, cloop is very opti- refuse to run and mized for reading data, but writing is terminate with a not currently supported and probably Figure 1: (C)loop maps a hard disk image file to a block device and lets segmentation never will be. you mount the image like a real hard disk partition. fault). This feature If you need write support on read-only may give cloop media, you are better off with a filesys- disk, optionally with encryption. How- a slight advantage against compressing tem such as AuFS or Unionfs that merges ever, this driver still only provides a filesystems, in which an entire file be- the read-only directory with a directory translation between different data repre- comes unreadable if there is a read error offering write support. sentations of the same size. right at the beginning of that file. In Cloop can handle parallel accesses to Cloop, which was first written in 1999 cloop, at maximum, one block is missing multiple cloop image files. If you have for kernel 2.2 by iptables author Paul when read errors occur within this to split files because of size limits in the ‘Rusty’ Russell, uses a blockwise com- block. underlying filesystem, this is important pressed input file that reduces the neces- One disadvantage is that cloop is read- (i.e., the 2GB and 4GB maximum file sary space to around 1/3 of the original only. Attempting to rewrite cloop to sup- size in FAT32 and iso9660, respectively). size for executable files. I ported, and later rewrote, cloop for newer kernel ver- Listing 1: Character and Block Device Nodes sions, extending the module to support 01 crw-rw---- 1 root root 10, 1 2007-10-18 10:41 /dev/psaux 64-bit file access (thus allowing file sizes 02 brw-rw---- 1 root disk 3, 0 2007-10-18 10:41 /dev/hda beyond 4GB), multiple input files, and ioctl extensions to permit the exchange 03 brw-rw---- 1 root disk 240, 0 2007-11-01 21:38 /dev/cloop of input files on-the-fly. Since cloop is a block device and not a Listing 2: Attaching a New Cloop Image filesystem like squashfs, the compressed 01 $ sudo losetup /dev/cloop1 /media/cdrom/backup.cloop image can be anything, from raw data to arbitrary filesystem or database struc- 02 $ sudo mount -r -t ext2 /dev/cloop1 /mnt tures. On a Live CD, the cloop image 03 $ dmesg | tail -2 usually contains an iso9660 file system, 04 cloop: Initializing cloop v2.622 which is the most popular read-only and 05 cloop: loaded (max 8 devices) read-optimized filesystem used for data CDs. Cloop supports all filesystem fea- 06 cloop: losetup_file: 3571 blocks, 65536 bytes/block, largest block tures necessary for a Unix system is 65562 bytes. through the standard RockRidge exten- sion (long file names, permissions, Listing 3: Starting the Cloop Block Device symlinks, etc.). 01 register_blkdev(major=240, cloop_name="cloop"); Cloop also comes with several validity checks that return non-fatal error codes 02 for(i=0; i<cloop_max; i++) cloop_dev[i] = cloop_alloc(i); to the calling application in case invalid 03 if(file) { compressed data is read or read errors 04 initial_file=filp_open(file,O_RDONLY|O_LARGEFILE,0x00); occur (scratches on the CD surface 05 cloop_set_file(0,initial_file,file); sometimes make data partly unread- able). Cloop only uses data that has 06 } JANUARY 2008 ISSUE 86 33 032-036_cloop.indd 33 15.11.2007 10:17:26 Uhr COVER STORY Cloop Cloop images can be attached and de- If a file is given as module parameter Block devices have two methods for tached at run time using the standard file="/path/to/first/image", this file is reading from the device. The direct losetup command (see Listing 2). opened and associated with the first method is setting up a request function Cloop can work with input files that cloop device /dev/cloop0. that will be called each and every time a are mounted over NFS. This feature is How does /dev/cloop0 know what to read() on the device is performed. This important if you plan to run completely do when a process opens the block de- request function is then associated with diskless clients with filesystems vice and reads from it? Unlike filesys- the default block queue that has been mounted over cloop. Another interesting tems, block devices have a more com- created for the major device ID of cloop feature of cloop is that it can read and plex way of performing input and out- when the block device was set up (see decompress data asynchronously using put. Listing 5a). a kernel thread, so it’s not blocking a Depending on which kind of block de- cloop_request_fn() has to find out process group and not staying in kernel vice you are using, there are some opera- which device was accessed and then space for a noticeable time. tions that have to be supported and oth- transfer data from the cloop file into the Access to cloop image files can be sus- ers that won’t. In case of a disk or parti- memory space that was given in buffer_ pended while they are in use (a feature tion, you can use a structure like the one head. This direct method was used until introduced by Fabian Franz). The idea in Listing 4, which tells the kernel what cloop version 2.06. behind this feature is that you can tem- to do if someone opens or closes a /dev/ A disadvantage of this direct method porarily remove a storage device that is cloop* file. is that the entire device is blocked until in use by cloop without getting read er- cloop_open() needs to increase the use cloop_request_fn() returns. Actually, rors. After receiving a special ioctl call, counter of the device so that the module probably because of this, the direct cloop will patiently wait until the under- knows it is in use. cloop_close() does the method ceased to work for cloop starting lying file is activated (and re-analyzed) opposite. cloop_ioctl(), which is a special from kernel 2.6.22. again, so you could eject a Live CD case that handles losetup for exchanging while an OS is running from it. the underlying cloop image file, also sus- Per-Device Wait Queue One drawback is that all processes pends device operations in case the The new approach to block device I/ O is that access the cloop device will be CLOOP_SUSPEND ioctl is sent.