TUTORIAL PYPARTED
PYPARTED: PYTHON DOES TUTORIAL DISK PARTITIONING
VALENTINE Build a custom, command line disk partitioning tool by joining the user-friendliness of Python and the power of C. SINITSYN artitioning is a traditional way to split disk install to build and install it. It’s a good idea to install WHY DO THIS? drives into manageable chunks. Linux comes PyParted you’ve built yourself inside the virtualenv • Take complete control of Pwith variety of tools to do the job: there are (see http://docs.python-guide.org/en/latest/dev/ your system’s hard disk. fdisk, cfdisk or GNU Parted, to name a few. GNU virtualenvs for details), to keep your system • Write a custom installer Parted is powered by a library by the name of directories clean. There is also a Makefile, if you wish. to make things easier libparted, which also lends functionality to many This article’s examples use PyParted 3.10, but the for your users. graphical tools such as famous GParted. Although it’s concepts will stay the same regardless of the version • Amaze your friends and family with your mastery powerful, libparted is written in pure C and thus not you actually use. of the libparted C library. very easy for the non-expert to tap into. If you’re Before we start, a standard caution: partitioning writing your own custom partitioner, to use libparted may harm the data on your hard drive. Back you’re going to have to manage memory manually everything up before you do anything else! and do all the other elbow grease you do in C. This is where PyParted comes in – a set of Python bindings Basic concepts to libparted and a class library built on top of them, The PyParted API has two layers. At the bottom one is initially developed by Red Hat for use in the Anaconda the _ped module. Implemented entirely in C, it tries to installer. keep as close as possible to the native libparted C API. So why would you consider writing disk partitioning On top of that, the ‘parted’ package with high-level software? There could be several reasons: Python classes, functions and exceptions is built. You You are developing a system-level component like can use _ped functions directly if you wish; however, an installer for your own custom Linux distribution the parted package provides a more Pythonic You are automating a system administration task approach, and is the recommended way to use such as batch creation of virtual machine (VM) PyParted in your programs unless you have some images. Tools like ubuntu-vm-builder are great, but special requirements. We won’t go into any details of they do have their limitations using the _ped module in this article. You’re just having fun. Before you do anything useful with PyParted, you’ll PyParted hasn’t need a Device instance. A Device represents a piece of made its way into the physical hardware in your system, and provides the “A word of caution: partitioning Python Package Index means to obtain its basic properties like model, may harm the data on your (PyPI) yet, but you may geometry (cylinders/heads/sectors – it is mostly fake be lucky enough to find for modern disks, but still used sometimes), logical hard drive. Back everything up!” it in your distribution’s and physical sector sizes and so on. The Device class repositories. Fedora also has methods to read data from the hardware and (naturally), Ubuntu and Debian provide PyParted write it back. To obtain a Device instance, you call one packages, and you can always build PyParted yourself of the global functions exported by PyParted (in the from the sources. You will need the libparted headers examples below, >>> denotes the interactive Python (usually found in libparted-dev or similar package), prompt, and … is an omission for readability reasons Python development files and GCC. PyParted uses the or line continuation if placed at the beginning): distutils package, so simply enter python setup.py >>> import parted >>> # requires root privileges to communicate ... with the kernel >>> [dev.path for dev in parted.getAllDevices()] [u’/dev/sda’, u’/dev/mapper/ubuntu--vg-swap_1’, u’/dev/mapper/ubuntu--vg-root’, u’/dev/mapper/sda5_crypt’] >>> # get Device instance by path >>> sda = parted.getDevice(‘/dev/sda’) PyParted was developed to >>> sda.model facilate Red Hat’s installer, u’ATA WDC WD1002FAEX-0’ Anaconda. >>> sda.hardwareGeometry, sda.biosGeometry
94 www.linuxvoice.com PYPARTED TUTORIAL
((121601, 255, 63), (121601, 255, 63)) # cylinders, heads, sectors Caution: partitioning may void your warranty >>> sda.sectorSize, sda.physicalSectorSize Playing with partitioning is fun but also quite is writable for the user that you are running (512L, 512L) dangerous: wiping the partition table on your scripts as). The last command removes the >>> # Destroy partition table; NEVER do this on machine’s hard drive will almost certainly device when it is no longer needed. ... your computer’s disk! result in data loss. It is much safer to do If you feel adventurous, you can also fake >>> sda.clobber() your experiments in a virtual machine (like your hard drive with a qcow2 (as used by VirtualBox) with two hard drives attached. If Qemu), VDI, VMDK or other image directly True this is not an option, you can ‘fake’ the hard supported by virt-manager, Oracle VirtualBox Next comes the Disk, which is the lowest-level drive with an image file ($ is a normal user or VMware Workstation/Player. These operating system-specific abstraction in the PyParted and # is a superuser shell prompt): images can be created with qemu-img and class hierarchy. To get a Disk instance, you’ll need a $ dd if=/dev/zero of=
www.linuxvoice.com 95 TUTORIAL PYPARTED
Let’s start with Alignment, which is defined by two values: offset and grain. Any sector number X with X = offset + N * grain (with N being non-negative integer) complies with Alignment. When you need to tell PyParted that your partitions should start (or end) at a 1MiB (or some other) boundary, Alignment is the way to do it. Any value satisfies Alignment(0, 1) which is equivalent to no alignment at all. Constraint is basically a set of six conditions on Geometry (not a Partition!) that are wrapped together to control the following: How the Geometry’s boundaries are aligned (startAlign/endAlign properties). Where the Geometry can start or end (startRange/ endRange). fdisk displays partition u’ntfs’, u’zfs’, u’affs4’, u’hfs’, u’affs6’, u’affs1’, u’affs0’, What the Geometry’s minimum and maximum table on the author’s u’affs3’, u’hp-ufs’, u’fat16’, u’sun-ufs’, u’asfs’, u’jfs’, sizes are (minSize/maxSize). computer. u’apfs2’, u’apfs1’] You do not always need to specify all of them. The To add a partition to the disk, use Constraint constructor provides the shortcuts disk.addPartition(): minGeom, maxGeom and exactGeom, which create a >>> disk.addPartition(new_partition) Constraint that fully embraces, is fully contained by, or Traceback (most recent call last): exactly coincides with the Geometry you pass as an ... argument. If you use one of these, any alignment will _ped.PartitionException: Unable to satisfy all constraints satisfy the Constraint check. As another special case, on the partition. Constraint(device=dev) accepts any Geometry attached to the Device dev. As you can see, partitions on the disk are subject to It isn’t easy to catch the meaning of all these some constraints, which we’ve occasionally violated properties at once. Have a look at the diagram below, here. When you pass a partition to disk.addPartition(), which depicts all of them in graphical form. Both its geometry may change due to constraints that you Alignment and Constraint provide the intersect() spefify via the second argument (in the example method, which returns the object that satisfies both above, it defaults to None), and constraints imposed requirements. You can also check that the given by libparted itself (for instance, in the MBR scheme, it Geometry satisfies the Constraint with the Constraint. won’t let you start a partition at the beginning of a isSolution(geom) method. The Constraint.solveMax() disk, where the partition table itself resides). This is method returns the maximum permitted geometry where things start to that satisfies the Constraint, and Constraint. “Managing constraints is get interesting. solveNearest(geom) returns the permitted geometry that is nearest to the geom that you’ve specified. probably the most complex Know your limits What’s ‘nearest’ is up to the implementation to decide. thing PyParted does for you.” Managing constraints is probably the most Partitioning on Ye Olde Windows NT complex and most Imagine for a moment you need to create system useful part of what PyParted does for you. Partitions partition for Windows NT4 prior to Service Pack 5 A visual representation on a hard disk generally can’t start or end where you (remember that weird creature?). As the hardware of different kwargs want. There should be a gap between the beginning of requirements suggest (http://en.wikipedia.org/wiki/ accepted by the Constraint a disk and your first partition to store internal Windows_NT#Hardware_requirements), it must be constructor. Red/Yellow partitioning data; today, many operating systems no more than 4GB in size, contained within the first rectangles are startRange/ reserve 1MiB for these purposes. Partitions should be 7.8GB of the hard disk, and begin in the first 4GBs. endRange, blue/green are maxGeom/minGeom. Large aligned to physical sector boundaries, or severe Here’s how to do this with PyParted: ticks denote startAlign performance degradation may occur. This is not to >>> optimal = sda.optimumAlignment (lower) or endAlign(upper). say that partitions can’t overlap; PyParted takes care >>> start = parted.Geometry(device=sda, Small ticks represent of all these nuances, and Constraint and Alignment ... start=0, sectors. classes play a central role in this process. ... end=parted.sizeToSectors(4, ‘GB’, ... sda.sectorSize)) >>> end = parted.Geometry(device=sda, ... start=0, ... end=parted.sizeToSectors(7.8, ‘GB’, ... sda.sectorSize)) >>> min_size = parted.sizeToSectors(124, ‘MB’, Hard disk space. ... sda.sectorSize) # See [ref:4]
96 www.linuxvoice.com PYPARTED TUTORIAL
>>> max_size = parted.sizeToSectors(4, ‘GB’, ... sda.sectorSize) Each disk needs a label >>> constraint=parted.Constraint(startAlign=optimal, Many modern operating systems enable you standards, it’s very limited: it may contain ... endAlign=optimal, to assign a label to a disk, which is especially at most four partitions (called ‘primary’) and ... startRange=start, endRange=end, useful for removable media (/media/ stores partition offsets as 32-bit integers. ... minSize=min_size, maxSize=max_size) BobsUSBStick says more than /media/sdb1). If you need more, one partition should be But they are not the disk labels that libparted marked as ‘extended’, and it may contain as >>> disk.addPartition(partition=new_partition, refers to. many logical partitions as you want. This is ... constraint=constraint) When we speak of disk labels on these the reason why logical partitions are always True pages, we mean partition tables. It is very numbered starting with 5 in Linux. >>> print new_partition.geometry uncommon for a hard disk to not to have one The newer GUID Partition Table (‘gpt’) is parted.Geometry instance -- (although many flash drives comes with no much more flexible. It’s usually mentioned partitions). Linux usually sees unpartitioned in connection with UEFI, however it is start: 2048 end: 262144 length: 260097 devices as /dev/sdX (with X being a letter); self-contained and can be used on BIOS ... partitions are suffixed with an integer (say, systems as well. In a ‘gpt’ disk label, If you want to specify startRange or endRange, /dev/sda1). partitions are identified by Globally Unique you’ll need to provide both alignments and size There are many different partitioning Identifiers (GUID) values. Their starting and constraints as well. Now, please go back and look at schemes (or disk labels). Traditionally, the ending offsets are 64-bit, so there is some the first line. As you probably guessed, device. ‘msdos’ (MBR) disk partitioning scheme was safety margin for hard disks of tomorrow’s the most popular one for PCs. By today’s capacities. optimumAlignment and its counterpart, device. minimumAlignment, provides optimum (or minimum) alignment accepted by the hardware device you’re creating the partition on. Under Linux, in-kernel device so I suggest you open the program code now in driver-reported attributes like alignment_offset, GitHub (https://github.com/vsinitsyn/fdisk.py) to minimum_io_size and optimal_io_size are generally follow it as you read the next section. used to determine the meaning of ‘minimum’ and ‘optimum’. For example, an optimally aligned partition Your very own fdisk on a RAID device may start on a stripe boundary, but a fdisk is probably the most basic partitioning tool. It’s fixed 1MiB-grained alignment (as in Windows 7/Vista) an console program: it reads single-letter commands will usually be preferred for an ordinary hard disk. entered by a user and acts accordingly, printing ‘Minimum’ is roughly equivalent to ‘by physical sector results on the screen. Originally, it supported MBR and size’, which can be 4,096 bytes even if the device also BSD/SUN disk labels; we’ll stick to MBR only. advertises traditional 512-bytes sector addressing. Our example (let’s call it fdisk.py) is a somewhat Back to the _ped.PartitionException we saw earlier. more elaborate version of fdisk/fdisk.py found in the In order to fix it, you need to specify the proper PyParted sources https://git.fedorahosted.org/cgit/ constraint: pyparted.git/tree/src/fdisk/fdisk.py, but it’s a bit >>> # Most relaxed constraint; anything on the simplified compared with the real fdisk. Since parted ... device would suffice and fdisk are not 100% compatible (although parted >>> disk.addPartition(new_partition, is more advanced in many ways), there are some ... parted.Constraint(device=sda)) discrepancies (see comments in the sources for True details). However, fdisk.py implements all the basic >>> print new_partition.geometry functions you’d expect from the partitioning software: parted.Geometry instance -- it can create partitions (both primary and logical), list start: 32 end: 262143 length: 262112 them, delete them, and even mark them as bootable. ... All of these options are implemented as methods of >>> print geometry the Fdisk class, which is instantiated when the parted.Geometry instance -- program starts. In Fdisk.__init__(), we check whether start: 0 end: 262143 length: 262144 the drive already has a partition table and create it if ... necessary. If the disk has any partition table other Note that the Geometry we’ve specified was adjusted than MBR, the program exits immediately. The main due to constraints imposed internally by libparted for loop simply dispatches commands entered by a user the MBR partitioning scheme. to Fdisk’s instance methods. If any of them raise an When you’re done, commit the changes you’ve ExitMainLoop exception, the program ends. made to the hardware. Otherwise they will remain in Let’s start with the code that displays a partition memory only, and the real disk won’t be touched: table. In the real fdisk, it looks like the image at the top >>> disk.commit() of page 96. And the following is the relevant part of True fdisk.py code: We’ve seen all the major bits that PyParted is made print “”” from. Now let’s use all of them together in a bigger Disk {path}: {size_mbytes:d} MB, {size:d} bytes program – an fdisk clone, almost full-featured, and {heads:d} heads, {sectors:d} sectors/track, \ just a little more than 400 lines of Python code in size! {cylinders:d} cylinders, total {sectors_total:d} sectors Not all of these lines will be in the magazine, obviously, Units = 1 * sectors of {unit:d} = {unit:d} bytes
www.linuxvoice.com 97 TUTORIAL PYPARTED
previous section). Since we don’t allow our user to change units (as the real fdisk does), unit variable is always equal to sector size. Everything else is straightforward. Parted has no concept of DOS disk label types such as ‘Linux native’, ‘Linux swap’, or ‘Win95 FAT32’. If you were to install good old Slackware using fdisk back in 1999, you would almost certainly use some of these. libparted provides the power behind many well-known So we emulate disk labels to some extent on top of free software tools, including GParted. the partition and filesystem types provided by PyParted. This is done in the Fdisk._guess_system() Sector size (logical/physical): {sector_size:d} \ method. We recognise things like ‘Linux LVM’ and bytes / {physical_sector_size:d} bytes ‘Linux RAID’, parted.PARTITION_SWAP maps to ‘Linux I/O size (minimum/optimal): {minimum_io_size:d} \ swap’, ext2/3/4, btrfs, ReiserFS, XFS, and JFS are bytes / {optimal_io_size:d} bytes displayed as ‘Linux native’, and we even support “””.format(**data) FAT16/32 and NTFS. As a bonus, PyParted enables you to identify hidden or service partitions added by width = len(disk.partitions[0].path) \ some hardware vendors (https://git.fedorahosted. if disk.partitions else len(‘Device’) + 1 org/cgit/pyparted.git/tree/src/fdisk/fdisk.py). If the print “{0:>{width}} Boot Start \ heuristic doesn’t work, we print ‘unknown’. End Blocks Id System”.format(‘Device’, width) The data dictionary is filled as follows: Creating partitions unit = device.sectorSize It is also easy to delete a partition. The only thing to size = device.length * device.sectorSize remember is that partitions on the disk can be out of cylinders, heads, sectors = \ order, so you can’t use the partition number as an device.hardwareGeometry index in the disk.partitions array. Instead, we iterate minGrain, optGrain = \ over it to find the partition with the number that a user device.minimumAlignment.grainSize, has specified: device.optimumAlignment.grainSize for p in self.disk.partitions: data = { if p.number == number: ‘path’: device.path, try: ‘size’: size, self.disk.deletePartition(p) ‘size_mbytes’: int(parted.formatBytes(size, ‘MB’)), except parted.PartitionException as e: ‘heads’: heads, print e.message ‘sectors’: sectors, break ‘cylinders’: cylinders, If we try to delete an extended partition that ‘sectors_total’: device.length, contains logical partitions, parted.PartitionException ‘unit’: unit, will be raised. We catch it and print a friendly error ‘sector_size’: device.sectorSize, message. The last break statement is essential. ‘physical_sector_size’: device.physicalSectorSize, PyParted automatically renumbers the partitions ‘minimum_io_size’: minGrain * device.sectorSize, when you delete any of them. So, if you have, for ‘optimal_io_size’: optGrain * device.sectorSize, instance, partitions 1–4, and delete the one numbered } 3, the partition that was previously number 4 will We can deduce the maximum and optimum I/O become the new 3, and will be deleted at the next sizes from corresponding alignment values (see the iteration of the loop. The largest method, not surprisingly, is the one that Chinese Remainder Theorem creates partitions. Let’s look at it step by step. First of all, we check how many primary and extended If you were curious enough to skim through basket, you’ll need to take them out in partitions are already on the disk, and how many the libparted documentation, you’ve batches of five. Using the Chinese Remainder primary partitions are available: probably spotted a reference to the Chinese Theorem, you can determine how many eggs # Primary partitions count Remainder Theorem. Despite the somewhat are in the basket. flippant name, it’s a serious statement that When you place a partition somewhere pri_count = len(self.disk.getPrimaryPartitions()) comes from number theory. Basically, it lets on a disk, libparted needs to satisfy both # HDDs may contain only one extended partition you to find a minimum integer that yields alignments (among other things). This is ext_count = 1 if self.disk.getExtendedPartition() else 0 given remainders for given divisors. If this accomplished by solving a system of linear # First logical partition number all sounds like gibberish, think of a basket of equations (see the natmath.c source code lpart_start = self.disk.maxPrimaryPartitionCount + 1 eggs. You don’t know how many of them are if you are really curious). It’s amazing to in it, however, if you take them out by twos realise that a 1,500-year old maths problem # Number of spare partitions slots or threes, you’ll have one of them remaining is useful for a free software library of the parts_avail = self.disk.maxPrimaryPartitionCount -\ in the bottom of the basket; to empty the 21st century. (pri_count + ext_count) Then we check if the disk has some free space and
98 www.linuxvoice.com PYPARTED TUTORIAL
return from the method if not. After this, we ask the user for the partition type. If there are no primary Resources partitions available, and no extended partition exists, PyParted homepage https://fedorahosted.org/pyparted one of primary partitions needs to be deleted, so we Virtual Environments guide http://docs.python-guide.org/ return from the method again. Otherwise, a user can en/latest/dev/virtualenvs create either a primary partition, an extended partition Partition types: properties of partition tables (if there isn’t one yet), or a logical partition (if an www.win.tue.nl/~aeb/partitions/partition_types-2.html Windows NT4 Hardware Requirements extended partition is already here). If the disk has http://en.wikipedia.org/wiki/Windows_NT#Hardware_ fewer than three primary partitions, a primary partition requirements is created by default; otherwise we default to an fdisk.py sources (this article’s version) extended or logical one. https://github.com/vsinitsyn/fdisk.py We also need to find a place to store the new PyParted’s fdisk.py sample code https://git.fedorahosted.org/cgit/pyparted.git/tree/src/ partition. For simplicity’s sake, we use the largest free fdisk/fdisk.py region available. Fdisk._get_largest_free_region() is responsible for this; it’s quite straightforward except one simple heuristic. It ignores regions shorter than After that, Fdisk._create_partition() asks for the optimum alignment grain (usually 2048 sectors): they beginning and the end of the partition. Fdisk._parse_ are most likely alignment gaps. last_sector_expr() parses expressions like +100M, Any logical partition created must fit inside the which fdisk(1) uses as the last sector specifier. Then, extended partition, and we use Geometry.intersect() the partition is created as usual: to ensure that this is the case. On the contrary, a try: primary partition must lie outside the extended, so if partition = parted.Partition( the intersection exists, we return from the method. disk=self.disk, The code is similar in both cases; below is the former type=type, check (which is a bit shorter): geometry=parted.Geometry( try: device=self.device, geometry = ext_part.geometry.intersect(geometry) start=part_start, except ArithmeticError: end=part_end)) print “No free sectors available” self.disk.addPartition(partition=partition, return constraint=constraint) If there is no intersection, Geometry.intersect() except (parted.PartitionException, raises ArithmeticError. parted.GeometryException, All the heavy lifting is done in the Fdisk._create_ parted.CreateException) as e: partition() method, which accepts the partition type raise RuntimeError(e.message) and the region that will hold the new partition. It starts If part_start or part_end are incorrect, the exception as follows: will be raised (see the comments in the source code alignment = self.device.optimalAlignedConstraint for the details). It is caught in the Fdisk.add_partition() constraint = parted.Constraint(maxGeom=geometry).\ method, which displays error messages and returns. intersect(alignment) To save the partition table on to the disk, a user data = { enters the w command at the fdisk.py prompt. The ‘start’: constraint.startAlign.\ corresponding method (Fdisk.write()) simply calls alignUp(region, region.start), disk.commit() and raises MainLoopExit to exit. ‘end’: constraint.endAlign.\ alignDown(region, region.end), Afore ye go } Python is arguably the scripting language of choice in As in the real fdisk(1), we align partitions optimally today’s Linux landscape, and is widely used for by default. The partition created must be no larger various tasks including the creation of system-level than the available free space (the region argument), components. As an interpreted language, Python is so the maxGeom constraint is enforced. Intersecting just as powerful as its bindings, which enable scripts these gives us a Constraint that aligns partitions to make use of native C libraries. In this perspective, optimally within boundaries specified. data[‘start’] and it’s nice to have tools like PyParted in our arsenal. data[‘end’] are used as guidelines when prompting for Implementing partitioners is hardly an everyday task the partition’s boundaries, and they shouldn’t be for most of us, but if you ever face it, the combination misleading. Thus we perform the same calculation of an easy-to-use language and a production-grade that libparted does internally: find start or end values library can greatly reduce your programming efforts that are in a specified range and aligned properly. Try and development time. to play with these; for example, change the alignment to self.device.minimalAlignedConstraint and see Dr Valentine Sinitsyn edited the Russian edition of O’Reilly’s what changes when you create a partition on an Understanding the Linux Kernel, has a PhD in physics, and is currently doing clever things with Python. empty disk.
www.linuxvoice.com 99