Moobi: A Thin Server Management System Using BitTorrent Chris McEniry – Sony Computer Entertainment America ABSTRACT We describe a tool that provides a method for running dataless caching clients – a hybrid combination of imaging systems with traditional diskless nodes. Unlike imaging systems, it is a single boot to get to a running system; unlike diskless systems, it is more robust and scalable as it does not continuously depend on central servers. The tool, Moobi, uses the peer-to-peer protocol BitTorrent to provide efficient distribution of the image cache, and combines standard diskless tools to provide the basis for the running system. Moobi makes it possible to run large installations of ‘‘thin server’’ farms. Introduction system for these setups must account and check for ev- ery piece individually to validate the consistency of the Diskless clients are appealing for large clusters environment. of similar systems performing similar functions since they limit the capability for local configurations and, Truly diskless clients do not have this problem. more specifically, local misconfigurations on each sys- Nothing is locally maintained, and a single instance, tem. This provides a more uniform environment and typically of an NFS server, is directly used for the en- allows for the management of a large number of sys- vironment. This provides the consistency, but loses in tems with fewer resources. The key for these systems performance: diskless clients are very hard to scale to is not so much that they are diskless in that they are 100s of nodes. This limitation on scaling is due to dataless – there is nothing which is inherently tied to their reliance on the large file servers that support the the nodes. In the desktop realm, these are typically re- diskless nodes. An installed environment suffers a ferred to as thin clients. In the server realm, ‘‘throw similar scaling issue whenever massive deployments away’’ and ‘‘field replaceable units(FRUs)’’ are some have to happen in short order – initial spin ups, disas- common terms. ter recovery, or massive updates. Diskless client envi- ronments have a continuous scaling issue, while the The difference between diskless and dataless installed environments have ‘‘impulse’’ scaling issues. hinges upon drive cost, application robustness over drive failure rates, and the need for local working disk. Moobi combines the consistency of diskless sys- The size of drives has increased significantly, especially tems with the efficiency of imaged or installed sys- when compared to the size of operating systems and ap- tems. It achieves this by dividing the operating system plications bases. Pinheiro, et al. [1] demonstrated that image into a small configurable portion – the root or annual hard drive failure rates were between 2% and /etc – and a large fixed image – /usr – and by caching 10% depending on age and usage; even with 2%, in a and sharing the large image via BitTorrent. The fixed large population of machines, there is an inherent need image is maintained as a whole instance, so only one for applications to account for downed nodes and item needs to be validated against the source. Every thereby make drive failure moot. In addition, a local time a node boots, or as often as is necessary for audit- drive provides a convenient location for cached data or ing purposes, it can scan the cache and downloads and temporary work space, and so they are included in most updates the cache with any new image. clusters regardless of management methods. These rea- BitTorrent is used for validating and updating the sons push commodity servers towards having local stor- image cache. Due to its nature, these two operations are age and treating it as a large cache or work disk, but not inherently connected. In order to download and update, as a true data disk. it must validate. It sees updating as the only logical fol- Since these systems have drives, they also have a lowup after validating. In addition, BitTorrent allows unique local instance of the operating system. Whether for an update mechanism which does not require large this operating system is installed via an imaging system, monolithic file servers; thus it alleviates the bulk of the or installed via an automated installation mechanism is scaling issues for large installations. In addition, BitTor- largely irrelevant since the result is an individual system rent allows for multiple systems to distribute the image instance with a unique instance of every part of that sys- without much additional logic. This makes it easier to tem. This individual identity is necessary for certain make the system most robust overall. items – hostname, ssh keys – but is undesirable for other The mechanisms used in Moobi provide addi- items – /usr software. Any configuration management tional availability for applications running on Moobi 21st Large Installation System Administration Conference (LISA ’07) 253 Moobi: A Thin Server Management System Using BitTorrent McEniry nodes. Not only does Moobi reduce the MTTR, but it operating systems, and also have a long history of per- also provides several hooks for system hardening. Peri- formance issues with a central server [7]. Traugott and odic image checking provides a level of confidence in Huddleston [8] highlight Sun Autoclient’s CacheFS the sanctity of the software being used. In addition, the features to reduce this. Moobi reduces these issues to static file system images, such as the software image occurring only at boot time, and allows for redundant /usr, can be mounted with read-only permissions. Appli- boot servers with little or no synchronization between cation stacks could be wrapped up in a similar manner them. to provide extra confidence in the stack. BitTorrent is heavily in use in distributing large Going to a hybrid solution also acts as a stepping software images to large audiences. Many linux distri- stone. Most organizations grow organically from indi- butions, live CDs, large software applications, and vidual system instances to dataless systems. Hard VMWare appliance images [9] are distributed using drives tend to remain an item in new systems, and are BitTorrent. There are discussions of integrating Bit- not removed from repurposed systems. Torrent with linux distributions’ installation or update systems, but this discussion has produced little. Related Work Recently, SystemImager [10] of the System Instal- Given Moobi’s hybrid nature, similar work falls lation Suite has incorporated BitTorrent as a distribution into either the imaging and installation category or in- method for its imaging system. Moobi and SystemIm- to the diskless client category, yet neither category ager run into many similar obstacles, especially when completely describes it. Unlike imaging systems, it is handling image versus host specific overlays, and when a single boot to get to a running system; unlike disk- attempting to achieve high performance and scalability. less systems, it is less fragile as it does not continuous- However, SystemImager still views the many systems ly depend on central servers. as single instances and can fall into divergent configura- Several commercial and noncommercial imaging tions. In addition, SystemImager has a two boot imaging systems exist to date. Norton Ghost [2] is the industry operation – one boot for the imaging, and another boot standard for commercial system imaging software. to start the imaged system for normal operation. Several noncommercial systems provide similar fea- Moobi Design tures to Ghost, and have been able to address many of the distribution issues. Frisbee [3], in particular, has Moobi Basics been shown to have very good image distribution per- Several aspects are assumed for the Moobi image formance [4]. Partition Image [5] provides a wide building and deployment discussion: how Moobi lever- range of image support, but is not very efficient in its ages the linux boot process, how Moobi distributes its distribution. own software, how additional configuration information Ty p i c a l system imaging packages use one of two is passed to identical nodes, and host specific overlays. network mechanisms for the transporting of the golden First, Moobi intercepts the normal linux boot image to the target machine: a unicast transfer, or a mul- process to prepare a specialized environment prior to ticast or broadcast blast. Unicast transfers can be very handing off to the standard init process. expensive on the boot servers. Broadcast is very stress- Almesberger [11] describes the linux boot process ing on the network. Multicast requires advanced syn- in great detail. The relevant portions for this discussion chronization techniques and places additional con- are summarized here. With loadable modules being as straints on how and when nodes can come online. In prominent as they are, almost all recent linux boots re- contrast, a BitTorrent swarm limits server load, limits quire the use of the initrd. Typically, the initrd is a pseu- the scope of network usage to the ports involved, and do-root file system which contains the kernel module automatically handles synchronization – if a BitTorrent utilities as well as any modules necessary for the system client is late to the swarm, its peers will catch it up. be able to mount the true root file system. If this is a lo- All imaging systems require some mechanism to cal disk, disk drivers and file system drives are loaded. If address host specific file overlays. Typically, this is done this is a network file system, the NFS modules are load- by just copying down the necessary files from a central ed. Once the kernel is done with its primary initializa- system. Ghost in particular is very good at performing tion, it mounts the initrd as a ramdisk and executes the post imaging instance fix-ups to reduce unique identifier linuxrc script contained therein. Typically, the linuxrc conflict without needing a central software repository. script is simplified to the point of just loading the mod- However, it does not address non-windows operating ules necessary and then hands off execution to init.