Diskless Debian Etch W. Trevor King∗ November 18, 2010 Contents 1 Overview 1 1.1 Physical setup.............................................1 1.2 Notation................................................1 2 Basic server setup 2 2.1 Installing the OS...........................................2 2.2 Configuring networking.......................................2 3 Remote booting 3 3.1 Server services............................................3 3.1.1 pxelinux............................................3 3.1.2 DHCP.............................................4 3.1.3 TFTP.............................................5 3.1.4 NFS..............................................6 3.2 Client setup..............................................6 3.2.1 Root file system.......................................6 3.2.2 Configuring etc........................................7 3.2.3 Kernel and initial ramdisk..................................7 3.3 Synthesis...............................................8 4 Adding clients 9 Appendix 9 A Compiling a kernel 10 B Troubleshooting 11 B.1 Waiting for /usr/........................................... 12 Bibliography 12 1 Overview This HOWTO details the procedure I used to set up the abax cluster for NFS-rooted network booting. The system is useful in this case because it centralizes the installation in the head node (server), which makes maintaining, upgrading, or altering the computational nodes (clients) easier. ∗Drexel University 1 This procedure follows mainly Tim Brom's Microwulf configuration notes[1] with two major differences. • Microwulf uses Ubuntu (gutsy?), and I'm using Debian etch. • Microwulf has a seperate partition for each client's root, populated with an independent installation from CD. I'm using a single partition for all of my clients, with the base system created using deboot- strap (no CD). For guidance in my deviations, I'm indebted to Bart Trojanowski's pxeboot and nfsroot notes[2] and Falko Timme's notes on kernel compilation in Debian Etch[3]. 1.1 Physical setup Our cluster has one server with eight clients. The server has two network cards, eth0 and eth1. eth1 is connected to the outsize world (WAN). All of the clients have one network card, eth0. All of the eth0s are connected together through a gigabit switch (LAN). 1.2 Notation Throughout this HOWTO, I will use # as the prompt for root, $ as the prompt for an unpriveledged user, and chroot# as the prompt for a root in a chrooted environment. File contents will be framed with the full path listed labeling the frame /path/to/file Contents of file All files are complete with the exception of lines containing ..., in which case the meaning of the example should be clear from the context. 2 Basic server setup 2.1 Installing the OS Boot the server with the Debian installation kernel following one of the options in the Debian installation guide[4]. I netbooted my server from one of the client nodes following this procedure to set up the DHCP and TFTP servers on the client and untarring http://http.us.debian.org/debian/dists/etch/main/ installer-i386/current/images/netboot/netboot.tar.gz in my tftpboot directory. After netbooting from a client, don't forget to take that client down so you woln't have DHCP conflicts once you set up a DHCP server on your server. Install Debian in whatever manner seems most appropriate to you. I partitioned my 160 GB drive manually according to Mount point Type Size / ext3 280 MB /usr ext3 20 GB /var ext3 20 GB /swap swap 1 GB /tmp ext3 5 GB /diskless ext3 20 GB /home ext3 93.7 GB I went with a highly partitioned drive to ease mounting, since I will be sharing some partitions with my clients. To understand why partitioning is useful, see the Partition HOWTO[5]. You can install whichever packages you like, but I went with just the standard set (no Desktop, Server, etc.). You can adjust your installation later with any of (not an exhaustive list) 2 • tasksel : command line, coarse-grained package control[6]. • apt-get : command line, fine-grained package control. • aptitude : curses frontend for apt-get. • synaptic : gtk+ frontend for apt-get. • dpkg : command line, package-management without dependency checking. The base install is pretty bare, but I don't need a full blown desktop, so I flesh out my system with # apt-get install xserver-xorg fluxbox fluxconf iceweasel xterm xpdf # apt-get install build-essentials emacs21-nox which gives me a bare-bones graphical system (fire it up with startx), and a bunch of critical build tools (make, gcc, etc.). 2.2 Configuring networking We need to set up our server so that eth1 assumes it's appropriate static IP on the WAN, and eth0 assumes it's appropriate static IP on the LAN. We achieve this by changing the default /etc/network/interfaces to /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback allow-hotplug eth0 # start on boot & when plugged in iface eth0 inet static # static LAN interface address 192.168.2.100 netmask 255.255.255.0 broadcast 192.168.2.255 allow-hotplug eth1 # start on boot & when plugged in #iface eth1 inet dhcp # WAN DHCP interface (not used) iface eth1 inet static # WAN static interface address XXX.XXX.YYY.YYY netmask 255.255.128.0 broadcast XXX.XXX.127.255 gateway XXX.XXX.ZZZ.ZZZ where I've censored our external IPs for privacy. The netmask selects which addresses belong to which networks. The way we've set it up, all 192.168.2.xxx messages will be routed out eth0, and everything else will go through eth1 to it's gateway. See the Net-HOWTO[7] for more details. 3 Remote booting 3.1 Server services The clients will boot remotely using the Pre eXecution Environment (PXE). The boot procedure is 3 1. Client powers on. 2. Client BIOS comes up, detects attached devices, and looks for a DHCP server for advice on network booting. 3. DHCP server gives client an IP address, domain name, host name, the IP address of the TFTP server, and the location of the bootloader on the TFTP server. 4. Client gets bootloader from TFTP server. 5. BIOS hands over control to bootloader. 6. Bootloader gets kernel and initial ramdisk from TFTP server. 7. Bootloader hands over control to kernel 8. Kernel starts up the system, mounting root via NFS. 9. after this point, it's just like a normal boot process. We can see that we need to set up DHCP, TFTP, and NFS servers (not necessarily on the same server, but they are in our case). 3.1.1 pxelinux The bootloader can be obtained with # apt-get install syslinux which installs it to /usr/lib/syslinux/pxelinux.0 along with a manual[8] and some other syslinux tools. 3.1.2 DHCP Install a server with # apt-get install dhcp Configure the server with /etc/dhcpd.conf allow bootp; # maybe? allow booting;# maybe? option domain-name "your.domain.com"; option domain-name-servers XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY; subnet 192.168.2.0 netmask 255.255.255.0 { range 192.168.2.150 192.168.2.200; # non-static IP range option broadcast-address 192.168.2.255; option routers 192.168.2.100; # Gateway server next-server 192.168.2.100; # TFTP server filename "pxelinux.0"; # bootloader host n1 { hardware ethernet ZZ:ZZ:ZZ:ZZ:ZZ:ZZ; fixed-address 192.168.2.101; option root-path "192.168.2.100:/diskless/n1"; option host-name "n1"; } ... more hosts for other client nodes ... } 4 This assigns the client a static hostname, domain name, and IP address according to it's ethernet address (aka MAC address). It also tells all the clients to ask the TFTP server on 192.168.2.100 for the bootloader pxelinux.0. For extra fun, it tells the clients to send packets to the router at 192.168.2.100 if they can't figure out where they should go, and to use particular DNS servers to resolve domain names to IP addresses. This gives them access to the outside WAN. I don't know yet if the booting options are necessary, since I don't know what they do. We also need to ensure that the DHCP server only binds to eth0, since starting a DHCP server on your WAN will make you unpopular with your ISP. /etc/default/dhcp INTERFACES="eth0" Once the DHCP server is configured, you can start it with # /etc/init.d/dhcp restart Check that the server is actually up with # ps -e | grep dhcp and if it is not, look for error messages in # grep -i dhcp /var/log/syslog 3.1.3 TFTP There are several TFTP server packages. We use atftpd here, but tftp-hpa is also popular. Install atftpd with # apt-get install atftpd xinetd where xinetd is a super-server (replacing inetd, see man xinetd for details). Configure atftpd with /etc/xinetd.d/atftpd service tftp { disable = no socket_type = dgram protocol = udp wait = yes user = nobody server = /usr/sbin/in.tftpd server_args = --tftpd-timeout 300 --retry-timeout 5 --bind-address 192.168.2.100 --mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1 --maxthread 100 --logfile /var/log/atftpd.log --verbose=10 /diskless/tftpboot } Note that the server_args should all be on a single, long line, since I haven't been able to discover if xinetd recognizes escaped endlines yet. This configuration tells xinetd to provide TFTP services by running in.tftpd (the daemon form of atftpd) as user nobody. Most of the options we pass to in.tftpd involve multicasting, which I believe is only used for MTFTP (which pxelinux.0 doesn't use). --logfile /var/log/atftpd.log --verbose=10 logs lots of detail to /var/log/atftpd.log if it exists.