Linux Clusters Institute: Lustre Hands On Exercise University of Oklahoma, May 13th – 17th 2019 J.D. Maloney | Storage Engineer National Center for Supercomputing Applications (NCSA) [email protected] This document is a result of work by volunteer LCI instructors and is licensed under CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/). Goal of Hands on Exercise • Create Lustre File System with 1 MDT & 6 OSTs • Bring in Sample Data • Configure Quotas • Test how stripe width works May 13th-17th 2019 2 Lay of the Land • You should have 4 storage servers; 1 for metadata, 3 for data • Each have 2 5GB volumes • Root SSH between all nodes in cluster • Chronyd keeping time in sync May 13th-17th 2019 3 Creating Lustre File System • Add Lustre Repos to Servers & Compute Nodes (create /etc/yum.repos.d/lustre.repo) [lustre-server] name=CentOS-$releasever - Lustre baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7/server/ gpgcheck=0 [e2fsprogs] name=CentOS-$releasever - Ldiskfs baseurl=https://downloads.hpdd.intel.com/public/e2fsprogs/latest/el7/ gpgcheck=0 [lustre-client] name=CentOS-$releasever - Lustre baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7/client/ gpgcheck=0 May 13th-17th 2019 4 Creating Lustre File System • Update/Install these packages on the four storage nodes yum upgrade -y e2fsprogs yum install -y lustre-tests • Create /etc/modprobe.d/lnet.conf and add the following on all four storage nodes options lnet networks=tcp0(eth0) • Reboot all four storage VMs shutdown –r now May 13th-17th 2019 5 Creating Lustre File System • Format the MDT on storage-0 mkfs.lustre --fsname=lci --mgs --mdt --index=0 /dev/sdc • Mount the MDT on storage-0 mkdir /mnt/mdt && mount –t lustre /dev/sdc /mnt/mdt • On storage-1, storage-2, storage-3 format the OSTs -- make sure to increment the index # for each disk in the cluster (0-5) mkfs.lustre --ost --fsname=lci --mgsnode=storage-0@tcp0 --index=0 /dev/sdc mkfs.lustre --ost --fsname=lci --mgsnode=storage-0@tcp0 --index=1 /dev/sdd …continue to do the remaining four May 13th-17th 2019 6 Creating Lustre File System • Mount the OST’s on the OSS nodes (storage-1,2,3) mkdir /mnt/ost0 && mount –t lustre /dev/sdc /mnt/ost0 mkdir /mnt/ost1 && mount –t lustre /dev/sdd /mnt/ost1 …continue to do the remaining four May 13th-17th 2019 7 Installing Client Nodes • On compute-0 and compute-1 run: yum -y install lustre-client • Probe the module: modprobe lustre • Create the mountpoint for the file system mkdir /mnt/lci May 13th-17th 2019 8 Mount the File System • On compute-0 and compute-1 run: mount -t lustre storage-0@tcp0:/lci /mnt/lci • You should have a fully mounted Lustre file system [root@Compute-0 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 30G 2.5G 28G 9% / devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 17M 3.9G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/sda1 497M 90M 408M 18% /boot /dev/sdb1 16G 45M 15G 1% /mnt/resource Login:/share/home 30G 2.2G 28G 8% /share/home 192.168.0.7@tcp:/lci 28G 199M 26G 1% /mnt/lci tmpfs 797M 0 797M 0% /run/user/0 May 13th-17th 2019 9 Bring In Test Data • Create /mnt/lci/home, /mnt/lci/projects, /mnt/lci/scratch directories just for posterity • Bring in folder 1 of test data • Same data and source as the Spectrum Scale Hands On May 13th-17th 2019 10 Setting Quotas • Set a quota for the user Tim & verify it worked: lfs setquota -u tim -b 1G -B 2G -i 10000 -I 15000 /mnt/lci lfs quota -hu tim /mnt/lci Disk quotas for usr tim (uid 7008): Filesystem used quota limit grace files quota limit grace /mnt/lci 0k 1G 2G - 0 10000 15000 - • Set a quota for the group hpc & verify it worked: lfs setquota -g hpc -b 3G -B 4G -i 20000 -I 25000 /mnt/lci lfs quota -hg hpc /mnt/lci Disk quotas for grp hpc (gid 7007): Filesystem used quota limit grace files quota limit grace /mnt/lci 0k 3G 4G - 0 20000 25000 - • Feel free to play around with more quotas May 13th-17th 2019 11 Playing with Stripe Width • Set the default stripe width and stripe size to 2 and 4M respectively: [root@Storage-0 lci]# lfs getstripe -d /mnt/lci stripe_count: 1 stripe_size: 1048576 pattern: 0 stripe_offset: -1 [root@Storage-0 lci]# lfs setstripe -S 4M -c 2 /mnt/lci [root@Storage-0 lci]# lfs getstripe -d /mnt/lci stripe_count: 2 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 • Look at a twitter stream file that was brought in at width 1 [root@Storage-0 twitter]# lfs getstripe 2012_12_07_stream.txt 2012_12_07_stream.txt lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5 0x5 0 May 13th-17th 2019 12 Playing with Stripe Width • Remove and re-sync the sample data over again, check it picks up the proper striping now: [root@Storage-0 twitter]# lfs getstripe 2012_12_07_stream.txt 2012_12_07_stream.txt lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 5 0x5 0 1 7 0x7 0 May 13th-17th 2019 13 Wrap Up • Further Exploration • Other Lustre commands or potentially set up Robinhood May 13th-17th 2019 14.