MooseFS 3.0 User's Manual Core Technology Development & Support Team January 7, 2017 c 2014-2017 v. 1.0.5 Piotr Robert Konopelko, Core Technology Development & Support Team. All rights reserved. Proofread by Agata Kruszona-Zawadzka Coordination & layout by Piotr Robert Konopelko. Please send corrections to Piotr Robert Konopelko { [email protected]. 1 Contents 1 About MooseFS 6 1.1 Architecture . .6 1.2 How does the system work . .8 1.3 Fault tolerance . .9 1.4 Platforms . 10 2 Moose File System Requirements 11 2.1 Network requirements . 11 2.2 Requirements for Master Servers . 11 2.2.1 CPU . 11 2.2.2 RAM size . 12 2.2.3 HDD free space . 12 2.3 Requirements for Metalogger(s) . 12 2.4 Requirements for Chunkservers . 13 2.4.1 CPU . 13 2.4.2 RAM size . 13 2.4.3 HDD space . 13 2.5 Requirements for Clients / Mounts . 13 3 Installing MooseFS 3.0 15 3.1 Configuring DNS Server . 15 3.2 Adding repositories . 16 3.2.1 Ubuntu / Debian . 16 3.2.2 RedHat / CentOS (EL7) . 16 3.2.3 RedHat / CentOS (EL6) . 17 3.2.4 Apple MacOS X . 17 3.3 Differences in package names between MooseFS Pro and MooseFS . 17 3.4 MooseFS Master Server(s) installation . 18 3.5 MooseFS CGI Monitor, CGI Server and Command Line Interface installation . 19 3.6 Chunk servers installation . 20 3.7 MooseFS Clients installation . 20 3.8 Enabling MooseFS services during OS boot . 22 3.8.1 RedHat / Centos (EL6) . 22 3.8.2 RedHat / Centos (EL7) . 22 3.8.3 Debian / Ubuntu . 23 3.8.4 FreeBSD . 23 3.9 Basic MooseFS use . 25 3.10 Stopping MooseFS . 25 2 4 Storage Classes 26 4.1 Introduction to Storage Classes functionality in MooseFS 3.0 . 26 4.1.1 What is a Storage Class? . 26 4.1.2 What are labels? . 26 4.2 How to use Storage Classes? . 27 4.2.1 Machines configuration . 27 4.2.2 Example of MooseFS installation without Storage Classes . 27 4.2.3 Labelling Chunkservers . 28 4.2.4 Creating Storage Classes . 30 4.2.5 Listing Storage Classes . 31 4.2.6 Assigning Storage Class to files / directories . 31 4.2.7 Creation, keep, archive labels . 33 4.2.8 Chunkserver states . 34 4.2.9 Chunk creation modes . 34 4.2.10 Preferred labels during read/write (in mfsmount).............. 35 4.3 Storage Classes tools . 36 4.3.1 MooseFS Storage Class administration tool { mfsscadmin ......... 36 4.3.2 MooseFS Storage Class management tools { mfssclass .......... 39 4.4 Common use scenarios . 41 4.4.1 Scenario 1: Two server rooms (A and B) . 41 4.4.2 Scenario 2: SSD and HDD drives . 42 4.4.3 Scenario 3: Two server rooms (A and B) + SSD and HDD drives . 44 4.4.4 Scenario 4: Creation, Keep and Archive modes . 46 5 Troubleshooting 47 5.1 Metadata save . 47 5.2 Master metadata restore from Metaloggers . 48 5.3 Maintenance mode . 48 5.4 Chunk replication priorities . 49 6 MooseFS Tools 50 6.1 For MooseFS Master Server(s) . 50 6.1.1 mfsmaster ................................... 50 6.1.2 mfsmetarestore ................................ 51 6.1.3 mfsmetadump .................................. 51 6.2 For MooseFS Supervisor . 55 6.2.1 mfssupervisor ................................. 55 6.3 For MooseFS Command Line Interface . 56 6.3.1 mfscli ..................................... 56 6.4 For MooseFS CGI Server . 58 6.4.1 mfscgiserv ................................... 58 6.5 For MooseFS Metalogger(s) . 59 6.5.1 mfsmetalogger ................................. 59 6.6 For MooseFS Chunkserver(s) . 60 6.6.1 mfschunkserver ................................ 60 6.7 For MooseFS Client . 61 6.7.1 mfsmount .................................... 61 6.7.2 mfstools .................................... 64 7 MooseFS Configuration Files 68 3 7.1 For MooseFS Master Server(s) . 68 7.1.1 mfsmaster.cfg ................................. 68 7.1.2 mfsexports.cfg ................................ 71 7.1.3 mfstopology.cfg ............................... 73 7.2 For MooseFS Metalogger(s) . 74 7.2.1 mfsmetalogger.cfg .............................. 74 7.3 For MooseFS Chunkservers . 75 7.3.1 mfschunkserver.cfg ............................. 75 7.3.2 mfshdd.cfg ................................... 76 8 Frequently Asked Questions 77 8.1 What average write/read speeds can we expect? . 77 8.2 Does the goal setting influence writing/reading speeds? . 77 8.3 Are concurrent read and write operations supported? . 77 8.4 How much CPU/RAM resources are used? . 78 8.5 Is it possible to add/remove chunkservers and disks on the fly? . 78 8.6 How to mark a disk for removal? . 79 8.7 My experience with clustered filesystems is that metadata operations are quite slow. How did you resolve this problem? . 79 8.8 What does value of directory size mean on MooseFS? It is different than standard Linux ls -l output. Why? . 79 8.9 When I perform df -h on a filesystem the results are different from what I would expect taking into account actual sizes of written files. 80 8.10 Can I keep source code on MooseFS? Why do small files occupy more space than I would have expected? . 80 8.11 Do Chunkservers and Metadata Server do their own checksumming? . 81 8.12 What resources are required for the Master Server? . 82 8.13 When I delete files or directories, the MooseFS free space size doesn't change. Why? .......................................... 82 8.14 When I added a third server as an extra chunkserver, it looked like the system started replicating data to the 3rd server even though the file goal was still set to2............................................ 83 8.15 Is MooseFS 64bit compatible? . 83 8.16 Can I modify the chunk size? . 83 8.17 How do I know if a file has been successfully written to MooseFS? . 83 8.18 What are limits in MooseFS (e.g. file size limit, filesystem size limit, max number of files, that can be stored on the filesystem)? . 84 8.19 Can I set up HTTP basic authentication for the mfscgiserv? . 85 8.20 Can I run a mail server application on MooseFS? Mail server is a very busy application with a large number of small files { will I not lose any files? . 85 8.21 Are there any suggestions for the network, MTU or bandwidth? . 85 8.22 Does MooseFS support supplementary groups? . 85 8.23 Does MooseFS support file locking? . 85 8.24 Is it possible to assign IP addresses to chunk servers via DHCP? . 85 8.25 Some of my chunkservers utilize 90% of space while others only 10%. Why does the rebalancing process take so long? . 86 8.26 I have a Metalogger running { should I make additional backup of the metadata file on the Master Server? . 86 8.27 I think one of my disks is slower / damaged. How should I find it? . 87 4 8.28 How can I find the master server PID? . 87 8.29 Web interface shows there are some copies of chunks with goal 0. What does it mean? . 87 8.30 Is every error message reported by mfsmount a serious problem? . 88 8.31 How do I verify that the MooseFS cluster is online? What happens with mfsmount when the master server goes down? . 88 5 Chapter 1 About MooseFS MooseFS is a fault-tolerant distributed file system. It spreads data over several physical loca- tions (servers), which are visible to user as one resource. For standard file operations MooseFS acts as any other Unix-alike filesystem: • Hierarchical structure (directory tree) • Stores POSIX file attributes (permissions, last access and modification times) • Supports special files (block and character devices, pipes and sockets) • Symbolic links (file names pointing to target files, not necessarily on MooseFS) and hard links (different names of files that refer to the same data on MooseFS) • Access to the file system can be limited based on IP address and/or password Distinctive features of MooseFS are: • High reliability (several copies of the data can.