Features Security Lessons: GlusterFS
Secure storage with GlusterFS Bright Idea You can create distributed, replicated, and high-performance storage systems using GlusterFS
and some inexpensive hardware. Kurt explains. By Kurt Seifried
ecently I had a CPU cache mem- ity, or AIC) was not properly addressed, ory error pop up in my file and now you have to deal with a lot of server logfile (Figure 1). I don’t angry users and managers. R know if that means the CPU is failing, or if it got hit by a cosmic ray, or Enter GlusterFS if something else happened, but now I GlusterFS is a network/clustering filesys- wonder: Can I really trust this hardware tem acquired by Red Hat in 2011. In a with critical services anymore? When it nutshell, GlusterFS has a server and cli- comes to servers, the failure of a single ent component. The server is basically component, or even a single system, “dumb” (i.e., the metadata is stored with should not take out an entire service. the file on the back end, simplifying the In other words, I’m doing it wrong by server considerably). You create a relying on a single server to provide my trusted pool of servers, and these servers file-serving needs. Even though I have contain storage “bricks” – basically disk the disks configured in a mirrored RAID space, which you can then export, either array, this won’t help if the CPU goes singularly or in groups (known as vol- flaky and dies; I’ll still have to build a umes) using a variety of replication, dis- new server, and move the drives over, tribution, striping, and combinations and hope that no data was corrupted. thereof to strike the right balance of per- Now imagine that this isn’t my per- formance, reliability, and capabilities for sonal file server but the back-end file the volumes you need. server for your system boot images The GlusterFS data can then be ex- and partitions (because you’re ported in one of three ways to clients, using KVM, RHEV, OpenStack, or using the native GlusterFS client, which something is your best bet for performance and fea- similar). tures like automated failover, NFS (the In this GlusterFS server can emulate NFS), or case, CIFS (using Samba to export the stor- the age). Of course, you could also mount failure of the storage on a server and re-export it a single using other file-serving technologies. server could The GlusterFS server storage bricks are bring a significant just normal devices with a supported file portion of your infra- system. XFS is recommended, and ext4 is structure to its knees. Thus, the avail- supported (more on this later). With no ability aspect of the security triad (i.e., need for special hardware, you could use Availability, Integrity, and Confidential- leftover space on a device to create a par- tition that is then exported as a storage Kurt Seifried brick, by dedicating an entire device or making a RAID device and exporting it. Kurt Seifried is an Information Security Consultant specializing in Linux and net- works since 1996. He often wonders how GlusterFS Installation it is that technology works on a large GlusterFS installation is pretty easy; Fe- scale but often fails on a small scale. dora, Debian, and several others include
62 August 2013 Issue 153 linux-magazine.com | Linuxpromagazine.com Features Security Lessons: GlusterFS
For example, if A note on GlusterFS security: Cur- you have a locally rently, GlusterFS only supports IP/port- I/O-intensive pro- based access controls; in essence, it is cess (e.g., log rota- meant to be used with a trusted network tion with com- environment. A variety of authentica- Figure 1: A CPU hardware error. pression), this tion and encryption options (e.g., Kerbe- could affect the ros and SSL) are being examined; how- GlusterFS server and client by default. performance of glusterd on that server ever, nothing had been decided at the The upstream GlusterFS project also and, in turn, affect all the clients using time of this writing. I suggest you use makes RPMs and DPKG files available the volume that include that storage firewall rules to restrict access to the cli- [1], as well as a NetBSD package. I brick. ents, and if you have clients with differ- strongly recommend running the stable Setup of GlusterFS and volumes is ent security levels (e.g., trusted internal version if possible (3.3.x at the time of quite easy, assuming you have two serv- clients and untrusted external clients), I this writing), although the latest beta ers with one disk each that you want to advise using different storage pools for version (3.4) has some really cool fea- make into a single volume, replicated each. tures, such as Qemu thin provisioning (basically RAID 1, a mirror). From the and Write Once Read Many. first server, run: Other Cool GlusterFS Tricks Because you can add and remove stor- GlusterFS Setup and # gluster peer probe ip.of.server.2 age bricks from a volume, you can add Performance # gluster volume create vol1 replica 2 U entirely new servers and remove existing As with any technology, there are good transport tcp 10.2.3.4:/brick1 U servers. Thus, you can remove dying ways and better ways to set up the serv- 10.3.4.5:/brick2 hardware from your storage pool with ers. As far as filesystems go, you want to # gluster volume start vol1 minimal interruptions. Because the use XFS rather than ext4 if at all possi- back-end data storage is simply a filesys- ble. XFS is far better tested and sup- On the client(s), mount the filesystem: tem like XFS or ext4, and the metadata is ported than ext4 for use with GlusterFS; contained within the files, you can easily for example, in March 2013, a kernel up- # mount ‑t glusterfs ‑o acl U back up your GlusterFS data on the stor- date broke GlusterFS on ext4 [2]. Also, if 10.2.3.4:/vol1 /vol1 age servers. You also can monitor the you’re using XFS, you’ll want to make contents of files easily by, for example, sure the inodes are 512 bytes or larger Replicated volumes are pretty simple: scanning for viruses or rootkits without because if you use ACLs on XFS, you’ll Basically it’s RAID 1, and the number of having to depend on the clients mount- want the inodes to be large enough to bricks should be equal to the replica ing and using the data. store the ACL directly in the inode; if it’s count. Plans have been made to support GlusterFS can also provide data stor- stored externally to the inode, every file more flexible RAID setups (similar in na- age to OpenStack Swift, Glance, and Cin- access will require additional lookups, ture to RAIDZ1/2/3, which allows you to der clients, and it can provide Hadoop thereby adding latency and affecting per- specify that you want data replicated to HDFS storage, among other capabilities. formance. withstand the loss of one, two, or three Because GlusterFS uses a plugin archi- You also need to ensure the system drives in the group). You can also create tecture, in the longer term, it will be pos- running the GlusterFS Server has enough distributed volumes; this is most similar sible for more capabilities to be added RAM. GlusterFS aggressively caches to JBOD (Just a Bunch of Disks), in that and for GlusterFS to be extended – for data, which is good for performance, but files are written in their entirety to one example, by adding file encryption or I have seen cases in which this caused of the bricks in the storage volume. The multitenancy support per volume. the system to run short on memory, in- advantage here is you can easily create The future of storage on Linux is voking the OOM killer and killing the very large storage volumes. bright. Several solutions (e.g., GlusterFS, largest process (in terms of memory A third option is distributed volumes; Ceph) that are under very active devel- use), which was invariably glusterd. these are basically RAID 0-style vol- opment allow you to create distributed, The same goes for clients, which will umes. Here, files are written in pieces to replicated, and high-performance storage need at least 1GB of RAM. I found in each storage brick, and for write and systems with the use of cheap hardware. production that cloud images with read access, you can get very high vol- If you aren’t already using these, I sug- 640MB of RAM would consistently result umes (e.g., 10 servers writing one tenth gest you investigate them. Not having to in the GlusterFS native client being killed of the file each compared with a single worry about disk or server failure bring- by the OOM killer. server handling all of it). Finally, you ing down your storage back end is a As far as storage, I recommend setting can mix and match these three types. For good thing indeed. nnn up one storage brick per device, or ide- example, you could create a distributed ally using RAID 5/6/10 devices for stor- striped volume for extremely high per- Info age bricks, so a single failed disk doesn’t formance, a distributed replicated vol- [1] GlusterFS download: take out the entire brick on that server. If ume for extremely high availability, or a http://www. gluster. org/ download/ possible, you should not share devices distributed striped replicated volume for [2] A kernel change breaks GlusterFS: between GlusterFS and the local system. both performance and reliability. http://lwn. net/ Articles/ 544298/
linux-magazine.com | Linuxpromagazine.com Issue 153 August 2013 63