Master's Thesis

HIGH AVAILABILITY LINUX

A Degree Thesis Submitted to the Faculty of the Escola Tècnica d'Enginyeria de Telecomunicació de Barcelona Universitat Politècnica de Catalunya by Marius Oltean

In partial fulfilment of the requirements for the degree in TELEMATICS ENGINEERING ENGINEERING

Advisor: José Luis Muñoz Tapia

Barcelona, October 2015 Abstract

Over the last few years the enterprise storage demand have grown exponentially, at a rate of 40 to 60 percent annually, and many companies are doubling their data footprint each year, reaching by the end of 2014 an estimated total digital data of 8,591 exabytes worldwide.

Traditional storage systems based on RAID or LV, and connected to servers via SAN or NAS, have solved storage problems since '80s, but working with large amount of data, these monolithic structures have become unsuitable. For this reason I have decided to investigate deeper these technologies and, using different virtualization tools (VirtualBox, LXC and Docker) in Linux operating system, I have installed and studied one of the possible solutions, that it is set to become the future of storage.

So, worldwide storage demands systems able to work with this large amount of data, be unified, distributed and reliable, with high performance and most importantly, massively scalable up to the exabyte level and beyond. The technology investigated and described in this project is one of the possible solutions to this growing explosion of data worldwide. It's about an open source software-defined (SDS) and distributed storage system called Ceph.

1 Resum

Durant els últims anys la demanda d'emmagatzematge de dades en les empreses ha crescut de manera explonencial, a un ritme de 40 a 60 per cent anual, i moltes empreses estan duplicant cada any la quantitat de dades que gestionen, arribant a la fi de 2014 a un volum estimat de 8,591 exabytes a tot el món. Els sistemes d'emmagatzematge tradicionals basats en RAID i LVM, i connectats als servidors a través de SAN o NAS, han resolt els problemes d'emmagatzematge des dels anys 80, però quan es tracta de treballar amb gran quantitat de dades, aquestes estructures monolítiques són inadequades. Per aquest motiu vaig decidir investigar totes aquestes tecnologies i, utilitzant diferents eines de virtualizació (VirtualBox. LXC i Docker) en el sistema operativo Linux, he instal·lat i estudiat una de les possibles solucions, que està cridada a ser el futur de l'emmagatzematge. Per tant, l'emmagatzematge en tot el món exigeix sistemes capaços de treballar amb aquestes grans quantitats de dades, que siguin unificats, distribuïts i fiables, amb un alt rendiment i, el més important, massivament escalable fins al nivell d'exabytes i més enllà. La tecnologia que he investigat i descrit en aquest projecte és una de les possibles solucions a aquesta creixent explosió de dades en tot el món. Es tracta d'un sistema d'emmagatzematge distribuït, definit per software (SDS) i de codi obert anomenat Ceph.

2 Resumen

Durante los últimos años la demanda de almacenamien de datos en las empresas ha crecido de manera explonencial, a un ritmo de 40 a 60 por ciento anual, y muchas empresas estan duplicando cada año la cantidad de datos que manejan, alcanzando a finales de 2014 un volúmen estimado de 8,591 exabytes en todo el mundo. Los sistemas de almacenamiento tradicionales basados en RAID o LVM, y conectados a los servidores a través de SAN o NAS, han resuelto los problemas de almacenamiento desde los años 80, pero cuando se trata de trabajar con gran cantidad de datos, estas estructuras monolíticas son inadecuadas. Por este motivo decidí investigar más a fondo estas tecnologías y, utilizando diferentes herramientas de virtualización (VirtualBox, LXC y Docker) en el sistema operativo Linux, he instalado y estudiado una de las posibles soluciones, y que está llamada a ser el futuro del almacenamiento. Por lo tanto, el almacenamiento en todo el mundo exige sistemas capaces de trabajar con estas grandes cantidades de datos, que sean unificados, distribuidos y fiables, con un alto rendimiento y, lo más importante, masívamente escalables hasta el nivel exabytes y más allá. La tecnología que investigado y descrito en este proyecto es una de las posibles soluciones a esta creciente explosión de datos en todo el mundo. Se trata de un sistema de almacenamiento distribuido definido por sofware (SDS) y de código abierto llamado Ceph.

3 Revision history and approval record

Revision Date Purpose

0 02/10/2015 Document creation

1 04/10/2015 Document revision

2 10/10/2015 Document revision

3 13/10/2015 Document revision

3 16/10/2015 Document final revision and aproval

DOCUMENT DISTRIBUTION LIST

Name e-mail

Marius Oltean [email protected]

José Luis Muñoz Tapia [email protected]

Written by: Reviewed and approved by:

Date 02/10/2015 Date 16/10/2015

Name Marius Oltean Name Jose Luis Muñoz Tapia

Position Project Author Position Project Supervisor

4 Table of contents

Abstract...... 1 Resum...... 2 Resumen...... 3 Revision history and approval record...... 4 Table of contents...... 5 List of Figures...... 6 List of Tables:...... 7 1.Introduction...... 8 1.1.Work Plan (Gantt and packages and deviations)...... 9 2.State of the art of the technology used or applied in this thesis:...... 12 2.1.Linux Containers...... 12 2.2.LXC Containers...... 1 2.3.Docker...... 4 2.4.Storage Technologies...... 6 2.4.1. Introduction...... 6 3.Methodology / project development:...... 16 4.Results...... 17 5.Budget...... 18 6.Conclusions and future development:...... 19 Bibliography:...... 20 Appendices:...... 2 Glossary...... 3

5 List of Figures

2.1: LXC vs VM 12 2.2: Docker vs VM 17 2.3: Docker architecture. 17

2.4: Raid 10 22

2.5: File vs Block vs Object 24

2.6:Ceph logo 25

2.7: Ceph architecture 26

2.8: Ceph Crush 27

6 List of Tables:

5.1: Project Budget 18 Appendices: Linux Storage Systems 2.1.2: WWN 59

7 1. Introduction

We live in the era of technology that is generating enormous amount of data each second, and with time, this data growth will be incalculable. As data it's the most critical element of any system that exists today, we need to store this ever-growing data such that it remains secure, reliable and, of course, future ready. But can the traditional technologies like RAID or LVM using storage system technologies like File System or Block System and employing storing data over the network methods like DAS, SAN or NAS manage all this amount of data, and above be robust, reliable and, if possible, economic? The answer is no, and we will see the reasons in next chapters. The main objectives of this project are: on one side, analyze the different traditional storage technologies, describing and analyzing the most important concepts involved in its operation; on the other side, perform in depth study of the open source software- defined storage technology called Ceph, explaining the main parts of its architecture and the multiple benefits that provides when managing large amount of data. All of these will be achieved using three different virtualization technologies like VirtualBox, LXC and Docker for simulations, in a Ubuntu 14.04 LTS (Trusty Tahr) release with linux-3.16.0 kernel. As the two main objectives of the project are quite broad issues that lie behind many concepts that until recently were unknown for me, this document is only a brief introduction to the most important concepts of Virtualization tools, Traditional Storage Technologies and Ceph project. So as to make a broader explanation of all concepts, I've decided to create the “Linux Storage Systems” book that you can find in section 6.1 of Appendix. There, I widely explain all about LXC and Docker virtualization tools (VirtualBox is widely known) and secondly, I explain the different traditional storage systems and, of course, the Ceph technology. This project is not a continuation of another project, which has implied a time consuming preliminary study about the existing bibliography on the software storage systems and the virtualization technologies so as to simulate different scenaries. To be able to carry out this project the many skills have been needed, such as the ability to autonomously learn complex new concepts and tools (LXC, Docker, ceph-deploy .etc..) the ability to be able to express learned concepts in a schematic and clear way, and the creativity and knowledge to design and implement the needed sceneries to simulate physical storage disks like in section “2.10.1. Ceph” of “Linux Storage Systems” book in section 6.1 of Appendix. It was also necessary to learn the Bash language and LaTeX in order to develop the documentation. This whole project has been developed by using open source tools over a desktop computer, i7 with 8 GB of RAM memory and running in a Ubuntu 14.04 LTS (Trusty Tahr) release with linux-3.16.0 kernel.

8 1.1. Work Plan (Gantt and packages and deviations) At first this project had a much more ambitious scope. The initial idea was to explain briefly all existing storage technologies so as to introduce the main objective of the project, that was the Linux-HA (High-Availability Linux) project, that provides a high- availability (clustering) solution for Linux, FreeBSD, OpenBSD, Solaris and Mac OS X which promotes reliability, availability, and serviceability (RAS). But as more and more I was getting further into the world of storage technologies, I discovered more interesting concepts, and finally ran into the problem of large amounts of data on the traditional systems. That led me to investigate a possible solution, and finally finding the interesting CEPH project. Also this project has been carried out while doing other professional projects, which has strongly limited the amount of hours available to invest in this project.

Figure 1.1 Project Gannt Diagram

9 Project: Preliminary study and planning WP ref: 1

Major constituent: Project Plan, software setup and Sheet 1 of 4 testing. Short description: Planned start date: 2/2/2015 Project plan elaboration, software installation and Planned end date: 3/4/2015 testing, preliminary virtualization, LXC and Storage Start event: First meeting. overview. End event: Project Plan approval.

Internal task T1: Project plan elaboration Deliverables: Dates: Internal task T2: Software setup Project Plan 25/2/2015 Internal task T3: Preliminary virtualization.

Project: Virtualization Technologies. WP ref: 2 Major constituent: LXC and Docker study. Linux Sheet 2 of 4 Containers chapter elaboration. Short description: Planned start date: In depth LXC study. Containers concept and testing 13/4/2015 examples. In depth Docker investigation and user Planned end date: manual elaboration. 10/6/2015 Start event: Study material gathering. End event: Documentation submission. Internal task T1: LXC Definition and virtualization. Deliverables: Dates: Internal task T2: Docker definition and virtualization. Virtualization 10/6/2015 Internal task T3: LXC and Docker comparison. technologies

Project: Storage Systems WP ref: 3 Major constituent: Storage systems investigation, Sheet 3 of 4 chaper elaboration. Short description: Planned start date: In depth traditional Storage Systems study. Chapter 25/5/2015 elaboration. In depth Hybrid Storage systems, Planned end date: Ceph chapter elaboration. 5/10/2015 Start event: Study material gathering End event: Documentation submission. IInternal task T1: LVM and RAID. Deliverables: Dates: Internal task T2: File-systems. Storage 5/10/2015 Internal task T3: DAS, NAS and SAN. Systems. Internal task T4: Object Storage. Internal task T5: Hybrid Storage.

10 Project: Book elaboration & Project report WP ref: 4 Major constituent: Book & Report Sheet 4 of 4 Short description: Planned start date:13/4/2015 Linux Storage Systems book. Planned end Project report writing. date:15/10/2015 Start event: Project report writing End event: Report Submission. Internal task T1: Project Report writing. Deliverables: Dates: Internal task T2: Linux Storage Systems book. Book & 15/10/2015 report

11 2. State of the art of the technology used or applied in this thesis:

As explained in Introduction section, this document it's a summary of the most important concepts that I explained in the “Linux Storage Systems” introduced in section 6.1 of Appendix, therefore, in case you want more details on any of the technologies outlined below, please consult this documentation. Next we will explain the two different virtualization tools used in our testing scenaries and next, an overview of the different storage technologies and its evolution.

2.1. Linux Containers 2.1.1. Introduction With several functionalities added to the Linux kernel,it has become very easy to isolate Linux processes into their own little environments. Isolation tools allow to build containers, which are a lightweight virtualization technology. While hardware virtualizations or para-virtualizations provide virtual machines, containers are an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a single host. With containers, a single Linux kernel is shared between the host and the virtual machines.

Containers can achieve higher densities of isolated environments than when using virtual machines

Figure 2.1. LXC vs VM

This concept is not new, as it was implemented a few years ago in BSD jails, Solaris Zones and other open-source projects. In Linux, one of the most widely known container technologies has been OpenVZ (OVZ). What OVZ does is, when installed, loads up a specialized kernel that allows us to segment disk space, CPU and memory usage, as well as other features such as bandwidth per container, and locks them in to that specific area of the server. Many engineers from OVZ contributed to the official Linux kernel to build standard isolation tools. These tools are essentially namespaces, cgroups and special network

12 interfaces. These are described below as well as the two main approaches to build containers: LXC (LinuX Containers) and Docker.

2.1.2. Namespaces

The key tool to build LXC containers is kernel namespaces, which is a specific feature of the Linux kernel to isolate applications from each other. Kernel namespaces provide a way to have varying views of the system for different processes. What that means is to have the capability to put applications in isolated environments with separate process lists, network devices, user lists and file-systems. Such a functionality is implemented inside the kernel without the need to run hypervisors or virtualization. The OS needs to keep separate internal structures and make sure they remain isolated. Kernel namespaces are created and manipulated using 3 basic syscalls:

- clone(): which is used to create a new children processes (this low-level function sits behind the well known fork()).

- unshare(): which allows to modify the execution context of a process without spawning a new child process.

- setns() which changes the namespace of the calling process.

Each namespace is materialized by a special file in /proc/$PID/ns. When the last process within a namespace exits, the associated resources: network interfaces, etc. are automatically reclaimed. It is also possible to “enter” a namespace, by attaching a process to an existing namespace. This is generally used to run an arbitrary command within the namespace. Currently, there are six different namespaces:

- process namespace.

- mnt namespace.

- uts namespace

- user namespace.

- net namespace & veth interfaces.

2.1.3. cgroups

Control groups, or “cgroups“, facilitate the management and administration of resources. They consist of a set of mechanisms to measure and limit resource usage for groups of processes. LXC relies on the Linux kernel cgroups, which are a feature to limit, control and isolate resource usage of process groups (CPU, memory, disk I/O,...).

Conceptually, it works a bit like the ulimit shell command or the setrlimit system call; but instead of manipulating the resource limits for a single process, they allow to set them for groups of processes. These groups are hierarchical, beginning with the top group which all processes are located in unless set otherwise. We can then define groups, subgroups, sub-subgroups, etc. to facilitate our limitation needs. Note bandwidth cannot be limited by now with cgroups. Cgroups are not dependent upon namespaces. This means that is possible to build cgroups without namespaces kernel support. next, an overview of the different storage technologies and its evolution.

13 2.2. LXC Containers 2.2.1. Install

The standard tools for complete Linux containers is LXC. Complete containers are like virtual machines with all its relevant environment isolated and being able to execute multiple processes inside.

LXC can be installed as:

2.2.2. Create

To create a container with LXC we just need to run the create command lxc-create.

By default, it will create a minimal Ubuntu install of the same release version and architecture as the local host. If we want, we can create Ubuntu containers of any arbitrary version by passing the release parameter.

For example, to create a Ubuntu 14.10 container:

2.2.3. SELinux

New linux distros apply some security mechanisms that support control security policies for applications. These are called Security-Enhanced Linux (SELinux). The solution to SELinux in Ubuntu is called App Armor. This causes problems to LXC containers. To overcome this issue we have to add the following line to the container’s config file (/var/lib/lxc/mycontainer/config):

2.2.4. Start/Stop

To start the container type the following command:

Once we do this we will be prompted with a login and asking for our username and password. This, by default, is ubuntu for both. In the console of the container, we can shut down it as any other Linux box:

1 We can also shut down the container from the hypervisor:

We can ran the start command with the -d or --daemon option to ”daemonize“ the container:

To connect to a container that has been started in the background, we can use the following command:

We can exit the console of a container without shutting down it by typing ctrl+a then q.

Finally, the command lxc-wait waits for a specific container state before exiting. This command is very useful in scripting. In this example the command will terminate when the specified container reaches the state RUNNING:

2.2.5. Destroy

The command lxc-destroy completely destroys the container and removes it from our system. The required argument is the container name but we can also pass along -f to tell LXC to force a deletion if the container is currently running (default is to abort/error out).

Example:

2.2.8. Run a Command

The command lxc-attach let us run a command in a running container. This command is mainly used when we want to quickly launch an application in an isolated environment or create some scripts.

Example: Configuring the network af a contianer called “webserver”.

2 2.2.9. Limiting Resources

The command lxc-cgroup allows us to set inner workings of a specific container (these can also be set in the container’s ”config“ file as well). The one that most people will probably find helpful is setting memory limits, which can be done like this (300M RAM limit):

2.2.10. Devices

The command lxc-device let us manage devices in running containers. At this point, only the add device action is supported. For example, with the following command we can create a device into a container based on the maching device on the hypervisor. Like this, an USB device will be available on the container:

2.2.11. Auto-start

By default a new container will not start if the host system is rebooted. If we want a container to start at system boot, we can configure that behavior in each container’s configuration file. To do so, add the following lines to config file:

With these parameters, the container will start when the host server boots, then the host system will wait 5 seconds before starting any other containers. We can also set a value for lxc.group parameter to group containers. A container can belong to several groups by using this syntax:

Then, with groups, we can easily shutdown, reboot, kill or list all the containers that belong to a specific group:

2.2.12. Networking

The default upstart configuration in Ubuntu is in the file: /etc/init/lxc-net.conf:

- By default, LxC creates a private network namespace for each container, which includes a layer 2 networking stack.

- LxC also automatically creates a network bridge called lxcbr0 that is started when the host is started.

3 - The bridge interface is NATed to enable the containers to access the outside world with a veth endpoint passed into the container. The NAT is created with iptables with a rule to basically masquerade all the traffic leaving the containers which are bridged with lxcbr0.

- By default, the LXC installation also launches a dnsmasq process that acts as DHCP server for the default linux bridge lxcbr0. This DHCP server uses IPv4 addresses from 10.0.3.2 to 10.0.3.254.

2.3. Docker 2.3.1. Definition

Docker is an open-source project that automates the deployment of applications in containers. Is a ontainer-based software framework for automating deployment of applications. It is a very important tool for building software that run in the backend.

Figure 2.2. Docker vs VM

With Docker we can easily create lightweight, portable and self-sufficient containers from any application with better performance than Virtual Machines. Docker also allows us to deploy and scale more easily, because Docker containers can run almost everywhere: on desktops, physical servers, virtual machines, into data centers, in public or private clouds.

Docker uses a client-server architecture where docker client and daemon communicate via sockets or through a RESTful API. The daemon operates and distributes Docker containers, so does the heavy lifting of building. The Docker client is the primary user interface to Docker, it accepts commands from the user and communicates with the Docker daemon.

Figure 2.3 Docher architecture

4 2.3.2. Docker Components

A Docker project is formed by the following elements:

- Docker Images: are ready-only templates from which Docker containers are launched. They are very similar to default operating-system disk images which are used to run applications on servers or desktop computers. Docker provides a simple way to build new images or update existing images, or we can download Docker images that other people have already created. Docker images are the build component of Docker.

- Docker Registries: are public or private stores from which we upload or download images. Docker Hub is the public Docker registry, and it provides a huge collection of existing images. These images can be created by the user or can be images that others have previously created. Docker registries are the distribution component of Docker.

- Dockerfiles: are scripts containing a successive series of instructions, directions, and commands which must be executed to form a new docker image. The Dockerfiles replace the process of doing everything manually and repeatedly.

- Docker Containers: are similar to a directory, which can be packed like any other, then can be shared and run across various different machines and platforms (hosts). A Docker container holds everything that is needed for an application to run. Each container is created from a Docker image and is an isolated and secure application platform. Docker containers are the run component of Docker.

2.3.3. Installing Docker on Ubuntu

Ubuntu Trusty (14.04 LTS 64-bit) comes with a 3.13.0 Linux kernel, and a docker.io package which installs all itprerequisites from Ubuntu’s repository.

Docker can be installed by executing these commands on any Linux system that supports LXC:

First, we must ensure the list of available packages is up to date before installing anything new:

Next, Install the docker-io package:

Link and fix paths with the following two commands:

Finally, if we want Docker to start when the server boots:

5 Typing docker we can check if Docker was successfully installed.

Docker, like many Linux tools, does not have a graphical user interface. And it is only supported by linux plattforms, but it can also be installed in OSX or Windows by doing some additional steps.

Next, if you want to create a container, you only need to follow the steps described in section 1.5.4 from the “Linux Storage Systems” book, in Appendix section.

2.4. Storage Technologies

2.4.1. Introduction Computer technology evolves rapidly, and this has contributed to the obligatory need to develop devices that can respond to the demand for more storage space, portability and durability, as well as improving the various systems that manage our data. Starting with punch cards, and going through the tapes, magnetic disks (floppy disks) and the various current hard disks, the evolution has always pursued a larger storage capacity and larger access speeds. This evolution has led us from counting our capacity in bits, to have a current technology with almost unlimited space available. Technologies explained in this document provide control and monitoring tools, and visualization techniques for all types of users, but they are more oriented to make life easier for professional administrator rather than people responsible only of their home network. These technologies are some of the most commonly used methods (LVM, RAID, BTRFS) so as to provide flexibility and robustness in all storage systems, which help achieve systems with higher performance and better protected systems. We also explain the different existing architectures for locally or in network datastorage (DAS, NAS, SAN), the advantages or disadvantages that they provide and the storage system technologies that they are based on (File Storage, Block Storage). Finally, we describe evolution tendency of storage systems technologies, explainig some new concepts like Object Storage and Hybrid Storage, and showing an example of some of the new Open Source distributed storage and File System solutions, like Ceph platform. But before proceeding to provide a deeper introduction to these different storage technologies, we make a brief description of some related concepts and technologies, like filesystem, the existing types, and the different techniques that they use. 2.4.2. Important concepts 2.4.2.1 Filesystem A filesystem (or file system) is a hierarchical way that an operating system structures data to store or reatrive it in storage devices. It separates data in individual groups, giving them different names, so that an individual file can be located by describing the path to that file. Information that might describe a file and its contents, such as its owner, who can access the file, and its size, etc. are conveniently stored as metadata in a file system.

6 There are many types of filesystems and the main differences between them are the OS, access speeds, bearing size, CPU usage, and the different functions or techniques to implement. In Windows systems, some of the most common are FAT32, exFAT, NTFS. In Linux, the most used file systems are ext4, F2FS, XFS and, recently, Btrfs. 2.4.2.2 Block Devices Data structures like cylinders, tracks, and sectors, used by traditional disk drives, are not used anymore by the modern systems and subsystems that use disk drives. Cylinder, track, and sector addresses have been replaced by a method called logical block addressing (LBA), which makes disks much easier to work with by presenting a single flat address space. With logical block addressing, the disk drive controller maintains the complete mapping of the location of all tracks, sectors, and blocks in the disk drive. There is no way for an external entity like an operating system or subsystem controller to know which sector its data is being placed in by the disk drive. As there are always going to be bad sectors on any disk drive manufactured, disk manufacturers compensate this by reserving spare sectors for remapping other sectors that go bad. In the disk, files are stored in blocks. A block is a chunk of data, and when appropriate blocks are combined, it creates a file. A block has an address, and the application retrieves a block by making a SCSI (or ATA) call to that address. It is a very microscopic way of controlling storage. How the blocks are combined or accessed is left up to the application. There is no storage-side metadata associated with the block, except for the address, but is not metadata about the block. In other words, the block is simply a chunk of data that has no description, no association and no owner. A block only takes on meaning when the application controlling it combines it with other blocks. 2.4.2.3 Journaling

Updating a filesystem usually requires several separate operations. This makes possible to leave data structures in an invalid intermediate state if there is a system crash between these operations. For example, deleting a file involves two steps: (1) removing its directory entry and (2) marking space for the file and its inode as free. If a crash occurs between steps (1) and (2), there will be an orphaned inode and hence a storage leak. On the other hand, if only step 2 is performed first before the crash, the not-yet- deleted file will be marked free and possibly be overwritten by something else. Detecting and recovering from such inconsistencies normally requires a complete walk of its data structures, for example by a tool such as fsck (the file system checker).

Journaling filesystems are designed to try to avoid disk corruption resulting from system crashes. Journaling is a feature that keeps track of the changes that will be made before committing them to the main file system. These changes are first written to the journal without changing the rest of the filesystem. Once all of those changes have been journaled, a “commit record” is added to the journal to indicate that everything else there is valid. Only after the journal transaction has been committed in this fashion can the kernel do the writes in the filesystem. If there is a system crash in the middle, the information needed to safely finish the job can be found in the journal. For this reason, such file systems are quicker to bring back online and less likely to become corrupted than filesystems without journaling. In general, it is not even necessary to run a filesystem integrity checker after a crash.

7 2.4.2.5 Thin provisioning

Thin Provisioning is a new way of allocating and distributing disk storage resources. Involves using virtualization technology to give the apprerance of having more physical resources that are actually available. Instead of reserving a portion of the disk storage space for each of the users that are connected, a device with "thin provisioning" defines a thin pool inside one of the large volume groups, and define the thin volumes inside that thin pool. So, every user has the space that requires, but we do not reserve anything to anybody, we deliver capacity as each user requires.

By means of “thin provisioning", a common repository of information is defined and for each application will be defined the volumes and size that each application demands, but the actual physical space will be delivered as the record of information progresses. Thus prevents the waste of unused spaces.

The benefits of this technology are multiple, but the most relevant are two. On one side, it could acquire between 13% and 30% less storage, which generates a saving in economic investment. On the other side, and most important, is the recovery of full control of storage by the storage administrators. Through traditional provisioning, the capacity planning are held by the owners of the applications, but with thin provisioning, storage system administrators are those who define storage requirements, using monitoring tools and managing the free space of each application as a common free space.

2.4.3. LVM

Logical Volume Manager (LVM) allows for a layer of abstraction between the operating system and the disks/partitions it uses. In traditional disk management, the operating system looks for what disks are available (/dev/sda, /dev/sdb, etc.) and then looks at what partitions are available on those disks (/dev/sda1, /dev/sda2, etc.). With LVM, disks and partitions can be abstracted to contain multiple disks and partitions into one device. We can create logical partitions that can span across one or more physical hard drives. First, the hard drives are divided into physical volumes, then those physical volumes are combined together to create the volume group and finally the logical volumes are created from volume group.

Figure 2.4 LVM scheme

8 2.4.4. RAID

RAID (redundant array of independent disks) is a storage technology that combines multiple disk drive components (array of disks) into a logical unit. The objective of this system is to protect data against hardware failure by adding redundancy. The general principle is: the data is stored across multiple physical disks instead of just one, giving the system a configurable level of redundancy. Depending on the amount of redundancy, in case of unexpected disk failure, we can reconstruct the data from the remining disks. Data is distributed across the drives in one of several ways called ”RAID levels”, depending on the level of redundancy and performance required. RAID can be implemented by hardware or by software.

Figure 2.5 Example of Raid 10

Note: Visit section 2.3 of “Linux Storage Systems” from Appendix so as to see differences between different Raid levels and how to implement Raid in Linux.

2.4.5. LVM vs RAID

Both RAID and LVM provide distinct advantages when compared to the simple case of a desktop computer with only one hard disk in which usage patterns do not change over time. But between the two systems, which use?

Of course, the most appropriate answer depend on current and anticipated requirements. If the requirements are to protect data against hardware failure, obviously RAID is the answer, as LVM really does not solve this problem. On the other hand, if we need a flexible storage scheme in which volumes are independent of the physical layout of the disks, RAID is not very helpful and LVM is the natural choice.

Then, of course, there’s the really interesting use case, in which the storage system must be resistant to hardware failures and also flexible in the allocation of volumes. Neither LVM or RAID can solve both requirements themselves; no matter, this is the situation in which we will use both at the same time, or rather, one over the other. To do this, first we must ensure redundancy in the data, grouping discs in fewer large RAID arrays and then use these RAID arrays as LVM physical volumes. The strength of this configuration is that when a disk fails, we only need to reconstruct a small amount of RAID arrays, thereby limiting the time used by the administrator to recover.

9 2.4.6. SAN vs NAS

Greater support for applications. Some applications and application deployment scenarios either aren’t certified or simply cannot operate over NFS. Some of the limitations, like the lack of NFS support with Microsoft Exchange Server, will always exist with NFS. Microsoft Cluster Services (MCS) is one example where SCSI pass thru is required to deploy and SCSI-3 reservation support. While this capability is not in the current VVols beta, one should expect that VMware will look to provide this in a future release and in turn retire the need for RDMs.

Enhanced storage bandwidth capabilities. This is an area where SAN has been and will continue to be more capable than NAS. The VVols beta currently does not include NFS v4.1 with pNFS, the latter is required for link aggregation. As such each VVol will be limited to a single link for IO with NFS and multi-links with SAN via VMware’s native or 3rd party multipathing software.

Enhanced storage IO capabilities. Like the previous item, this is an area where SAN has been and will continue to be more capable than NAS. As the IO to an NFS VVol is limited to a single Ethernet link, any congestion cannot be addressed by VMware’s native or 3rd party multipathing software. Link congestion could manifest itself in a storage VMotion operation which will actual add load to the link, exacerbating the problem before it can shift IO access for the VM to another link.

Although a SAN often is faster than a NAS, it also is less flexible. For example, the size of or the filesystem ofNAS usually can be changed on the host system without the client system having to make any changes. With a SAN, because it is seen as a block device like a local disk, it is subject to a lot of the same rules as a local disk. So, if a client is running its /usr filesystem on an iSCSI device, it would have to be taken off-line and modified not just on the server side, but also on the client side. The client would have to grow the filesystem on top of the device.

There are some significant differences between a SAN volume and a local disk. A SAN volume can be shared between computers. Often, this presents all kinds of locking problems, but with an application aware that its volume is shared out to multiple systems, this can be a powerful tool for failover, load balancing or communication. Many filesystems exist that are designed to be shared. GFS from Red Hat and OCFS from Oracle (both GPL) are great examples of the kinds of these filesystems.

2.5. Object Storage

Today we have more and more digital data, and many of them are difficult to control because they are unstructured and displayed in various formats such as emails, documents, images or videos. This growth of unstructured data is a challenge for storage managers as companies require data to be accessible immediately to meet the requirements of their users.

It has been attempted to solve this problem using traditional storage management system like NAS, which manage data as a file hierarchy, or SAN (Block Storage) which manages data as blocks within sectors and tracks, but these are proving to be expensive and inadequate.

An emerging alternative used for example by Quantum LattusCaringo Swarm, EMC Atmos, Hitachi HCP or OpenStack Swift is Object Storage. Also called object-based storage, it is a new storage architecture that manages data as objects. Each object typically includes three things:

10 - The data itself. It can be anything that users want to store, from an image to a instructions manual.

- A variable amount of metadata. This metadata is defined by the creator of the object storage. It contains contextual information about what the data is, its confidentiality, its put-pose and anything else that is relevant to the way in which data must be used.

• A globally unique identifier. Each object has an adress (identifier) in order for the object to be found over a distributed system. This ID serves as a pointer to a specific object, much the same way a universal resource locator (URL) points to a specific file on a web server. This way, users are able to find the data without having to know the physical location of the data.

Object storage can be implemented at multiple levels, including the device level (object storage device), the system level, and the interface level. In each case, object storage seeks to enable capabilities not addressed by other storage architectures, like interfaces that can be directly programmable by the application, a namespace that can span multiple instances of physical hardware, and data management functions like data replication and data distribution at object level granularity.

Figure 2.6 File vs Block vs object

2.5.1 Benefits versus previous technologies

As explained in previous sections file storage is a type of storage that stores data in a hierarchical structure, with directories, sub-directories and files. In file storage, when a file is stored to disk it is split into thousands of pieces, each with its own address. When that file is needed the user enters the server name, directory and file name, the file system finds all of its pieces and reassembles them. It is great and works rightly when the number of files is not very large or when we know exactly where our files are stored, but when we manage large amounts of data, like servers with astronomical number of files, this hierarchical structure has no sense, and folders are just a waste of space and time.

Regarding to block storage, we know that a block is a chunk of data, and when appropriate blocks are combined, it creates a file. With block storage, files are split into evenly sized blocks of data, each with its own address but with no additional information (metadata) to provide more context for what that block of data is.

Object storage, by contrast, doesn’t split the file. The file is stored as a single object complete with metadata, and is assigned an ID and stored as close to contiguously as

11 possible. When a user need a content, he only needs to present the ID to the system and the content will be fetched along with all the metadata, which can include security, authentication, etc. This can happen directly over the Web, eliminating the need for Web servers and load balancers.

In Object storage there is no limit on the type or amount of data and metadata, which makes Object storage powerful and customizable. Because Object storage technologie treats all types of data and can save it as an object and store multiple copies of it, often on different storage servers and drives, it is said that Object storage can replace RAID for data availability.

Note: So as to see all characteristics that Object Storage provides, and also how this technologies scales in comparison with File Storage or Block Storage characteristics, please visit section 2.9.2 from “Linux Storage Systems” book in document Appendix.

2.5. Hybrid Storage (CEPH)

A method combining different storage systems, intended for companies or users who can not or do not want to completely change their traditional storage system. Storage architecture that provides block-based storage, files storage and object storage on the same network.

Currently there are several proprietary solutions for hybrid storage, but in this document we will focus on open source solutions such as the Ceph system, offered by RedHat.

Before we define and explain thoroughly the Ceph storage system, we must understand that, although Ceph can be used as a combination of the three storage systems discussed above, this will not always be the best option to choose, it will always depend on the case in which we want to use it.

2.5.1. CEPH

Figure 2.7 Ceph logo

2.5.10. Introduction

Traditional storage systems based on RAID or LVM and connected to servers via SAN or NAS, work fine in small and medium environments, because in these scenes it’s not possible to reach their inner limits, but because of the incredible growth of managed data, like Big Data or Cloud Computing, in several situations they are becoming unsuitable.

Firstly, when data grow, the traditional storage systems have computing capacity problems, because these systems can only be updated by adding new disks, and only in a few models it can also add new controllers. Also, the capacity growth has a side effect in a reduction in performances, since a single controller has a well-defined maximum computing capacity, and with it it has to manage a larger number of data.

Secondly, with data growth, RAID is becoming an inefficient redundancy technology. These is because, first of all, redundancy is achieved by sacrificing disk space. As an example, if we have a RAID 10 using a 1 TB disks, a stripe of 20 TB is wasting 10 TB,

12 but if we think in more data, for example a storage of 1 PB is wasting 500 TB of space only for redundancy. If we choose a RAID 5 or RAID 6, we reduce the disk waste, but on the other side we are also lowering the performances, because write penalty is higher on these kind of RAID. With increasing of data, we will also be penalized with the time we will need to rebuild data because during these time RAID is running in a degraded state, and this implies additional impacts in performance.

So, worldwide storage demands systems able to work with this large amount of data, be unified, distributed and reliable, with high performance and most importantly, massively scalable up to the exabyte level and beyond. One of the available solutions of the worldwide growing data explosion is an open source distributed storage system called Ceph.

2.5.11 Overview

Figure 2.8 Ceph Architecture

2.5.11.1 Definition

Ceph is an open source distributed storage system with high-performance and scalability, built on top of commodity components, demanding reliability to the software layer. We can say that Ceph is a scale out Software Defined Object Storage built on commodity hardware. Ceph storage software it’s one of the pillars of the OpenStack project, which involved developers around the world working on different components of open source cloud platforms.

Ceph was developed to make possible the standardized storage of Object, Block and Files with a distributed x86 cluster. Ceph has the ability to manage vast amounts of data and delivers extraordinary scalability–thousands of clients accessing petabytes to exabytes of data. A Ceph Node leverages commodity hardware and intelligent daemons, and a Ceph Storage Cluster accommodates large numbers of nodes, which communicate with each other to replicate and redistribute data dynamically.

2.5.11.2 Ceph technical terms

-RADOS:

Ceph Storage Cluster is based on an open source object storage service called Reliable Automatic Distributed Object Store (RADOS). RADOS has the ability to scale to thousands of hardware devices by making use of management software that runs on each of the individual storage nodes. The software provides storage features such as thin provisioning, snapshots

13 and replication. RADOS distributes objects across the storage cluster and replicates objects for fault tolerance.

- CRUSH

The Controlled Replication Under Scalable Hashing algorithm determines how the data is replicated and mapped to the individual nodes. CRUSH ensures that data is evenly distributed across the cluster and that all cluster nodes are able to retrieve data quickly without any centralized bottlenecks.

The algorithm determines how to store and retrieve data by computing data storage locations. CRUSH empowers Ceph clients to communicate with OSDs (explained below) directly rather than through a centralized server or broker. With an algorithmically determined method of storing and retrieving data, Ceph avoids a single point of failure,performance bottleneck, and a physical limit to its scalability.

CRUSH requires a map of our cluster (contain a list of OSDs) , and uses the CRUSH map to pseudo-randomly store and retrieve data in OSDs with a uniform distribution of data across the cluster.

Figure 2.9 CRUSH

- Ceph Node

Any single machine or server in a Ceph System.

- Ceph OSDs

A Ceph OSD Daemon stores data as objects on a storage node, handles data replication, recovery, backfilling, rebalancing, and provides some monitoring information to Ceph Monitors by checking other Ceph OSD Daemons for a heartbeat. So, as Ceph OSDs run the RADOS service, calculate data placement with CRUSH and maintain their own copy of the cluster map they should have a reasonable amount of processing power. A Ceph Storage Cluster requires at least two Ceph OSD Daemons to achieve an active + clean state when the cluster makes two copies of our data (by default).

- Ceph Monitor

Maintains a master copy of the cluster map (cluster state), including the monitor map, the OSD map, the PG map, and the CRUSH map. Ceph maintains a history (called an “epoch”) of each state change in the Ceph Monitors, Ceph OSD Daemons and PGs. They are not CPU intensive.

14 - Ceph Metadata Server

A MDS stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers make it feasible for Portable Operating System Interface (POSIX) file system users to execute basic commands like ls, find, etc. without placing an enormous burden on the Ceph Storage Cluster.

- Placement Group (PG)

Aggregates objects within a pool (logical partitions for storing objects) because tracking object placement and object metadata on a per-object basis is computationally expensive.

- Ceph Cluster Map

The set of maps comprising the monitor map, OSD map, PG map, MDS map and CRUSH map.

2.5.12 Ceph portfolio

Ceph is an enterprise-ready storage system that offers multiple products and gives support to a wide range of protocols and accessibility methods. The unified Ceph storage system supports Block, File, and Object storage, and all these storage systems are deployments of the Ceph flagship product, the Ceph Storage Cluster.

However, Ceph File System (CephFS) currently is under development because lacks a robust ‘fsck’ check and repair function, but Ceph Block Storage and Ceph Object Storage are ready and recommended for production usage.

All Ceph Block, Ceph File and Ceph Object read data from and write data to the Ceph Storage Cluster, so they need to have access to a running Ceph Storage Cluster so as to run properly.

NOTE: So as to learn more about the Ceph project, and see how to deploy a Ceph Storage Cluster yo can visit section 2.10.1 from the “Linux Storage Systems” form Appendix.

15 3. Methodology / project development:

As explained in previous sections this project methodology has 4 very differentiated parts:

1. Linux vitualization technologies: First of all, a long and tedious process of selecting and going through information sources was carried. Reading, studying and comparing the bibliography so as to select the better tool in order to virtualize and try all the examples that you can find in the Appendix section.

2. Scenaries design, coding and study: Once I've selected the correct tools, I started to virtualize and try step by step all technologies and examples that you can find in Linux Storage System book.The scenario is created and configured to behave as expected in order to put the mechanisms to test and observe and analyze it.

3. Storage Systems analyse: Then, when I practiced enough with the virtualization tools, I started to investigate all available Storage systems and how could I try them in Linux. And for that propose I used Linux LXC containers in traditional technologies, and VirtualBox, LXC and Docker when I've needed to simulate the Ceph Storage Cluster creation using ceph-deploy tool.

NOTE: You can find the three examples of Ceph cluster creation in section 2.10.1.6 from the “Linux Storage System” book in Appendix.

4. Linux Storage Systems book redaction: Once I have investigated and simulated almost all cases, and the working sceneries are fully understood and the theory has been confirmed, the theory and scenaries are combined and documented on the textbook using LaTex so as to to create the Linux Storage System documentation.

16 4. Results

The main result of the project is the Linux Storage Systems documentation, which includes: by one hand, a study of two different Linux virtualization technologies, examples of installation and commands output examples; and by the other hand, a study of the traditional Storage technologies, their limitations, and how the new storage systems like Ceph can overcome these limitations. The other important result of the project is the documented example of the Ceph Storage Cluster installation using three different Virtualization technologies, and that can be found in the 2.10.1.6 section of the “Linux Storage Systems” book. I must say that, as Docker and Ceph projects are relatively new technologies, when I tried to combine them mounting a Ceph Storage Cluster using Docker, I found some performance problems that haven't been solved yet by the Docker and Ceph community and which prevented me from mounting a totally healthy Ceph Storage Cluster. These issues are explained in the “Linux Storage Systems” Book.

17 5. Budget

This project has been carried out by using open source tools over a desktop computer of an estimate cost of 1500€ and a Microsoft Office license has been needed in order to fill this document adding 234€ to the total amount. Because of the speed that technology evolves nowadays, an amortization period of 3 year is considered before the obsolescence of the hardware and software used. Eight months of part-time work of a junior engineer is considered, resulting on a wage of 10500 €/year and 853€/months after taxes.

Table 5.1:

Concept Price/month Price/project Desktop Computer 31,60€ 343,25€ Microsoft Office 2013 -- 234€ Salary 853€ 7564€ Total Price 8141,25€ Table 5.1: Project Budget

18 6. Conclusions and future development:

As the present project has given me the opportunity to investigate, study and discover different types of technologies, starting with virtualization technologies, and ending with Ceph, the solution, evolution and the future of storage, I could draw several conclusions: On one hand, the project has allowed me to see that the LXC Linux Containers technology is in a really advanced state of development allowing us to replace almost all solutions given by proprietary products such as VmWare. Regarding Docker containers, I must say that even if is not without its downsides, as I noted trying to implement a Ceph Storage Cluster, I'm sure that in a short period of time it will be fully ready to implement in companies, with a significant benefits. On the other hand, I have seen all the concepts related with traditional storage technologies, I have studied the differences between RAID or LVM, also the way we can store our data over the network with DAS, NAS or SAN, and also which are the differences between the File Storage, Block Storage or the new Object storage technology. From all these I can say that with the explosion of amount of data that today’s companies must manage the traditional storage technologies have become unsuitable. Yes, of course that RAID technology has been the fundamental block for storage systems for many years, but all eras must come to an end, and new ones must begin. As RAID-based storage systems have started to show limitations and are incapable of delivering future storage needs, I think that the future of storage is based on Object storage and, of course, Ceph Storage System. I think that Ceph System is the future of storage because has no fail-over, as it was designed to have no single point of failure in its architecture. It also gives us the possibility to work with blocks, files or objects, and it is able to manage vast amounts of data delivering extraordinary scalability–thousands of clients accessing petabytes to exabytes of data. The last reason, but not the least is that Ceph is an open-source solutions, so it can be more economic than other proprietary solutions, and if we need help, we are not alone, we can consult RedHat.

19 Bibliography:

[1]. “Latex User Guide”. 2015. [Online]. Available: https://latex-project.org/guides.

[2]. "LxC Users Mailing List". [Online]. Available: https://lists.linuxcontainers.org/listinfo.

[3]. Pasztor, J. “LXC tutorial”. 2013. [Online].Available: http://www.janoszen.com/2013/05/14/lxc-tutorial.

[4]. “Use Linux containers to set up virtual networks”. [Online]. Available: http://www.nsnam.org/wiki/HOWTO_Use_Linux_Containers_to_set_up_virtual_networks.

[5]. Rosen, R. “Linux containers and the future cloud”. 2013. [Online]. Available: http://media.wix.com/ugd/295986_d5059f95a78e451db5de3d54f711e45d.pdf.

[6]. Rosen, R. “Resource management Linux kernel Namespaces and cgroups”. 2013. [Online]. Available: http://media.wix.com/ugd/295986_d73d8d6087ed430c34c21f90b0b607fd.pdf.

[7]. “Configuring LXC using libvirt”. 2013. [Online]. Available: http://kaivanov.blogspot.com.es/2013/01/configuring-lxc-using-libvirt.html.

[8]. “Setting up Linux cgroups – Control Groups”. 2012. [Online]. Available: http://kaivanov.blogspot.com.es/2012/07/setting-up-linux-cgroups-control-groups.html.

[9]. Margaret Bierman with Lenz Grimmer. “How I Got Started with the Btrfs File System for Oracle Linux”. 2014. [Online]. Available: http://www.oracle.com/technetwork/articles/servers-storage- admin/gettingstarted-btrfs-1695246.html.

[10]. “How to create a Linux loopback file system with a regular disk file”. 2007. [Online]. Available: http://www.walkernews.net/2007/07/01/create-linux-loopback-file-system-on-disk-file.

[11]. Alex Vaqué "Los principales sustitutos de EXT4 (ZFS, XFS, BRTFS)". 2014. [Online]. Available: http://www.cloudadmins.org/2014/06/los-principales-substitutos-de-ext4-zfs-xfs-brtfs.

[12]. “Understanding Docker”. 2013 [Online]. Available: https://docs.docker.com/introduction/understanding-docker.

[13]. Petazzoni, J. “Virtualization with Linux containers and Dockers”. 2013. [Online]. Available: https://tech.yandex.com/events/yac/2013/talks/14/.

[14]. ”Introducing Docker Online”. 2015 [Online]. Available: http://blog.docker.com/2014/03/

[15]. “Getting Started with Docker”. 2014. [Online]. Available: https://serversforhackers.com/getting-started-with-docker.

[16]. “Data structures on disk drives”. 2015. [Online]. Available: http://www.snia.org/education/storage_networking_primer/stor_devices/data_structure.

[17]. Jonathan Corbet: “Barriers and journaling filesystems”. 2008. [Online]. Available: http://lwn.net/Articles/283161.

20 [18]. Hellman, Brian; Haas, Florian; Reisner, Philipp; Ellenberg, Lars: “The DRBD User’s Guide”. 2008. [Online]. Available: https://drbd.linbit.com/users-guide.

[19]. ”Linux lvm - Logical Volume Manager”. 2015. [Online]. Available: http://linuxconfig.org/linux-lvm- logical-volume-manager#h8-extend-logical-volume.

[20]. “Raid”. 2015. [Online]. Available: http://www.acnc.com/raid.

[21]. “How to Manage Btrfs Storage Pools, Subvolumes And Snapshots on Linux”. 2014. [Online]. Available: http://www.linux.com/learn/tutorials.

[22]. “Setting up NFS. How To”. 2014. [Online]. Available: https://help.ubuntu.com/community/SettingUpNFSHowTo.

[23]. Mesnier, Mike; Gregory R. Ganger; Erik Riedel: “Object-Based Storage”. 2003. [PDF]. IEEE Communications Magazine: doi:10.1109/mcom.2003.1222722.

[24]. Singh, Karan: “Learning Ceph – A practical guide to designing, implementing, and managing your software-defined, massively scalable Ceph storage system”. 2015. [Packt Publishing]. 978-1- 78398-562-3.

[25]. “Red Hat Ceph Storage”. 2015. [Online]. Available: https://www.redhat.com/es/technologies/storage/ceph.

[26]. Inktank Storage. “Welcome to Ceph”. 2014. [Online]. Available: http://docs.ceph.com/docs/master.

1 Appendices:

6.1 Linux Storage Systems

2 Glossary

Ceph Project The aggregate term for the people, software, mission and infrastructure of Ceph. cephx The Ceph authentication protocol. Cephx operates like Kerberos, but it has no single point of failure. Ceph Platform All Ceph software, which includes any piece of code hosted at http://github.com/ceph. Ceph Stack A collection of two or more components of Ceph. Ceph Node Any single machine or server in a Ceph System. RADOS Reliable Autonomic Distributed Object Store. The core set of storage software which stores the user’s data (MON+OSD). Ceph Cluster Map The set of maps comprising the monitor map, OSD map, PG map, MDS map and CRUSH map. Ceph Object Storage The object storage “product”, service or capabilities, which consists essentially of a Ceph Storage Cluster and a Ceph Object Gateway. Ceph Object Gateway (RGW) The S3/Swift gateway component of Ceph. Ceph Block Device (RBD) The block storage component of Ceph. Ceph Block Storage The block storage “product,” service or capabilities when used in conjunction with librbd, a hypervisor such as QEMU or Xen, and a hypervisor abstraction layer such as libvirt. Cloud Stacks Third party cloud provisioning platforms such as OpenStack, CloudStack, OpenNebula, ProxMox, etc. Object Storage Device (OSD) A physical or logical storage unit (e.g., LUN). Sometimes, Ceph users use the term “OSD” to refer to Ceph OSD Daemon, though the proper term is “Ceph OSD”.

3 Ceph OSD Daemon The Ceph OSD software, which interacts with a logical disk (OSD). Sometimes, Ceph users use the term “OSD” to refer to “Ceph OSD Daemon”, though the proper term is “Ceph OSD”. Ceph Monitor The Ceph monitor software. Ceph Metadata Server The Ceph metadata software. Ceph Client The collection of Ceph components which can access a Ceph Storage Cluster. These include the Ceph Object Gateway, the Ceph Block Device, the Ceph Filesystem, and their corresponding libraries, kernel modules, and FUSEs. Ceph Kernel Modules The collection of kernel modules which can be used to interact with the Ceph System (e.g,. ceph.ko, rbd.ko). Ceph Client Libraries The collection of libraries that can be used to interact with components of the Ceph System. Ceph Release Any distinct numbered version of Ceph. Ceph Point Release Any ad-hoc release that includes only bug or security fixes. Ceph Interim Release Versions of Ceph that have not yet been put through quality assurance testing, but may contain new features. Ceph Release Candidate A major version of Ceph that has undergone initial quality assurance testing and is ready for beta testers. Ceph Stable Release A major version of Ceph where all features from the preceding interim releases have been put through quality assurance testing successfully. CRUSH Controlled Replication Under Scalable Hashing. It is the algorithm Ceph uses to compute object storage locations. Pools Pools are logical partitions for storing objects. ...