OpenIO SDS on ARM A practical and cost-effective infrastructure based on SoYouStart dedicated ARM servers.

Copyright © 2017 OpenIO SAS All Rights Reserved. Restriction on Disclosure and Use of Data 30 September 2017 1 OpenIO OpenIO SDS on ARM 30/09/2017

Table of Contents

Introduction 3 Benchmark Description 4 1. Architecture 4 2. Methodology 5 3. Benchmark Tool 5 Results 7 1. 128KB objects 7 Disk and CPU metrics (on 48 nodes) 8 2. 1MB objects 9 Disk and CPU metrics (on 48 nodes) 10 5. 10MB objects 11 Disk and CPU metrics (on 48 nodes) 12 Cluster Scalability 13 Total disks IOps 13 Conclusion 15

2 OpenIO OpenIO SDS on ARM 30/09/2017

Introduction

In this white paper, OpenIO will demonstrate how to use its SDS Object Storage platform with dedicated SoYouStart ARM servers to build a flexible private cloud. This S3-compatible storage infrastructure is ideal for a wide range of uses, offering full control over data, but without the complexity found in other solutions.

OpenIO SDS is a next-generation object storage solution with a modern, lightweight design that associates flexibility, efficiency, and ease of use. It is open source software, and it can be installed on ARM and servers, making it possible to build a hyper scalable storage and compute platform without the risk of lock-in. It offers excellent TCO for the highest and fastest ROI.

Object storage is generally associated with large capacities, and its benefts are usually only visible in large installations. But thanks to characteristics of OpenIO SDS, this next-generation object storage solution can be cost effective even with the smallest of installations. And it can easily grow from a simple setup to a full-sized , depending on users’ storage needs: one node at a time, with a linear increase in performance and capacity.

OpenIO SDS’s simple, lightweight design allows it to be installed on very small nodes. The minimal requirements for a node on ARM are 512MB RAM and 1CPU core, with packages available for Raspbian and Ubuntu (supporting computers). The software’s true flexibility is powered by Conscience technology, a set of algorithms and mechanisms that continuously monitors the cluster, computing quality scores for all nodes, and choosing the best node for each operation. Thanks to Conscience, all operations are dynamically distributed and there is no need to rebalance the cluster when new resources are added.

By partnering with SoYouStart, a dedicated server provider built on OVH’s infrastructure around the world, we have been able to build a cluster and run a complete set of benchmarks to demonstrate the benefts of an ARM-based storage solution that can quickly scale from three nodes to an unlimited number of nodes at a reasonable cost. Thanks to its very competitive offer, SoYouStart enables small and medium-sized organizations to build private infrastructures and making them available on high-speed public networks without the hassle of managing physical hardware purchasing and maintenance.

For this benchmark, the OpenIO team worked on a 48-node confguration to take advantage of erasure coding and parallelism, but the minimal confguration supported in production starts at three nodes, allowing end users with the smallest storage needs to take advantage of object storage at reasonable prices.

SoYouStart offers ARM-based servers in European and North American datacenters, and the confguration described in the following pages is replicable in those countries, allowing end users to be compliant with local laws and regulations. (https://www.soyoustart.com/en/server-storage/)

3 OpenIO OpenIO SDS on ARM 30/09/2017 Benchmark Description

1. Architecture

For our test, we chose to work on a two-tier architecture. This allowed us to install the necessary software to run the benchmark testing suite on two X86 nodes, which also acted as Swift gateways, while the storage layer was built around 48 ARM servers (2 CPU cores, 2GB RAM, 2TB HDD, unlimited traffic, and a public IP address: https://www.soyoustart.com/fr/offres/162armada1.xml).

Each node was confgured to reflect the minimum requirements for OpenIO SDS. FileSystem: • Rootvol (/): EXT4, 8 GB • Datavol for OpenIO SDS data (/var/lib/oio): XFS, 1.9 TB

The object store was confgured using a dynamic data protection policy which enabled erasure coding for objects larger than 250 KB and three-way replication for smaller ones.

OpenIO SDS is easy to deploy and scale thanks to the available Ansible role (available at https:// github.com/open-io/ansible-role-openio-sds) and the OVH/SoYouStart .

4 OpenIO OpenIO SDS on ARM 30/09/2017 2. Methodology

For the benchmark, the platform was populated with 500 containers and 50 objects in each container. Sizes of objects were 128KB, 1MB, or 10MB distributed in equal parts.

A frst type of test (80% read / 20% write), which is close to a real use case scenario, was performed for each object size, and we launched seven different runs based on different levels of parallelism (5, 10, 20, 40, 80, 160, and 320 workers).

A second test was designed to test the linear scalability of the solution. In this case, a 100% read run was launched against 10MB objects on three different cluster confgurations: 12, 24, and 48 nodes.

Each runs lasted 5 minutes (long enough to see any performance issues).

3. Benchmark Tool

We ran the tests using COSbench, a tool developed by Intel to test object storage solutions. It is open source, so results can be easily compared and verifed. In this case, we chose to use the Swift API, but OpenIO SDS is also compatible with the S3 API.

COSbench (https://github.com/intel-cloud/cosbench) features include:

- Easy to use via a web interface or on the command line - Exports signifcant metrics for comparative usage - All metrics are saved in CVS format

5 OpenIO OpenIO SDS on ARM 30/09/2017

The benchmark was organized in five phases: init: container creation prepare: container population with objects main: bench scenario (with all possible read/write/delete combinations) cleanup: object deletion dispose: container deletion

Here is an example of a result page.

6 OpenIO OpenIO SDS on ARM 30/09/2017 Results

1. 128KB objects

Response time (ms) Throughput (op/s) 128KB objects 80% read 800 160

144,59 138,9 139,67 141,23 600 120

98,46 400 80 70,46 Throughput(op/s) Responsetime(ms) 200 40 34,13

0 0 5 10 20 40 80 160 320 Number of workers

Response time (ms) Throughput (op/s) 128KB objects 20% write 1800 40

34,23 34,46 34,82 35,23 1350 30

25,01 900 20 17,94 Throughput(op/s) Responsetime(ms) 450 10 8,72

0 0 5 10 20 40 80 160 320 Number of workers 7 OpenIO OpenIO SDS on ARM 30/09/2017

Disk and CPU metrics (on 48 nodes)

8 OpenIO OpenIO SDS on ARM 30/09/2017 2. 1MB objects

Response time (ms) Throughput (op/s) 1MB objects 80% read 800 140 136,84 137,59 130,81 126,25

600 105

84,06 400 70 Throughput(op/s)

Responsetime(ms) 48,26 200 35 29,49

0 0 5 10 20 40 80 160 320 Number of workers

Response time (ms) Throughput (op/s) 1MB objects 20% write 2400 40

34,43 33 33,68 1800 31,23 30

1200 20,55 20 Throughput(op/s) Responsetime(ms) 600 11,83 10

7,15

0 0 5 10 20 40 80 160 320 Number of workers 9 OpenIO OpenIO SDS on ARM 30/09/2017

Disk and CPU metrics (on 48 nodes)

10 OpenIO OpenIO SDS on ARM 30/09/2017

5. 10MB objects

Response time (ms) Throughput (op/s) 10MB objects 80% read 4000 70 69,68 63,41

3000 52,5 53,08

43,99

2000 35 32,09 Throughput(op/s) Responsetime(ms) 1000 21,17 17,5

11,38

0 0 5 10 20 40 80 160 320 Number of workers

Response time (ms) Throughput (op/s) 10MB objects 20% write 6000 16 15,69 15,89

13,22 4500 12

10,75

3000 8 8,13 Throughput(op/s) Responsetime(ms) 5,14 1500 4

2,95

0 0 5 10 20 40 80 160 320 Number of workers

11 OpenIO OpenIO SDS on ARM 30/09/2017

Disk and CPU metrics (on 48 nodes)

12 OpenIO OpenIO SDS on ARM 30/09/2017 Cluster Scalability

To demonstrate the linear scalability of OpenIO SDS, as well as its ability to scale quickly when needed, we simulated a cluster of 12 nodes as it was expanded to 24, then 48 nodes. We ran three benchmarks using 80 workers confgured to perform 100% read operations on 10MB objects with the three confgurations.

Bandwidth (MB/s) Throughput (op/s) 10MB objects 100% read 1100 1.044,48 100

825 75

553,45 550 99,42 50 Bandwidth(MB/s) Throughput(op/s) 235,15 275 54,05 25

22,96

0 0 12 24 48 Number of nodes

80 workers

12 nodes 2,5 KIOps

24 nodes 5 KIOps

48 nodes 8,7 KIOps Total disks IOps

Each disk delivers between 180 IOps and 210 IOps, which is very good for SATA disks (and is also the limit as we reach 100% disk utilization).

13 OpenIO OpenIO SDS on ARM 30/09/2017

On the 48-node cluster, we confgured the number of workers to increase progressively from 20 to 40, then 80, 160, and 320. After the COSbench preparation phase, we found that the highest bandwidth was achieved with 80 workers. After that, disks reached saturation and performance decreased.

Maximum bandwidth is 1.02 GB/s, with 8.16 Gbps of data delivered to the client application.

14 OpenIO OpenIO SDS on ARM 30/09/2017 Conclusion

The limitation of this infrastructure comes from the SATA drives, with the exception of the frst benchmark run with small 128KB objects.

With OpenIO SDS, SoYouStart ARM servers can achieve the full performance of the single drives they host, without being limited by their ARM CPU. Using the default confguration, cluster performance grows linearly as the number of nodes increases. The more nodes there are in a cluster, the better the performance.

At peak performance, using a 100% read benchmark scenario, and with objects of 10MB in size, we achieved 8.16 Gbps (1GB/s) on the 48-node cluster. This performance is what could be expected from the SATA drives handling both data (sequential IOps) and metadata (random IOps).

The principle of a single-drive ARM server is very appealing, as long as it is able to deliver the full performance of its drive. It reduces the failure domain to its smallest form: one drive. Nowadays, as x86 storage boxes get larger and larger, a failed server can mean that up to 90 disks (900 TB raw) can be taken offline simultaneously. On the contrary, with this type of ARM node, only one drive is affected by a server failure. It also simplifes maintenance as drives are not easy to replace within a large x86 server. This operation requires a lot of care, considering the risk of losing the rest of the server (or switching the wrong drive). In the case of a single-drive server, drives are replaced alongside the rest of the server components. This operation is equivalent to the simple task of adding a drive (and its server) to the cluster, very much like adding additional capacity when needed.

Even though this test was meant to demonstrate OpenIO SDS’s capabilities, it also highlights the fact that SoYouStart ARM servers are inexpensive and can be the base of an interesting hardware infrastructure to build next generation private cloud services. These can allow end users to maintain full control of data while avoiding lock-in, and provide services at a reasonable price. This is a compelling solution, especially for development and testing scenarios.

15 OpenIO OpenIO SDS on ARM 30/09/2017

Next-generation Object Storage and Serverless Computing

openio.io

FR US JP 2 bis avenue Antoine Pinay 180 Sansome Street, FI4 1-35-2 Grains Bldg. #61 Parc d’Activité des 4 Vents San Francisco, 94104 CA Nihonbashi-Kakigara-cho, 59510 Hem Chuo-ku, Tokyo , Japan 103-0014 16