Reference Architecture
Total Page:16
File Type:pdf, Size:1020Kb
REFERENCE ARCHITECTURE Service Providers Data Center Build a High-Performance Object Storage-as-a-Service Platform with Minio* Storage-as-a-service (STaaS) based on Minio* with Intel® technology simplifies object storage while providing high performance, scalability, and enhanced security features Executive Summary What You’ll Find in This Solution Reference Architecture: This Emerging cloud service providers (CSPs) have an opportunity to build or expand solution provides a starting point for their storage-as-a-service (STaaS) capabilities and tap into one of today’s fastest- developing a storage-as-a-service growing markets. However, CSPs who support this market face a substantial (STaaS) platform based on Minio*. challenge: how to cost effectively store an exponentially growing amount of If you are responsible for: data while exposing the data as a service with high performance, scalability • Investment decisions and and security. business strategy: You’ll learn how Minio-based STaaS can File and block protocols are complex, have legacy architectures that hold back help solve the pressing storage innovation, and are limited in their ability to scale. Object storage, which was challenges facing cloud service born in the cloud, solves these issues with a reduced set of storage APIs that are providers (CSPs) today. accessed over HTTP RESTful services. Hyperscalers have looked into many options • Figuring out how to implement when building the foundation of their cloud storage infrastructure and they all have STaaS and Minio: You’ll adopted object storage as their primary storage service. learn about the architecture components and how they work In this paper, we take a deeper look into an Amazon S3*-compatible object storage together to create a cohesive service architecture—a STaaS platform based on Minio* Object Storage and business solution. optimized for Intel® technology. An object storage solution should handle a broad spectrum of use cases including archival, application data, big data and machine learning. Unlike other object storage solutions that are built for archival-only use cases, the Minio platform is built for CSPs to deliver a high-performance cloud storage alternative to the hyperscalers. The Minio platform offers several benefits: • Hyperscale architecture that enables multi-data center expansion through federation • High performance object store to serve the most demanding cloud-native workloads • Ease of use with non-disruptive upgrades, no tuning knobs and simple support requirements • High availability so objects continue to be available despite multiple disk and node failures • Security-enabled storage by encrypting each object with a unique key Minio provides a compelling STaaS object storage platform when combined with Intel’s broad selection of products and technologies, such as Intel® Solid State Drive Data Center Family for NVM Express* (NVMe*), Intel® Ethernet products and Intel® Xeon® Scalable processors, augmented by Intel® Advanced Vector Extensions Reference Architecture | Build a High-Performance Object Storage-as-a-Service Platform with Minio* 2 Table of Contents 512 (Intel® AVX-512) single instruction multiple data (SIMD) instructions for x86 architecture. A collaborative open source developer community is also available Executive Summary .................1 for Minio. High-Performance Object Storage ..............................2 Using the information and reference architecture (see Figure 1) presented here, Solution Architecture ...............2 you can take the next step toward unleashing the power of STaaS. Minio* Overview and Benefits ........3 Minio Object Storage Architecture ...4 Linear Scaling ........................4 Erasure Code .........................5 Client Client Client Minio — The Perfect Fit for Open Source STaaS .........................5 Better Together: Intel® Technology Accelerates Minio Performance ......6 Network Latest Intel Technology Drives Performance Advantage for Minio ...7 Read and Write Performance Minio* Minio Minio Testing . 7 NVMe* NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe Linear Scalability Testing . 8 Architecture Design Considerations ..9 *NVMe = Non-volatile Memory Express General Guidelines ....................9 Recommended Configuration .........9 Figure 1. Our tests were run on an eight-node Minio* cluster. Obtaining Support and Fixing Bugs ..10 Summary ...........................10 High-Performance Object Storage Appendix A - System Tuning Details ..11 STaaS is the second-fastest growing cloud workload worldwide, representing a Network and Kernel Tuning USD 4.8 billion annual market1. And yet a mere handful of CSPs control the majority Parameters in /etc/sysctl.conf ......11 of that market. With data growing exponentially every year—by 2025, experts 2 Setting IRQ Affinity ..................12 predict that the world will create and replicate 163 zettabytes (ZB) of data —there Solutions Proven by Your Peers ....12 is tremendous potential for emerging CSPs to benefit from the 30-percent annual 3 Learn More. 12 growth of STaaS . Fueling the growth is an increasing focus on big data applications, Internet of Things (IoT) and artificial intelligence (AI) workloads. Object storage is the primary medium for storing big data because it is designed to provide high rates of throughput, offers excellent data integrity, and has a cost-effective deployment model. Although file and block storage solutions are being shoehorned into use with big data workloads, these solutions are limited in their ability to provide what is required. They were designed for enterprise applications like databases and file shares. High-performance object storage has a completely different design goal, which is to provide the extreme rates of throughput required by big data workloads, along with namespaces that span data centers. Neither file nor block storage solutions can perform as fast as, nor scale as large as, object storage. Minio is an object storage solution that provides performance and scalability without suffering from the compromises of file and block storage. By following the methods and design philosophy of hyperscale computing providers, Minio is an object storage solution that provides both high performance and massive scalability. With Minio, which is optimized for Intel® architecture, you can build a fast, reliable STaaS platform with the scalability and flexibility you need to thrive in a data-centric world. Reference Architecture | Build a High-Performance Object Storage-as-a-Service Platform with Minio* 3 Solution Architecture Minio consists of a server, an optional client, and an optional software development kit (SDK): • Minio Server. Minio is a distributed object storage server released under Apache* License v2.0. It is compatible with the Amazon S3 API. Minio is feature-complete, providing enterprise-grade encryption, identity management, access control and data protection capabilities, including erasure code and bitrot protection. • Minio Client. Called mc, the Minio Client is a modern and cloud-native alternative to the familiar UNIX* commands like ls, cat, cp mirror, diff, find and mv. This client provides advanced functionality that is suitable for web-scale object storage deployments. For example, powerful object mirroring tools that synchronize objects between multiple sites and tools for generating shared, time bound links for objects. • Minio SDKs. The Minio Client SDKs provide simple APIs to access any Amazon S3-compatible object storage. Minio repositories on Github offer SDKs for popular development languages such as Golang*, JavaScript*, .Net*, Python* and Java*. Minio* Overview and Benefits Minio is unique in that it was built from the ground up to be simple, fast and highly scalable. With the belief that a complex solution cannot be scalable, a minimalist design philosophy forms the foundation of the Minio architecture design. Minio is designed to deliver multiple benefits to object storage: • Performance. With its focus on high performance, Minio enables enterprises to support multiple use cases with the same simple, scalable platform. Minio performance means you can run multiple Spark*, Presto* and Hive* queries, or quickly test, train and deploy AI algorithms, and not encounter a storage bottleneck. Minio object storage is used as the primary storage for cloud-native applications that require higher throughput and lower latency than traditional object storage can provide. Efficient code underlies Minio’s fast performance. By using Intel® Xeon® Scalable processors in combination with 40 GbE Intel® Ethernet Network Adapters network interface cards (NICs) and Intel SSD Data Center Family for NVMe, Minio delivers highly performant object storage (see “Latest Intel Technology Drives Performance Advantage for Minio” for performance details.) • Scalability. A design philosophy that “simple things scale” means that scaling starts with a single cluster that can be federated with other Minio clusters to create a global namespace—one that can span multiple data centers. Gradual expansion of the namespace is possible by adding more clusters, racks and/or data centers. Minio takes advantage of the hard-won knowledge of the hyperscalers to bring a simple scaling model to object storage. • Ease of use. Minio can be installed and configured within minutes simply by downloading a single binary and then executing it. The amount of configuration options and variations has been kept