white paper

Building a Flexible OpenStack* Cloud from the Ground Up

Contributors This paper describes the process of designing and building a flexible Dr . Yih Leong Sun OpenStack* environment that takes advantage of Intel® platform features Senior Software Cloud Architect, and supports open-ended customer requirements. It outlines steps taken at Intel OTC the Intel Open Source Technology Center to create a lab cloud that supports Michael Kadera OpenStack development. Cloud and Data Center Manager, Intel OTC A well-designed OpenStack environment must not only meet the present needs of the developers and end users it supports but also offer the flexibility John Geier and scalability to satisfy future requirements as broadly as possible. It follows Cloud & Data Center Engineer, that, while balancing those factors with budget limitations and other practical Intel OTC constraints is vital, the common practice of starting the design process with a Table of Contents budget discussion is not ideal. Defining Requirements and Instead, it is a best practice to begin by identifying what the cloud needs to do and Constraints for the Architecture. . . 1 creating a preliminary design to meet those needs. Once that work is complete, Planning for Growth of the practical limitations can be thoroughly identified and accommodated as the design Environment...... 3 is refined and finalized. This focus helps preserve meeting the present and future Entry Stage ...... 3 capabilities needed from the environment as the core priority of the process. Intermediate Stage...... 3 Enterprise Stage ...... 3 Establishing the A well-designed, flexible cloud provides long-term support for end users and Rack-Level Design...... 4 their applications as their needs evolve. Just as the business it supports is Specifying Requirements dynamic and ever-shifting, an OpenStack environment must be able to adapt against Available Resources. . . . 4 to the need for ongoing change. Per-Rack Power Usage. . . . . 4 Rack Density...... 4 Refinements to the Rack-Level Defining Requirements and Constraints for the Architecture Design...... 4 Identifying core design requirements begins by establishing what users will need Compute Resources ...... 5 to accomplish using the OpenStack environment being built. Some examples of Management Resources. . . . 5 the types of capabilities to be supported by the compute, storage, and network Networking Resources. . . . . 5 infrastructure are illustrated in Figure 1. At this stage, designers must consider Providing a Flexible beyond the known applications to be hosted on the system, including factors such Lab Topology ...... 6 as runtimes, data access, and services for identity and asset management. As usage Compute Flexibility...... 6 scenarios are outlined, interlocking sets of requirements are typically revealed. Storage Flexibility...... 6 For example, one aspect of service monitoring relates to charging for usage, Network Flexibility...... 6 if applicable, including chargeback to other business units within a company Deployment Mechanisms...... 6 or billing to external users. Depending on the details of these arrangements, they may have implications with regard to Sarbanes-Oxley or other regulatory Design Best Practices for Flexible OpenStack Clouds...... 6 frameworks. Data access, export restrictions, and privacy regulations may also Compute Best Practices...... 6 Storage Best Practices...... 7 Network Best Practices...... 7 Deployment and Management Best Practices. . . . 7 Conclusion...... 7 Building a Flexible OpenStack* Cloud from the Ground Up 2

impact multi-tenancy strategies, resulting in the logical or physical partitioning of users and data to remain compliant. Compliance efforts, in turn, will impact mechanisms for how users, applications, and systems access specific resources and how rights that govern that access are established and maintained. Additional aspects of gathering requirements Desktops Cloud Servers Smartphones include the following: • Devices and external systems to be supported, including clients such as PCs, smartphones, and endpoints on the Internet of Things, as well as cloud-based resources and enterprise back-end systems. Tablets IoT Devices Laptops • Data and infrastructure interoperability, including functionality related to analytics and interactions between public and internal big data resources. Devices • Orchestration of resources and operational management, maintaining optimal functioning of both physical and virtual resources, including facilitating end-user support for break- fix and adds, moves, and changes. Making this process more complex, each of these factors must take into account not only current needs but also Monitoring Content Finance Communications projections for the future. In the case of the lab cloud at the Intel Open Source Technology Center, the purpose was to support internal developers and architects globally as they write, test, and Asset push OpenStack code upstream to the community at large. Identity Data Runtime Management In this sense, Intel’s production environment can be a downstream consumer of the work done in this lab and by Capabilities other contributors. The corresponding goals of this work are to enhance the growth and stability of OpenStack for the enterprise, making the use of cloud resources as simple and effective as possible. The lab-cloud development effort also includes work to enable enterprise capabilities such as rolling upgrades, bare metal provisioning, and service plane high availability Compute Storage Network (HA), as well as Intel® platform capabilities such as trusted compute pools, Intel® AES New Instructions, Intel® Advanced Infrastructure Vector Extensions 2, and single-root I/O virtualization. In addition, design projects such as the Intel® Power Thermal Figure 1 . Example requirements for the architecture. Aware Solution help foster efficient management strategies for the data center based on the latest platforms. The lab environment also supports enablement for cloud-related technologies such as the Data Plane Development Kit (DPDK), Open vSwitch*, and OpenDaylight*. This effort is part of Intel’s ongoing commitment to expanding its participation in OpenStack and related projects. Because the user base for this lab environment is global, the ability to make changes to the environment’s configuration remotely on demand is critical. To avoid the need for physically moving cables and equipment to support ongoing project-specific changes, the team established that capabilities such as KVM/PDU over IP, Intelligent Platform Management Interface (IPMI), managed switches, and software-defined infrastructure must play a significant role in the solution. Building a Flexible OpenStack* Cloud from the Ground Up 3

Planning for Growth of the Environment The design of an OpenStack environment must be based on a scalable rack design that allows both the cloud’s capacity and its corresponding features and capabilities to grow over time. This progression also corresponds to the reality of budget limitations, which may significantly limit what an organization is able to build in the early stages of an OpenStack deployment. Nevertheless, fiscal soundness requires that the initial design consider the long-term lifespan of the environment, which may be considered in a three-stage model, as illustrated in Figure 2. Enterprise Intermediate Entry

Multiple Availability Zones Scale Single Single Availability Zone Availability Zone

Minimal Dual Rolling Upgrades Provisioning Availability Zones

Single Availability Zone High Availability Rolling Upgrades Enhanced Scalability

Core Functionality Operational High Availability Flexible Software-Defined Netowork and Storage Figure 2 . Growth phases of the solution.

Entry Stage Testing of operational HA allows for improving the efficiency of patching and other updates to the system by minimizing A rudimentary deployment of OpenStack at the entry the delay associated with moving virtual machines (VMs) stage includes initial provisioning and getting core cloud around when taking physical nodes offline. Optimizing this functionality in place, typically in a single rack comprising a functionality before the environment becomes too large can single availability zone. One could consider this a “minimum generate significant savings later. viable cloud,” meaning the minimal set of capabilities required to deliver basic cloud capabilities while remaining fully extensible and scalable for long-term evolution and Enterprise Stage growth. The physical infrastructure of this relatively simple As the OpenStack environment continues to grow, it implementation serves as a building block that can be may encompass a large number of racks, each typically used as a template for additional racks as they are added patterned after the design established in the entry stage. later. To support user needs, failover testing for HA may be The environment may become significantly more complex undertaken at this stage. at this stage, extending to multiple availability zones or regions. Availability zones can be used to divide a compute Intermediate Stage deployment into multiple logical groups that provide redundancy at the power supply or network layer; regions An expansion of the initial OpenStack rollout uses multiple provide a more discrete separation across multiple sites. racks and establishes the scalability of services the Refinements to support for rolling upgrades play a significant environment offers. This may include the fundamental use role in ensuring that the data plane and control plane remain of a secondary availability zone, including the intentional available at all times. introduction of network latency as part of application testing. Initial testing of OpenStack rolling upgrades may take A significant and ongoing challenge at the enterprise place at this stage within a single availability zone, allowing stage is to scale the environment’s capabilities to push administrators to upgrade each component, piece by piece, the limits of OpenStack services with massive data sets in support of the goal of migrating to the most up-to-date and extensive groups of users. As the scope of workloads version of OpenStack with minimal downtime. and other requirements becomes more extensive and complex, software-defined networking and storage help the environment gracefully handle the need for change while taking optimal advantage of physical data center resources. Building a Flexible OpenStack* Cloud from the Ground Up 4

Establishing the Rack-Level Design spikes), the team used Prime95* for Linux*, a stress-testing tool designed to place load on all a system’s processor cores The preliminary physical rack-level design for the OpenStack and memory with heavy use of both integer and floating- lab cloud was based on an ideal case and did not account point instructions. A custom script was used to read and write for constraints such as those associated with a specific quasi-random data to the local drives. physical facility or budget. It included 30 servers per rack, with one full-rack battery backup unit (BBU) sufficient to Rack Density support all of the components in two server racks. This design enabled high rack density and a small overall physical In addition to power availability, the other primary factors footprint. Management infrastructure included KVM and PDU that determine how much equipment can be placed in each capabilities that could be carried out over IP, as well as IPMI, rack are the cooling and weight capacity of the physical to allow administration from anywhere in the world. facility. Cooling requirements are largely a factor of power consumption, and sufficient cooling capacity is necessary Specifying Requirements against Available in order to utilize the full power budget, regardless of the Resources amperage available in the facility. Refinement of the preliminary design considered factors Weight is especially a factor in the case of a raised-floor that ranged from power and cooling capacity to the environment (which was used for this project). The full-rack weight capacity of the raised-floor room where the racks BBUs in particular weighed approximately 2,000 pounds would be housed and the available budget. To facilitate each, far exceeding the 1,000-pound capacity for the these calculations and measurements, a spreadsheet corresponding footprint on the raised floor. Networking was developed to document the cost, electrical power equipment also contributes to resource requirements, requirements, and weight of all components that could particularly in terms of power draw and cost, which may potentially be used in the solution. These included the entire help dictate what network capabilities are available for bill of materials. the solution. Some of the calculations around rack density require a relatively specialized skill set, so the team consulted Per-Rack Power Usage with a facilities engineer. To gauge the ability of available power sources to support a Refinements to the Rack-Level Design given hardware configuration, it is necessary to consider all components of the solution, including supporting hardware Based on the findings of the refinement process described such as local disk drives. The design must consider both above, the team found that the power, cooling, and weight power consumption while all hardware is operating at constraints required the number of systems per rack to be maximum capacity and usage spikes when equipment is decreased from 30 to 15. In addition, the weight of BBUs starting up. Further, the total consumption budget must led to the limitation that only network and management include an extra margin to allow for changes to the system resources (that is, switches and KVM devices) in the solution configurations over time. Considering all those factors, power are supported. Cost efficiency called for the use of BBUs usage may be determined by either of two methods: with capacity sufficient to handle three racks per unit. The high-level hardware configuration of racks in the solution is • Calculated , using specifications published by the illustrated in Figure 3. equipment manufacturer. 1x 48-port 10GbE switch • Measured , based on samples of the equipment similar to (SFP+, managed) that intended to be used in the final design. 2x 24-port 1GbE switches Either of these approaches requires consideration of all (unmanaged) system components, including options such as various 5x 1U servers backplanes and add-in cards, as well as the inclusion of (8 drive bays each) switches and other ancillary parts of the environment. In determining the power requirements for this project, the team elected to use a combination of these methods to enable a more comprehensive approach. For example, the team was unable to test some parts of the environment 10x 2U servers such as switches and KVMs at full load without having final (26 drive bays each, hardware configurations or full quantities of equipment enterprise RAID) available. To overcome that limitation, the team relied on manufacturer specifications. On the other hand, both Distribution Units (2x per rack) Smart Power methods were used to determine power consumption of the servers. The facility available to house the lab systems provided Battery Backup Unit (1x per three racks) a power budget of 60 amps per rack, which was a key KVM console (1x per rack) consideration in terms of how much equipment could be used per rack. To test power draw (including during usage KVM over IP (1x per rack)

Figure 3 . Hardware components in each rack. Building a Flexible OpenStack* Cloud from the Ground Up 5

Compute Resources two network-controllable power distribution units (PDUs), which support power cycling and power metering, on a per- The initial design includes five 1U servers and ten 2U servers node, per-PDU-circuit, and total-PDU basis. IPMI capability per rack, based primarily on cost and weight considerations. is also included to provide advanced autonomous server The 1U servers are used for compute and light storage, while management and monitoring. the extra drive bays in the 2U servers provide flexibility for increased storage capacity in addition to compute, with the ability to add more drives in the future without affecting the Networking Resources rack design. This increase in drives was also included in the The solution’s primary networking infrastructure is based on power-consumption budget. The 1U systems were calculated one 48-port managed 10GbE switch per rack, which provides with 8 bays full, and the 2U systems were calculated with 26 capacity for three connections to each server. Because of the bays full. The 2 TB SATA drive power requirements were used reduction from the preliminary design to the final one of 30 for all drive-expansion calculations. servers per rack to just 15, less switch capacity is needed, which allowed the team to buy a more sophisticated switch The smaller physical size of the 1U servers also leaves space than might otherwise be possible. in each rack for potentially adding equipment to the solution in the future. For example, this equipment might include As a result, the solution has a highly flexible set of adding 4U servers based on the Intel® Xeon® processor E7 capabilities for VLANs, allowing one each for public (lab product family or additional networking capacity. network), management, and storage and backup. The numbers of network ports on each server could also be Each server is based on two Intel® Xeon® processors E5-2699 increased dramatically, if desired, such as by using four-port v3, with a clock speed of 2.3 GHz, 45 MB cache, and 18 cores converged network adapters. By increasing switch capacity per socket. Other main components include 128 GB of DDR4 as well, many additional use cases could be explored. For RAM, four 10-Gigabit Ethernet (10GbE) ports and two Gigabit example, NIC bonding on each subnet could provide higher Ethernet (1GbE) ports, a combination of solid-state drives throughput and redundancy for mission-critical usages, while (SSDs) and SATA-based hard disk drives (HDDs), a trusted multiple 10GbE switches could provide redundancy in case platform module, and a remote management module. While one of the switches failed. only three of the 10GbE ports per node are currently in use, cost efficiencies made it a sound decision to buy two-port In addition to the 10GbE switch, each rack is outfitted with network adapters, providing additional future capacity. two 1GbE switches that support management and related functionality. The first of these provides a private network for Management Resources the KVM, PDUs, and BBUs. The second 1GbE switch is used for the baseboard management controller (BMC) services and The solution’s management infrastructure includes a 16- out-of-band connections. port KVM to support the 15 servers with remote BIOS access and remote boot media. Each rack is also supplied with

10GbE

Lab Network Management Storage (Optional) VLAN VLAN VLAN Use-Case VLAN

1GbE 1GbE

BMC KVM/PDU/BBU Network Network

Figure 4 . Networks included in the solution design. Building a Flexible OpenStack* Cloud from the Ground Up 6

Providing a Flexible Lab Topology Deployment Mechanisms The lab design offers the flexibility for supporting multiple There are multiple options available for provisioning and development configurations, which can have a compute, deployment within the environment. One approach is to storage, or network focus. The ability to remotely provision conduct automated provisioning of physical hosts with and configure the environment according to the needs of a Bifrost and Ironic, with automated deployment of OpenStack given project contributes to this flexibility. services using Ansible. The initial design uses Linux Bridge as the underlying networking plug-in agent for neutron, Compute Flexibility and network bonding is also configured. Using other tools such as Kolla, Fuel, or RDO Manager is also possible; the The use of standardized servers helps create a homogeneous flexible network design of the environment supports testing environment where any node can stand in for any other one, various setup changes to improve the robustness of these allowing repeatability by groups running tests on different approaches. hardware within the lab. This consideration is behind the approach of configuring both the 1U and the 2U servers with the same processors, memory, storage, and networking Design Best Practices for Flexible OpenStack hardware as well. Clouds Flexibility must be a core design precept from the beginning Significant flexibility is provided by the fact that the 15 nodes of a successful OpenStack implementation, rather than in each rack can be provisioned as needed for various usage a concept to be observed after the fact. Insight early in scenarios. Specifically, three nodes are typically set aside for the process about how customers will want to use the use as controllers, and the remaining 12 servers can be set cloud can go a long way toward defining requirements for up for any combination of compute, storage, and networking the infrastructure. Based on that information, the design resources desired and for infrastructure support such as should build outward from an understanding of OpenStack network booting, bastion services, and provisioning. architecture and dependent services. Storage Flexibility Gathering complete information about the constraints to be addressed by the design is also critical. Areas to As mentioned above, significant space, power, and cooling be considered include physical space available for the is available on the systems and in the rack to expand local construction, operation, and growth of the environment, storage as emerging needs dictate, with as many as 26 power and cooling resources, weight capacity, budget drive bays included in the 2U servers, of which only six are limitations, and requirements for security and compliance. currently in use. Adding more hot-swappable HDDs or SSDs can therefore easily and inexpensively provide additional As a related matter, implementation teams must identify local storage at scale, without affecting the core rack design. how they will provision, service, and maintain the physical hardware in the solution. Patching, upgrades, disaster The storage back-end for the environment is currently based recovery, and business continuity must all be addressed. This on Nova, Glance, and Cinder, with support for hardware RAID planning will also include determining the actual location and logical volume management (LVM). Additional storage of resources, the people who will have physical and remote functionality is also under consideration to support various access to them, and what aspects of the solution lend storage options that developers may want to use. The team is themselves to automation. currently working on integrating Ceph* block storage into the environment, and future projects will address Swift object storage and Manila file-share-management. Compute Best Practices Testing multiple virtualization technologies (for example, Network Flexibility KVM, VMware, Microsoft HyperV*, or Xen*) may reveal advantages in terms of the capabilities supported, as well The robustness of the 10GbE top-of-rack switches and as performance and reliability in various usage scenarios. Intel® Ethernet Converged Network Adapters X520 allows Present and future requirements should be identified and developers to make use of advanced networking capabilities. documented as part of this design decision; for example, • Robust VLAN support allows a single rack or a larger challenges may exist with live migration in OpenStack environment that includes multiple racks to be segmented environments using KVM. In addition, comparison of local as needed to allow separate experiments to proceed in versus shared storage for VM ephemeral disks may reveal parallel at the same time. significant operational advantages depending on specific workloads. • Bonded network interfaces provide options for increasing uplink speed (up to 40 Gb) and/or reliability, depending on Depending on the needs and characteristics of the workload, the bonding architecture used. it may be valuable to fine-tune host over-provisioning. OpenStack’s default overcommit ratios are 16:1 for CPU and • Jumbo frames increase network performance by allowing a 1.5:1 for RAM, meaning for example that Nova schedules 16 larger data payload to be included in each network packet. virtual cores for each physical core. Over-commitment of compute resources can allow a greater number of instances to coexist on a given cloud topology, although each of those instances will achieve lower performance because of the smaller allocation of resources per instance. Building a Flexible OpenStack* Cloud from the Ground Up 7

Storage Best Practices Conclusion Choosing server hardware with ample physical slots for Rather than following the common practice of starting HDDs or SSDs allows for flexibility in the deployment of with a project budget and building outward from that local storage for the solution. In addition, it is worthwhile point, the approach followed here relies on first gathering to consider the fact that Glance can support a number of user requirements and then developing a preliminary back-end storage types (for example, LVM, NFS, Ceph, and design. Once that first design is in place, it is refined as Cinder), adding further to the potential flexibility of storage needed, according to constraints in areas such as power, on OpenStack clouds. cooling, weight, and budget. This prioritization helps keep capabilities, rather than constraints, at the center of design Network Best Practices considerations. Compute, storage, and networking resources are designed at a per-rack level, providing a building block Examining the options for network topology is an important that can be repeated as the solution environment scales out. aspect of OpenStack design. The standard topology includes public, management, and storage networks; including Building from a foundation based on user requirements, separate networks for deployment or API access may also be Intel Open Source Technology Center has designed and desirable. built an OpenStack cloud with the ability to scale to 100 or more nodes and to flexibly support emerging requirements When confronting the complexity of networking with in the years to come. While this environment is specifically OpenStack, it is worth considering the use of provider- geared toward the development of OpenStack features as network versus tenant-network models. Specifically, provider part of Intel’s commitment to the technology, it provides a networks are provisioned by an OpenStack administrator (as methodology that can help other organizations pattern their it requires changes to physical network infrastructure) and own efforts. can be shared among tenants, while tenant networks are provisioned by the tenants. Deployment and Management Best Practices Learn more: Automation is a critical aspect of streamlining the day-to- day operation of OpenStack environments, particularly with intel com/openstack. regard to making the ongoing changes that underlie flexibility of the overall environment. It is worthwhile to invest time and effort to research and test tools for common activities. • Operating System deployment, network configuration, and disk configuration can be automated using tools that include Preboot eXecution Environment (PXE) and IPMI. • OpenStack services deployment can be automated by using tools that include Ansible and Kolla. • Management of OpenStack users, projects, and subnets can be automated using approaches such as Ansible scripts. Building a Flexible OpenStack* Cloud from the Ground Up

Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at intel.com. No computer system can be absolutely secure. Copyright © 2016 Intel Corporation. All rights reserved. Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries. *Other names may be trademarks of their respective owners. 0816/NKH/MESH/PDF 334626-001US