Cloud Computing Series: Products to Build A

Systems Engineering at MITRE CLOUD COMPUTING SERIES

Products to Build a Private Cloud

Lawrence Pizette Geoffrey Raines July 2010 Executive Summary “With today’s virtual infrastructure solutions and a Federal information technology (IT) leaders who are growing list of internal cloud platform technologies, seeking to leverage cloud computing concepts while it’s fairly easy for an enterprise to start up a cloud- maintaining full control and ownership of their IT like environment within its own domain.” capability may elect to implement a private cloud computing environment. A private cloud capability – James Staten et al., Deliver Cloud Benefits Inside Your Walls, Forrester Research, April 13, 2009 can offer increased security and provide the flexibility to implement unique requirements along with the benefits and attributes of ownership, including that they list are on-demand self-service, broad net- data rights and control over the system. Some of work access, resource pooling (e.g., multi-tenancy), the benefits of using a private cloud approach over rapid elasticity (e.g., rapid scaling), and measured traditional IT are: it allows for better utilization of service. As aspects of these characteristics are pres- hardware, the ability to easily provide continuity of ent in other IT approaches, it is the combination that operations (COOP), the ability to quickly provision describes cloud computing. capabilities, and the ability to provide location independent access. Organizations that pursue develop- Categories of Infrastructure for a Private Cloud— ment of a private cloud infrastructure are investing The product categories listed below were selected their own capital with commensurate development based upon a generalization of the private cloud schedules, risks, and operating costs. They are trad- computing marketplace and available technologies. ing off the attributes of a private cloud for a differ- However, industry innovation is not constrained to ent set of benefits and risks that they might attain if neat categories; some vendors are offering suites of they chose a different cloud computing deployment capabilities that span multiple categories. model. In order to help these leaders to understand the private cloud computing products available in • Virtualization technology provides the hypervi- early 2010 and how they may fit into their data cen- sor software, which allows multiple instances of ters, this paper surveys the private cloud computing “guest” operating systems to run concurrently on market. the same physical server. This capability enables “multi-tenancy,” which is the sharing of physical While the benefits of a private cloud may be numer- resources providing for better server utilization. ous, shared community or public cloud offerings The improved server utilization can reduce heat- may be available sooner at lower costs, but will likely ing, ventilation, and air conditioning (HVAC) offer a different set of attributes for mission assur- and electric costs, data center, size and other ance and other important characteristics. related infrastructure costs. It also can allow advanced users or data center operators to provi- Attributes of Cloud—While there is no single sion capabilities quickly, facilitating scalability, agreed upon definition for cloud computing, the self-service provisioning, and COOP. National Institute for Standards and Technology • Storage technology, such as disk arrays, storage (NIST) has listed several essential characteristics of a area networks (SANs), and storage connection private cloud approach.1 The essential characteristics

Executive Summary—Products to Build a Private Cloud i technologies can provide the underlying persis- employed that considers the Federal IT leader’s tent storage for a private cloud. The market for high-level objectives and specific requirements. these types of systems is huge: In Q4 2009, the While some private clouds may be driven by worldwide external disk systems market was the desire to reduce costs, others may be driven $5.288 billion with the top five vendors, EMC, primarily by new capabilities that can be pro- IBM, HP, NetApp, and Dell taking 69 percent of vided. Additionally, there can be many specific the market share.2 Regardless of which vendor requirements in a private cloud such as perfor- provides it, storage is essential to a private cloud mance, self-service provisioning, COOP, location as it can enable burst capacity and provide capa- independent access, and robust security, which bilities to facilitate COOP and location indepen- need to be considered carefully before acquiring dent access. specific hardware and software products. • Security capabilities for a private cloud are • Security should be considered from the begin- essential. Products offered to the private cloud ning of the structured systems engineering computing market provide capabilities such process. Security architects should perform an as identity management, logging and audit- analysis and consider a pilot to understand new ing, anti-malware software, intrusion detection threats and mitigating strategies. The analysis systems (IDS) and intrusion prevention systems should consider how private cloud capabilities (IPS), and firewall capabilities between virtual would be integrated into the rest of the data machines. While there are many products on center’s environment (e.g., identity management the market, addressing security concerns within systems). multi-tenant, cloud computing environments is • A private cloud will require updated management an active area for research for many organiza- and governance. For example, if virtual environ- tions. Therefore, products will continue to evolve ments are not managed, the absence of controls as more knowledge is learned about threats and coupled with the relative ease of creating virtual measures to mitigate them. machines can result in a proliferation of environ- • Provisioning, management, and metering tools ments and a condition informally known as “VM are key to providing Federal IT leaders and users sprawl.” This condition can offset anticipated with many of the advantages a private cloud can efficiency gains and allow for security exposures offer: self-service provisioning, burst capability, due to unmaintained environments. In order to service-level agreement (SLA) compliance, and, if mitigate the risk, piloting and testing of the inte- needed, metering for “pay as you go” functional- grated environment should be considered as part ity. Tools in this category provide the ability to of the overall systems engineering plan. monitor server utilization, system resources, and A private cloud implementation can offer signifi- other operational attributes. The ability to provi- cant benefit to Federal organizations seeking to sion environments on-demand, independent of realize some of the benefits of cloud computing the underlying hardware enables the ability to while maximizing control over their environment. scale environments in near real time and, if nec- Recognizing that the benefits coupled with control essary, to instantiate environments in a different can be compelling for many Government organi- location. zations, industry has provided a wide selection of Considerations— products for realizing the vision. This paper pro- • When developing a private cloud, a compre- vides insight into example products and consider- hensive systems engineering process should be ations for their use.

ii Cloud Computing Table of Contents 1.0 Introduction to Products to Build a Private Cloud 1

2.0 Virtualization and Hypervisors 2

3.0 Storage Solutions 4

4.0 Security Products 6

5.0 Provisioning, Management, and Metering 8

Appendix A—Products Mentioned 11

References 12

THE BIG PICTURE: A private cloud computing environment can offer a path forward for Federal IT leaders seeking to realize some of the benefits of cloud computing while maximizing control over their environment.

Lawrence Pizette Products to Build a Private Cloud Geoffrey Raines

1.0 Introduction to Private Clouds “Companies that want the benefits of cloud computing Summary—For Federal information technology services without the risks are looking to create cloud-like (IT) leaders considering employing a private cloud environments in their own data centers. To do it, they’ll approach, this paper provides a window into prod- need to add a layer of new technologies—virtualization ucts that are available from the commercial marketplace. The paper is organized into the general management, cloud APIs, self-service portals, categories of virtualization and hypervisors, storage chargeback systems, and more— (e.g., storage area networks, storage arrays), security, to existing data center systems and processes.” and provisioning, management, and metering. The – Charles Babcock, Information Week, April 11, 2009 product categories in this paper were selected based upon a generalization of the private cloud computing marketplace and the technologies available in early computing: private, community, public, and hybrid.3 2010. However, vendor products do not always fall According to the NIST definition: directly within one category; some vendors are offering suites of capabilities that span multiple categories • Private clouds are “operated solely for an and other vendors are offering appliances, some of organization.” which package both hardware and software into “a • Community cloud “infrastructure is shared by cloud in a box.” Similarly, other vendors are offering several organizations and supports a specific capabilities for hybrid clouds, which can blur the line community that has shared concerns.” between private cloud capabilities and capabilities • Public cloud “infrastructure is made available that leverage public cloud offerings over the network. to the general public or a large industry group and is owned by an organization selling cloud Due to space constraints in the paper, we felt it was services.” best to describe exemplars in each general category. • Hybrid cloud “infrastructure is a composition of While the marketplace may have many products in two or more clouds.” a category, we looked for offerings that would best represent the concepts. Our selection of one vendor In this paper, hybrid cloud will refer the ability to over another is not an endorsement of the vendor or leverage a private cloud capability along with a com- its products. Also, while the general considerations munity or public cloud offering. described in this paper are expected to endure for Reasons for Building a Private Cloud—A private some time, cloud computing represents a fast mov- cloud capability can offer increased security over ing market so that specific products will likely be other deployment models, while still providing the replaced rather quickly. flexibility to implement unique capabilities with the Cloud Deployment Model Definitions—The benefits of ownership (e.g., data rights, control of National Institute of Standards and Technology the system). Forrester Research states, “An internal (NIST) describes four deployment models of cloud cloud [is] a multitenant, dynamically provisioned

Products to Build a Private Cloud 1 and optimized infrastructure with self-service devel- usage information for billing and accounting oper deployment, hosted within the safe confines of purposes. your own data center.” 4 Considerations— The benefits of using a private cloud approach (over • Before initiating a private cloud investment, traditional IT) may involve better utilization of Federal IT leaders should examine the attri- hardware, the ability to easily provide continuity butes of a private cloud in comparison to other of operations (COOP), the ability to quickly provi- deployment options. The private cloud can offer sion capabilities, and the ability to provide location complete control, but it may not provide the independent access or fast access to Software as a same economies of scale as community or public Service (SaaS) capabilities within a private organiza- clouds. tion. While these benefits of a private cloud may be • The goals of Federal leadership should sig- numerous, organizations that pursue development nificantly impact the technology investment. of a private cloud infrastructure are investing their While some private clouds may be driven by own capital with commensurate development sched- the desire to reduce costs, others may be driven ules, risks, and operating costs. primarily by its other tangible benefits, such Categories of Infrastructure for a Private Cloud— as scalability, location independent access, and • Virtualization technology provides the hypervi- the ability to support COOP. For more details sor software, which allows multiple instances of on analyzing costs, MITRE will be releasing “guest” operating systems to run concurrently on a Cloud Computing Cost and Business Case the same physical server. This capability enables Considerations white paper in Q4 2010. “multi-tenancy,” which is the sharing of physical • When developing a private cloud, a compre- resources providing for better server utilization. hensive systems engineering process should be The improved server utilization can reduce heat- employed; there can be many specific require- ing, ventilation and air condition (HVAC) and ments in a private cloud such as performance, electric costs, data center size and other related self-service provisioning, and robust security, infrastructure costs. It also can allow advanced which need to be considered carefully before users or data center operators to provision capa- acquiring specific hardware and software bilities quickly, facilitating scalability, self-service products. provisioning, and COOP. • As Federal organizations can have a large port- • Storage technology, such as storage area net- folio of legacy systems, IT leaders should develop works (SANs), can provide the underlying per- a migration plan that identifies the capabilities sistent storage for the data center. It can provide that will be moving to the cloud, and testing and capabilities to scale usage to petabyte levels of integration activities. storage, while facilitating COOP and location independent access of applications. • Security products offered to the private cloud 2.0 Virtualization and Hypervisors computing market provide capabilities such as identity management and virtual firewalls. They Summary—Hypervisor software provides a virtual will continue to evolve as private cloud products operating system (OS) environment, enabling mature and more knowledge is learned about multiple virtual OSs to be present on one physi- threats. cal server—the hypervisor is considered by many • Provisioning, management, and metering tools to be the foundational layer of a cloud computing provide the ability to deploy virtual operating environment and a critical component of a pri- systems, services, and applications. They may vate cloud. The hypervisor software can reduce provide the ability to provision Platform as a IT capital expenditures and operating expenses Service (PaaS) environments with enhanced fea- through higher utilization of processing resources; tures and have self-service portals. Tools in this the higher utilization of servers allows for a decrease category provide the ability to monitor server in the total number of servers, providing for lower utilization and system resources, application overall data center costs through reductions in service-level agreements (SLAs), and other opera- electric usage, HVAC costs, and data center floor tional attributes. Some products also provide space. Additionally, the hypervisor can facilitate

2 Cloud Computing There are two types of hypervisors on the market: A “Federal Government organizations increasingly Type 1 “bare-metal” hypervisor runs directly on the are incorporating virtualization to consolidate IT hardware without the overhead of a host OS. A Type resources, reduce space and power consumption 2 “hosted” hypervisor runs on a host OS; all system 5 and gain greater agility in launching new applications resource requests go through the host OS. As a and services for employees and constituents.” result, the Type 1 hypervisor technology can provide better performance than the Type 2 hypervisor, but – Barbara DePompa, Benefits of Virtualization for Government, a Type 2 hypervisor “supports the broadest range of Federal Computer Week hardware configurations.” 6 Figure 2-2 shows an example Type 1 hypervi- multi-tenant usage and the ability to provision envi- sor technology stack. Examples of this type of ronments rapidly. hypervisor on the market are VMware ESX and ESXi, XenServer from Citrix, and Hyper-V from Server virtualization products are available from Microsoft.7 In addition to the COTS hypervi- multiple sources, including commercial-off-the-shelf sors, there are two open source hypervisors, Xen (COTS) offerings and open source software prod- and KVM, that are Type 1. Xen is the underlying ucts such as Xen and Kernel-based Virtual Machine foundation for Citrix Systems XenServer and Oracle (KVM). While there are many options, as shown in VM.8 KVM has been adopted as the foundation for Figure 2-1, VMware and Microsoft have the greatest Red Hat’s Virtualization Hypervisor, which can penetration in the fast-moving x86 server virtual- be “deployed either as the standalone bare metal ization market as of Q3 2008. (IBM has a non-x86 hypervisor, or as Red Hat Enterprise Linux 5.4 and virtualization offering for its System Z platform, later installed as a hypervisor host technology.” 9 which would not appear on this list.) Figure 2-3 shows a Type 2 hypervisor technology stack, which requires a host operating system to run. “Which vendor(s) do you use today to enable x86 server virtualization?”

VMware 81%

Microsoft 47% Applications Applications Applications Citrix (XenServer, previously 25% XenSource) Guest Guest Guest OS OS OS Red Hat 17% Hypervisor Sun 16%

Oracle 12% Server Hardware Xen open source code, without 8% vendor support

Novell 7% Figure 2-2. Type 1 Hypervisor

Parallels (formerly 5% SWsoft)

Virtual Iron 2% Applications Applications Applications Don’t know 1% Guest Guest Guest Base: 106 North American and European x86 server virtualization decision-makers at OS OS OS companies with 100+ employees that have already implemented x86 server virtualization or are implementing it in the next 12 months (multiple responses accepted) Hypervisor Source: Enterprise and SMB Hardware Survey, North America and Europe, Q3 2008 Host Operating System

Figure 2-1. Virtualization Market Server Hardware

http://www.forrester.com/rb/Research/ server_trends_hypervisor_wars_heat_up/q/id/48165/t/2 Figure 2-3. Type 2 Hypervisor

Products to Build a Private Cloud 3 VMware Server is an example of a hosted virtu- factors (including processor, memory, disk input/ alization technology. All application requests for output, network input/output) and application resources need to go through both the hypervisor performance is essential. “Some virtualization and the operating system to get to the hardware. implementations fail to produce the expected savings because the cost per virtual machine Hardware and OSs Supported—When examin- (VM) is too high. This usually results from poor ing the hypervisor market, there are fundamental VM density and high hardware costs in shops features that must be understood, such as which that are too conservative with their deployments,” processors are compatible with the hypervisor and states Forrester Research’s Galen Schreck. As an which guest OSs are supported. For example, Intel example, he notes, “Many companies break even and AMD both provide special processor features, with physical hardware after three VMs, but they VT and AMD-V, respectively, known as hardware- only put a total of five onto a server capable of assisted virtualization, which are required by some running 15 VMs. They could have provisioned 10 hypervisors. As of March 2010, KVM requires Intel- more VMs for no incremental cost.” 12 based processors with virtualization technology or • An analysis of server environments and testing AMD-based processors with virtualization technol- will need to be planned as part of the process ogy.10 Similarly, Microsoft requires “a 64-bit envi- of migrating legacy applications to virtualized ronment” for the Hyper-V platform and processors environments. Not all legacy applications can which “support hardware-assisted virtualization seamlessly be migrated from traditional servers (Intel VT or AMD-V) technology.” 11 to virtualized environments. Not all operating Considerations— systems are supported by all hypervisors, and • While there is a significant benefit to increasing some applications may require dedicated physical the utilization of servers through virtualization, resources. Applications that require extensive use the hypervisor adds new software and potential of a graphics processing unit and systems with complexity to the IT stack. IT organizations will real-time requirements often are poor choices for need to plan for the incorporation of this new a virtualized environment. technology. • Virtual environments require management and 3.0 Storage Solutions governance. If they are not managed, the absence of controls coupled with the relative ease of cre- Summary—A persistent storage solution is an ating virtual machines can result in a prolifera- essential capability for a private cloud environment. tion of environments and a condition informally Commonly used storage solutions, incorporating known “VM sprawl,” which could offset antici- technologies such as disk arrays and SANs, can pro- pated efficiency gains. vide the underlying persistent data storage for high- • Virtualization products can increase software end, private cloud offerings. Disk arrays typically are licensing costs and the complexity of maintain- hardware units with multiple disk drives and con- ing software license compliance, which should troller units that allow for individual disk failures be factored into the cloud computing cost-benefit without loss of data (see Figure 3-1). SANs provide analysis. a network of storage resources that can be allocated • Virtual machines need to be allocated to physical to multiple servers. With a SAN, individual servers servers in an optimal manner for the data center are not directly attached to individual disks. While and monitored and configured for appropri- disk arrays, SANs, and traditional data center stor- ate performance. The exact number of virtual age solutions may be satisfactory for a private cloud machines per server depends on the resources implementation, cloud-focused storage features used in the virtual machine and other factors are starting to emerge in the marketplace that add such as networking requirements. If the servers features specifically targeted for the cloud architec- are under-utilized, cost savings will not accrue. If tures. As noted on SearchStorage, “Over the past the servers are over-utilized, SLAs may not be met year or so, vendors have come out with new products and performance will suffer. Getting the num- or re-worked old ones by adding multi-tenancy and ber of virtual machines and configuration right, security features that make the products suited to based upon an analysis of usage and performance power private storage clouds in the enterprise.” 13

4 Cloud Computing Operating System “Theoretically, almost any storage system and software can be used to build a private storage cloud. But over the past year or so, vendors have come out with new

products or re-worked old ones by adding multitenancy Virtual and security features that make the products suited Disk(s) to power private storage clouds in the enterprise.” – Private Storage Cloud Software And Hardware Product Guide, Controller February 15, 2010, SearchStorage.com

Important factors to understand with storage Disk Disk Disk Disk Parity solutions are how the features support the overall 1 2 3 4 Disk requirements of the private cloud, such as perfor- Data Disks mance, scalability, location independence, and COOP. Existing data center storage technologies Figure 3-1. Example RAID Disk Array may be used in today’s private cloud implementations. However, in the future, more products will http://docs.hp.com/en/B7961-90019/ch03s02.html likely emerge in the market place with new features that are unique and compelling for private clouds. The private cloud architect will have to choose Scalability—As Federal IT leadership plans their among a number of storage connection technologies. private cloud investments, scalability in the underly- When combining solutions, it is important to rec- ing storage infrastructure should be considered in ognize that some storage solutions utilize industry their technology selection. The ability for users to standard protocols, facilitating a heterogeneous data leverage cloud infrastructure in an “on demand” center, while others are limited to propriety inter- environment with burst capability is an important faces; Fibre Channel, hardware and software iSCSI attribute of cloud computing; therefore, scalability and NFS are common standards-based protocols for in the underlying storage is essential. For example, moving data to and from storage devices. In a 2008 EMC Atmos provides the ability to scale to the study, VMware compared these protocols for perfor- Petabyte level.17 Similarly, IBM Scale Out Network mance using their ESX Server 3.5 and the IOmeter Attached Storage (SONAS) offers “file-based storage, for benchmark comparison.14 They found that the supporting over 14 PBs.” 18 Fibre Channel was superior in throughput, latency, and CPU utilization, but that did not exclude the Continuity of Operations and Location For the Federal Government, other protocols that could meet requirements in Independent Access— the ability to provide COOP when adverse events certain circumstances with lower costs. happen (whether the result of a natural occurrence Storage Market—When Federal IT leaders look to or an adversary) is essential. Being able to access acquire storage technology—which can be a sig- and reconstitute data within a specified time limit nificant cost driver for a new private cloud invest- is an imperative for many systems. Frequently, the ment—they find a handful of major vendors that underlying storage system participates in the deliv- command a large portion of the market. In a 2006 ery of this capability. As an example of a capability survey, Forrester Research found that storage spend- that a storage product can provide, Atmos provides ing was almost 19 percent of overall hardware IT a GeoProtect feature. This GeoProtect feature allows spending in North America.15 In a more recent 2010 data to be replicated in multiple geographic loca- report, IDC indicates that the worldwide external tions according to policy and the ability to recover disk storage system factory revenue in Q4 2009 was data distributed across geographies.19 Other storage $5.288 billion.16 The top five vendors, EMC, IBM, vendors also have solutions for COOP and loca- HP, NetApp, and Dell commanded 69 percent of the tion independent access. For example, HP offers market; however, the remaining companies had a 31 the “StorageWorks B-series Remote Replication percent market share that is worth $1.643 billion. Solution.” 20 Similarly, IBM’s SONAS provides

Products to Build a Private Cloud 5 capabilities needed for many private clouds. They state, “In a cloud environment, applications and “[A] cloud computing model changes the services are not tethered to specific hardware way data is accessed, and private clouds components. Instead, processing is handled across a must have safeguards in place to protect an distributed, globally accessible network of resources, organization’s most critical and sensitive data.” which are dispensed on demand.” 21 – IBM discussion with Phil Hochmuth, senior analyst at research firm Considerations— Yankee Group, Public clouds, private clouds and your security • Given the cost of incorporating new technology into a data center, an analysis of whether existing data center storage technologies can be repur- posed for a private cloud should be considered. will be similar, there are new products to integrate • System architects should weigh costs against the (e.g., the virtualization infrastructure) and new capa- need for performance when selecting connec- bilities that require controlled user access (e.g., the tion technologies. While Fibre Channel provides ability to provision virtual environments). Identity much better performance than iSCSI and NFS, management includes authentication to validate the additional performance may not be needed that a requester is legitimate and authorization to for all systems. determine a legitimate requester’s access privileges. • When investing in storage technologies, orga- As this functionality is a “must” for many organiza- nizations should be aware of whether the prod- tions, most existing data centers already will have an ucts integrate with their existing infrastructure identity management capability that may be able to through open standards (e.g., Fibre Channel, be extended or integrated with a private cloud imple- iSCSI, NFS) or whether they have proprietary, mentation. For example, there are products on the closed interfaces. market, such as Centrify Suite for Linux/Unix, that provide the capability to leverage Active Directory for identity management and single sign-on to 4.0 Security Products virtualized environments (e.g., Citrix XenServer and 22 Summary—Federal IT leadership should acquire VMware ESX Server). There are many variations to and employ security technology within the context identity management products ranging from simple of an overall security plan, system requirements, systems to comprehensive capabilities that can pro- and organizational needs. Given the variation in vide robust authentication, facilitate operational and security requirements and approaches for private policy management, and federate identity capabilities. For example, RSA offers a wide range of security clouds, the products listed in this section should be 23 considered as examples of security products on the products, including RSA Access Manager. RSA market rather than a comprehensive list of ingredi- Access Manager supports authentication, authoriza- ents for securing a private cloud. Additionally, many tion, single sign-on, and centralized control across modern data centers will have variations of these an enterprise. Another example is Oracle’s Identity capabilities for their existing infrastructure. New Management and Access Management suite, which can be used for managing access in a PaaS private capabilities may need to integrate seamlessly with 24 existing infrastructure, which can have a significant cloud environment. Oracle’s offering supports impact on the choice of technologies and products. single sign-on, centralized access control policies, and identity federation. (Identity federation allows To exemplify the market offerings for securing more than one system to trust in the authenticity of a private clouds, we list products with the follow- user or requester of services.) ing capabilities: identity management; logging and auditing; firewall, intrusion detection systems (IDS) Logging and Auditing—Logging of events and and intrusion prevention systems (IPS); and antivi- auditing is important for forensics, helping to rus software for virtualized environments. identify threats and eliminating access and attack vectors. In a virtualized environment, there is a Identity Management—Federal IT leadership need to monitor the host environment, virtualized should consider the IT capabilities that are required environments, and communication between the vir- for identity management and access management in tualized environments. For example, OPNET ACE a private cloud. While identity management concepts Live VMon can provide information on the status of

6 Cloud Computing transactions that are “flowing between VMs on the capabilities, it can inspect the virtual environments same physical host” and provide a packet-based data for the installation of malicious rootkits, which is set.25 Logging and auditing also may be required for malware that can compromise the operating system. regulatory and statutory compliance. For example, the Health Insurance Portability and Accountability Anti-malware Software for Virtualized Anti-malware software, which can Act (HIPAA) requires “procedures to regularly Environments— include anti-virus, anti-rootkit, anti-Trojan horse, review records of information system activity, such and anti-spyware protection, is necessary for virtual as audit logs, access reports, and security incident guest operating systems. This is similar to the need tracking reports” for protected health information.26 for anti-malware client software in a traditional RSA enVision is an example tool that can provide server environment. There are many anti-malware auditing and event management for both the host software products on the market that can be used. environment and a VMware virtualized environ- For example, Trend Micro offers their ServerProtect ment.27 It also provides event correlation across a product for preventing malware. They also offer broad range of data center inputs to help identify a suite of tools to run in VMware ESX and ESXi threats. Oracle’s offering mentioned above provides environments that can leverage the VMsafe APIs.31 reports on authentication and authorization records, Trend Micro Core Protection for Virtual Machines which may be needed for demonstrating reporting protects active and dormant virtual machines and compliance. Other products can be used to provide provides a layer of separation between the virtual audit records through simple data downloads; for machine being scanned and the scanning agent; the example, Citrix provides the XenServer Audit Tool, scanning agent is run on a separate virtual machine. which can be used to export the XenServer log file.28 Intrusion Detection and Intrusion Protection— Firewalls for Virtualized Environments—A IDS and IPS capabilities are needed to detect and virtual firewall is essential for allowing authorized prevent malicious access to data, applications, and communications and preventing unauthorized or systems. IDS and IPS software products can require malicious network traffic. In a traditional server, information from multiple sources, such as operat- this network traffic is monitored at the boundaries ing system events and network activity, to detect of the server. However, in a virtual environment it is and prevent intrusions. IDS systems log these events necessary to provide the capability between virtual (e.g., for forensics) and IPS systems attempt to machines as shown in Figure 4-1. For example, Altor prevent them by automatically modifying configu- Networks provides a product that can implement rations. Similar to the malware detection software firewall policies at the virtual machine level and described above, Trend Micro uses VMware’s monitor inter-virtual machine communications.29 VMsafe API to provide IDS and IPS capabilities. The Altor VF Firewall leverages VMware’s VMsafe suite of APIs, which provides third-party access Considerations— directly into the VMware vSphere virtualized • Depending on security needs, security archi- environment. Similarly, IBM Virtual Server Security tects should perform an analysis and consider a for VMware can inspect traffic between VMware pilot to understand new threats and mitigating virtual machines.30 In addition to other security strategies. • The data center IT security plan, including policies and procedures, should be modified to address and mitigate new threats as they emerge. Applications Applications Applications • An overall security plan should ensure that virtual environments are appropriately isolated Guest Virtual Guest Virtual Guest Virtual from each other and that communication is OS Firewall OS Firewall OS Firewall monitored between virtual machines, even on Hypervisor the same physical server (e.g., consider a virtual firewall and intrusion detection and prevention Server Hardware software). • Security architects should consider the additional needs for logging and auditing capabili- Figure 4-1. Virtual Firewall ties within a virtualized environment. For each

Products to Build a Private Cloud 7 physical server, there can be multiple guest oper- Self-Service Portal ating system environments generating events, which may need to be correlated and examined. Provisioning • There may be cost savings and reduced complexity for data centers when acquiring private SLAs/Resource Management cloud security products that integrate with their existing data center solutions (e.g., anti-malware Metering software and identity management software). Virtualization

5.0 Provisioning, Management, and Metering Physical Server Hardware

“The impact of rapid provisioning on ROI Figure 5-1. Provisioning, Billing, and Self-Service Capabilities business cases can be profound.” – Open Group, Building Return on Investment from Cloud Computing example, IBM’s approach to provisioning in their CloudBurst hardware-plus-software appliance is to provide an integrated tool set based upon a suite of IBM CloudBurst and Tivoli software products (e.g., Overview—For Federal IT leaders looking to har- Tivoli Service Automation Manager).32 As another ness the benefits of cloud computing in a private example, 3Tera, recently purchased by CA, is similar environment, provisioning virtual machine envi- to IBM in that they provide an integrated solution ronments and applications to the cloud is one of the with a provisioning and management software most important aspects of a private cloud technol- suite for use in private clouds; however, they differ ogy. Provisioning an environment is the act of creat- significantly in their approach to hardware. Some ing a server or virtual server environment with all vendors are choosing to bundle hardware and the requisite software and configurations needed for software together, while others are offering solutions it to provide the desired capabilities. For example, a that operate on generic commodity hardware. For server may require an operating system, application example, the IBM CloudBurst appliance is com- software, security software, and multiple configura- prised of hardware and software, whereas 3Tera tion settings to function correctly. The mixture of AppLogic comes with a software suite that runs on software and configurations can vary significantly, “commodity” hardware.33 based upon the function of the server. The ability to dynamically provision preconfigured environ- Self-Service Portal—Driven by the need to provide ments, independent of the underlying hardware, capabilities to users “on-demand” and lower costs, facilitates “on-demand” scalability and COOP. This Federal IT leaders should consider employing self- capability can allow IT organizations to stand up service capabilities in their private cloud implemen- virtual servers quickly in geographically disparate tations. For example, IBM CloudBurst provides a locations or scale application resources based upon self-service portal capability that provides the ability increased demand. System resource utilization can to provision and scale preconfigured environments be monitored through management software, which (e.g., Linux, Apache, MySQL, applications, and can also measure compliance with SLAs to ensure environment variables). In addition, when provi- the IT infrastructure is meeting operational needs. sioning a virtual machine environment through Additional capabilities may include “metering,” the portal, users can allocate system resources to which is the gathering of usage date for accounting the virtual machine, such as the amount of random or billing purposes. As shown in Figure 5-1, these access memory (RAM) that it can use. Similar to software capabilities provide many of the compel- IBM CloudBurst, AppLogic allows comprehensive ling features for Government organizations to application environments to be dynamically provi- employ private cloud technology. sioned into virtual machines. Provisioning Environments—The market for Metering—Federal IT leaders may want to look for provisioning in cloud computing is burgeoning metering capabilities if they intend to allocate costs with products and a variety of approaches. For or charge projects based upon usage. For example,

8 Cloud Computing both the IBM and AppLogic suites described above integration with storage hardware (iSCSI, NAS, and provide metering capabilities. Metering is the SANs) and works with multiple hypervisors, includ- process of collecting usage information that cloud ing VMware vSphere, VMware ESX, VMware ESXi, computing operators can feed into billing systems KVM, and Xen.34 or other similar downstream accounting systems In an effort to provide hybrid capabilities, some ven- to allocate costs or charge users. By implement- dors are collaborating across platforms. RightScale ing metering, the data center may be able to offer provides integration with Eucalyptus, Amazon users, projects, or other organizations a “pay as you Web Services, and others, enabling users to man- go” model. For some private clouds, this capability age hybrid cloud implementations through a single may not be needed, but others may find it essential portal.35 In addition, there are commercial products due to accounting rules or for funding the cloud on the market that can directly provision a precon- environment. figured environment and applications to either a Monitoring and Service-Level Agreements—In private environment or Amazon’s EC2. sharing and scaling resources, Federal IT leadership In another example, Cloudswitch has a new product will want to ensure that operational needs are being in beta as of March 2010 for transparently hosting met. Monitoring software can be used to maintain VMware-based applications in a private cloud or SLAs and ensure users have the system “up time” and public cloud offering with a secure networking con- responsiveness they need. For example, the AppLogic nection (like a VPN) between the cloud provider platform enables a scalable “grid” and allows for the and the organization’s data center.36 Products in this mirroring of data across the commodity hardware, space are emerging and provide significant promise providing for increased availability and backup for expansion in the future. The expanded capabili- through the software. AppLogic allows for an N-tier ties may enable portability, facilitate COOP, provide environment across virtual machines to be managed scalability, and lower costs. as a single entity through a graphical user interface. IBM also provides monitoring software and manage- Considerations— ment of SLAs in their CloudBurst product. • Federal leaders should plan to pilot and test provisioning and management software to ensure Hybrid Clouds—Federal IT leaders whose organi- that it meets requirements and integrates with zations are implementing private clouds may want other data center capabilities as anticipated. to consider hybrid cloud technology in the future as • Examine capabilities of products in this domain the technology matures. While many organizations for their ability to integrate with the layers of the are pursuing private cloud approaches to maximize private cloud. For example, some products may security and control over processing and informa- require a particular type of hypervisor software tion, there may be opportunities to pursue a hybrid for integration or only work with certain security approach in the future to provide burst capacity, products. ensure SLAs are met, and lower costs. Innovative • If workflows need to be monitored in order companies are working on hybrid cloud solutions to meet SLAs, ensure that the management that will enable IT organizations to extend their pri- software can meet this need. For example in vate clouds to Government-run community cloud or multi-tier application architectures, a remedy to public cloud offerings. For example, the Eucalyptus degraded performance may require a sophisticated open source platform provides an innovative self- response with the ability to provision multiple new service provisioning environment with a different instances of virtual servers with different capabili- set of attributes and trade-offs from many other pri- ties (e.g., Web servers, application servers). vate cloud environments. By using the same APIs as • Within an enterprise, it can be very efficient to Amazon Elastic Compute Cloud (EC2), Eucalyptus have a self-service portal, but it also opens up can offer its users easy portability—for both appli- new resource management and security chal- cations and tools—between a private cloud and a lenges. Therefore, data center leaders should public cloud. By reusing the EC2 API, Eucalyptus consider the security controls and policies facilitates the ability to move capabilities between necessary for a self-service portal. The need a private cloud and a public offering. The enter- goes beyond malicious activity and includes prise edition with commercial extensions provides efficient, prioritized use of data center resources.

Products to Build a Private Cloud 9 If organizations do not control this access, they Conclusion—A private cloud computing approach could deploy many virtual environments, which is a possible path forward for Federal IT leaders may be similar or not high priority, resulting in looking to leverage cloud computing while main- inefficient use of computing resources. These low taining control over their environment. A private priority, virtual environments could compete cloud approach can be used to harness the process- with high value processing needs—a situation ing power of data centers efficiently while delivering that can be avoided through security, procedure, new features to users. As part of a comprehensive and policy. systems engineering process, existing capabilities • If a private cloud spans multiple geographic loca- can be integrated with innovative new products for tions, Federal leaders should determine policies features such as COOP, on-demand scalability, and and procedures for provisioning applications location independent access. These features coupled to specific data centers (e.g., based upon per- with appropriate processes, a self-service portal, and formance, COOP requirements, cost efficiency, comprehensive security, can be effective at achieving available capacity). the vision of a private cloud.

10 Cloud Computing Appendix A—Products Mentioned

Product Website Description Altor Networks VF Firewall http://altornetworks.com/products/vf/ Virtual firewall software AMD-Virtualization http://sites.amd.com/us/business/it-solutions/virtualization/Pages/ Extensions to x86 instruction set to support amd-v.aspx virtualization Amazon Elastic Compute Cloud http://aws.amazon.com/ec2/ Public infrastructure as a service cloud computing Apache http://www.apache.org/ Open source software Web server CA 3Tera AppLogic http://www.3tera.com/AppLogic/ Distributed, scalable private cloud infrastructure that runs on commodity hardware Citrix Systems XenServer http://www.citrix.com/english/ps2/products/product. Server virtualization platform asp?contentID=683148 EMC Atmos http://www.emc.com/products/detail/software/atmos.htm Private cloud storage Eucalyptus Systems Enterprise http://www.eucalyptus.com/products/eee Private cloud infrastructure software with Edition APIs compatible with Amazon EC2: open source software plus proprietary extensions Eucalyptus (open source http://open.eucalyptus.com/ Private cloud infrastructure software with software) APIs compatible with Amazon EC2: open source software HP StorageWorks B-series http://h71028.www7.hp.com/enterprise/cache/512015-0-0-0-121. Storage replication solution Remote Replication Solution html IBM Cloud Burst http://www-01.ibm.com/software/webservers/cloudburst/ Private cloud appliance, “Cloud-in-a-box” IBM Scale Out Network http://www-03.ibm.com/systems/storage/network/sonas/ Network attached storage Attached Storage (SONAS) IBM SystemZ http://www-03.ibm.com/systems/z/ IBM mainframe computers IBM Tivoli Service Automation http://www-01.ibm.com/software/tivoli/products/tsam-facts.html Provisioning automation Manager IBM Virtual Server Security for http://www-01.ibm.com/software/tivoli/products/ Security software for VMware environment VMware virtual-server-protection/ Intel VT http://www.intel.com/technology/virtualization/technology.htm Extensions to x86 instruction set to support virtualization Kernel-based Virtual Machine http://www.linux-kvm.org/page/Main_Page Open source software Linux virtualization solution Microsoft Hyper-V http://www.microsoft.com/windowsserver2008/en/us/hyperv.aspx Virtualization software MySQL http://www.mysql.com/ Open source database software OPNET ACE Live VMon http://www.opnet.com/whitepapers/EMA_Vmon_Reveals_Virtual_ Monitoring software for inter-virtual machine Network_Performance.pdf traffic/transactions Oracle Identity Management http://www.oracle.com/us/products/middleware/identity- Identity management tool and Access Management Suite management/index.html Oracle VM http://www.oracle.com/us/technologies/virtualization/oraclevm/ Virtualization software index.html RSA Access Manager http://www.rsa.com/node.aspx?id=1186 Identity management tool RSA enVision http://www.rsa.com/node.aspx?id=3170 Software to collect and analyze security event information RightScale http://www.rightscale.com/ Cloud management environment Trend Micro ServerProtect http://us.trendmicro.com/us/products/enterprise/ Anti-malware software for Windows servers serverprotect-for-microsoft-windows/ Trend Micro Core Protection for http://us.trendmicro.com/us/solutions/enterprise/security- Anti-malware software for VMware ESX/ Virtual Machines solutions/virtualization/index.html ESXi environments VMware ESX http://www.vmware.com/products/esx/ Bare-metal hypervisor VMware ESXi http://www.vmware.com/products/esx/ Bare-metal hypervisor VMware vSphere http://www.vmware.com/products/vsphere/ Virtualization platform Xen http://www.xen.org/ Open source software hypervisor

Products to Build a Private Cloud 11 References

1 Mell, P. and T. Grance, October 7, 2009, “The NIST Definition of Cloud Computing,” http://csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc. 2 IDC, March 5, 2010, “Total Disk Storage Systems Turn a Corner, Posting First Year-Over-Year Gain in More Than Four Quarters,” http://www.idc.com/about/viewpressrelease.jsp?containerId=prUS22236010§ionId=null&elementId=null&pageType=SYNOPSIS. 3 Mell, P. and T. Grance, October 7, 2009, “The NIST Definition of Cloud Computing,” http://csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc. 4 Staten, J. et al., April 13, 2009, “Deliver Cloud Benefits Inside Your Walls,” Forrester Research, http://www.forrester.com/rb/Research/deliver_cloud_benefits_inside_walls/q/id/54035/t/2. 5 IBM, “Virtual Systems Overview,” http://publib.boulder.ibm.com/infocenter/eserver/v1r2/index.jsp?topic=/eicay/eicayvservers.htm. 6 VMware, “Understanding Full Virtualization, Paravirtualization, and Hardware Assist,” http://www.vmware.com/files/pdf/VMware_paravirtualization.pdf. 7 Jones, D., June 1, 2009, “Bare Metal Hypervisors: A Group Test,” ZDNet Australia, http://www.zdnet.co.uk/reviews/bare-metal/2009/06/01/bare-metal-hypervisors-a-group-test-39662582/. 8 October 13, 2009, “Xen vs. KVM Linux Virtualization Hypervisors,” SearchServerVirtualization.com, http://searchservervirtualization.techtarget.com/generic/0,295582,sid94_gci1371226,00.html. 9 “Red Hat Enterprise Virtualization for Servers,” http://www.redhat.com/virtualization/rhev/server/. 10 KVM, http://www.linux-kvm.org/page/Status. 11 “Microsoft Virtualization with Hyper-V: FAQ,” http://www.microsoft.com/windowsserver2008/en/us/hyperv-faq.aspx#TechnicalInformation. 12 http://www.forrester.com/rb/Research/why_isnt_server_virtualization_saving_us_more/q/id/48081/t/2., accessed October 22, 2010. 13 February 15, 2010, “Private Storage Cloud Software and Hardware Product Guide,” SearchStorage.com http://searchstorage.techtarget.com/generic/0,295582,sid5_gci1381130,00.html. 14 VMware, “Comparison of Storage Protocol Performance, ESX Server 3.5,” http://www.vmware.com/files/pdf/storage_protocol_perf.pdf. 15 Balaouras, S. and F.Gillett, April 18, 2007, “Storage Buyer Profile: 2006,” http://www.forrester.com/rb/Research/storage_buyer_profile_2006/q/id/41962/t/2. 16 IDC, March 5, 2010, “Total Disk Storage Systems Turn a Corner, Posting First Year-Over-Year Gain in More Than Four Quarters,” http://www.idc.com/about/viewpressrelease.jsp?containerId=prUS22236010§ionId=null&elementId=null&pageType=SYNOPSIS. 17 “EMC Atmos Product Information,” http://www.emc.com/products/detail/software/atmos.htm. 18 “IBM Scale Out Network Attached Storage (SONAS),” http://www-03.ibm.com/systems/storage/news/center/sonas/index.html. 19 “EMC Atmos Product Video,” http://www.youtube.com/watch?v=yuy-HZh0FJI. 20 “HP StorageWorks B-series Remote Replication Solution,” http://h71028.www7.hp.com/enterprise/cache/512015-0-0-0-121.html. 21 “IBM Scale Out Network Attached Storage,” http://www-03.ibm.com/systems/storage/network/sonas/features.html. 22 “Centrify Suite for Linux & UNIX Systems,” http://www.centrify.com/products/active-directory-integration-linux-unix.asp. 23 RSA, http://www.rsa.com/node.aspx?id=1155.

12 Cloud Computing 24 Oracle, October 2009, “Platform-as-a-Service Private Cloud with Oracle Fusion Middleware,” http://www.oracle.com/us/technologies/cloud/036500.pdf. 25 “OPNET ACE Live VMon Reveals Virtual Network Performance,” http://www.opnet.com/whitepapers/EMA_Vmon_Reveals_Virtual_Network_Performance.pdf. 26 “HIPPA Administrative Simplification, Regulation Text,” p. 40, 164.308, http://www.hhs.gov/ocr/privacy/hipaa/administrative/privacyrule/adminsimpregtext.pdf. 27 “RSA enVision platform,” http://www.rsa.com/node.aspx?id=3170. 28 “XenServer Auditing Tool Readme,” http://community.citrix.com/download/attachments/54591507/XenServerAuditingReadme.pdf?version=1. 29 “Altor Virtual Firewall,” http://altornetworks.com/products/vf/. 30 “IBM Security Solutions: Threat Mitigation,” http://www-935.ibm.com/services/us/index.wss/offering/iss/a1031810. 31 Trend Micro, “Cloud Virtualization Environment,” http://us.trendmicro.com/us/solutions/enterprise/security-solutions/virtualization/index.html. 32 “IBM CloudBurst,” http://www-01.ibm.com/software/tivoli/products/cloudburst/ 33 “CA 3Tera AppLogic - What It Can Do for You,” http://www.3tera.com/AppLogic/What-it-can-do.php. 34 “Eucalyptus Enterprise Edition Data Sheet,” http://www.eucalyptus.com/themes/eucalyptus/pdf/eee_datasheet_v4_sept92009.pdf. 35 Right Scale, “Multi-Cloud Engine,” http://www.rightscale.com/products/features/multi-cloud-engine.php. 36 Brodkin, J., March 2, 2010, “CloudSwitch Moves Virtual Apps to Amazon Cloud,” TechWorld, http://news.techworld.com/data-centre/3213977/cloudswitch-moves-virtual-apps-to-amazon-cloud/.

Products to Build a Private Cloud 13