Pivotal Ready Architecture for vSphere Reference Architecture Pivotal Cloud Foundry 2.4 edition 1 Pre-Reqs VMware vSphere 6.5U2 (not v6.7) VMware NSX-T 2.3 or newer Pivotal Ops Manager 2.4 or newer Pivotal PAS 2.4 or newer Pivotal VMware PKS 1.2 or newer vxRail ESSM 4.5.150 or newer Dell EMC ECS 3.1 or newer This recommended architecture includes VMware vSphere and NSX-T, a software-defined network virtualization that runs on VMware ESXi virtual hosts and combines routing, firewall, NAT/SNAT, and load balancing. In the absence of NSX-T, see below for architectures that do not rely on it. To use all features listed here, NSX-T requires at least Advanced licensing from VMware when used with ​ ​ PAS. The equivalent of that licensing is included with the PKS product. For more information about installing and configuring NSX-T for use with PCF on vSphere, see VMware’s official documentation and the Pivotal Cookbook for NSX-T. ​ ​ This design supports long-term use, capacity growth at the IaaS (vSphere) level and PaaS (PAS/PKS) level, and maximum installation security through the NSX-T firewall. It allocates a minimum of four servers to each cluster to meet VMware’s best practice around HA and VSAN. It also spreads PCF components across three clusters, as recommended for High Availability. ​ ​ Major areas of consideration are: 1. Introducing Dell EMC PRA a. Key Benefits b. Hardware Reference Design 2. SDN or No SDN Deployments (Will NSX be used?) 3. PAS without NSX-T 4. PKS without NSX-T 5. Network design and allocations 6. PAS and PKS with NSX-T 7. Storage design 8. Compute and HA considerations 9. Scaling up and capacity management considerations 2 If your design approach includes PKS, you will have NSX-T included and, though not required, that technology opens up many interesting and flexible options. If you want to deploy without NSX-T involved (see separate ​ materials on NSX-V design considerations) you can be successful with PAS and/or PKS native on vSphere but ​ the design choices are very different. Introducing Dell EMC Pivotal Ready Architecture Pivotal Ready Architecture (PRA) is an integrated development platform that is based on cloud computing and storage, offering performance and convenience, addresses the challenges of creating, testing, and updating applications in a consolidated production environment. Key Benefits PRA enables you to quickly and reliably create an integrated development platform and provides a variety of deployment options to meet your needs. It is a tested PCF deployment that uses the scalable, hyperconverged VxRail Appliance and VMware vSphere with VMware vSAN and NSX-T. With PRA, you can deploy a production-ready PAS environment or PKS Kubernetes clusters more quickly and reliably than when using a do-it-yourself approach. The PAS and PKS deployments are designed to meet application operational needs. Dell EMC, VMware and Pivotal Services can help with installing and testing PRA in your IT environment. IT professionals tasked with planning compute, network, and storage capacity can work with Dell EMC Services to determine the resource requirements for each component. PRA enables developers to publish, run, and scale legacy, COTS & cloud-native applications. The PRA platform is based on an integrated infrastructure that enables IT operators to easily manage and scale the development environment with tools such as VxRail Manager and PCF Operations Manager. With PRA, you benefit from: ● A reference architecture for a variety of highly available Pivotal software deployments ● A private cloud and infrastructure service with simplified deployment ● A highly available platform to develop and quickly deploy applications, reducing the time to application delivery ● A modern developer platform that boosts developer productivity by combining application services, service discovery, container management, and orchestration with an ecosystem of plug-ins for developer tools ● Increased cloud solution portability and agility, because you are not locked into public cloud APIs 3 ● External S3-compatible storage or an internal Web Distributed Authoring and Versioning (WebDAV) server for the Pivotal Application Service (PAS) blobstore Architectural Overview PRA is a highly available reference architecture that describes how to create a production-ready PCF deployment using vCenter, NSX-T, for PAS and/or PKS. The Management and Edge cluster is spread across three racks, and it is used to run a vCenter that manages the compute clusters, also known as Availability Zones (AZs), where NSX-T is deployed. You can deploy and run PAS or PKS or both on the three compute clusters or AZs. The following figure shows a high-level overview of PRA. Management and Edge Cluster for NSX-T The Management cluster is used to deploy the NSX-T Management Plane (NSX-T Manager, and three Controller instances). The Edge Cluster is used to deploy large Edge VMs (16 GB memory, 8 vCPUs), two instances per rack. The Management and Edge Cluster is deployed using the internal vCenter Server. A separate vCenter is used to manage the Compute Cluster (the hosts the compute nodes use for deployment of PAS or PKS VMs and application containers). All PAS and PKS deployments use the external vCenter server. The vCenter setups use a three cluster, six node arrangement to provide high availability. 4 The minimum sizing for the Management Cluster is four nodes, but the standard is two nodes from each rack, starting at six total nodes. For NSX-T Edges, minimum sizing is co-located with the Management components in four nodes, standard sizing is co-located with the six node/three rack Management components or for max performance, a stand-alone four node cluster exclusive to Edges. Top-of-rack switches The VxRail Appliance is compatible with many customer-supplied networks and switches. VxRail Appliance nodes communicate over one or more customer-provided network switches, typically a top-of-rack (ToR) switch. The ToR switches must support IPv6 multicast pass-through and IPv4 unicast. A single ToR switch can be a single point of failure, so the recommendation is to use dual ToR switches, such as the Dell EMC Switch S4048. Each rack's ToRs must be connected to the customer's spine/core network. Management and Edge cluster node networking 5 Compute cluster networking Non-SDN Deployments VLAN Considerations Without an SDN environment, you will draw network address space and broadcast domains from the greater datacenter pool. For this approach, you will need to deploy PAS and PKS in a particular alignment. ● Choose VLANs and address space for PAS that does not overlap with PKS ● Load balancing will be performed external to the PCF installation ● Client SSL termination will need to happen either at the load balancer or gorouters, or both, so plan for certs to accomplish this ● Firewalling will be accomplished external to the PCF installations C2C & CNI Considerations ● Container-to-container (C2C) networking is a feature of the tile, not the environment ● Changing the C2C strategy after deployment is possible, but with NSX-T, you’ll need both the C2C tile and a infrastructure of NSX-T to be successful. Migrating to an SDN-enabled solution from this design is possible but best considered as a greenfield deployment; inserting a SDN layer under an active PCF installation is very disruptive. PAS Without SDN Pivotal PAS, as a part of the PCF umbrella, is fully functional without an SDN overlay. Flexibility and other features are lost without SDN, but it is a valid approach. There’s no hard dependency on SDN features in Pivotal PAS to date. 6 Refer to Installing PAS on vSphere for more information. ​ ​ A small scale implementation would start with a Management Stack compute stack and a single AZ of compute for the PAS workloads. More AZs can be added as new capacity in the future as needed. This approach is represented here: 7 A full scale production approach would include three AZs of compute capacity at the start, with the growth vector being vertical (adding more compute hosts to the existing three clusters. All components would be spread across multiple racks to manage fault domains. This approach is represented here: PKS Without SDN 8 Pivotal Container Service (PKS) is supplied with NSX-T SDN included, along with the associated licensing. The assumption is that the reader will want to use all of this technology at once to get full benefit from both. However, the bundled NSX-T SDN solution can be ignored, as there is a built-in network stack (flannel) that takes care of container networking. Refer to Installing PKS on vSphere for more information on PKS without NSX-T. ​ ​ SDN-Enabled Deployments using NSX-T Network Requirements for PAS PAS requires a number of statically defined networks to host its main elements. If we consider the Tenant side of an NSX-T deployment, we define a series of non-routable address banks that the NSX-T routers will manage for us. Similar to reference designs in the past, those are the following: NOTE: Relative to reference designs from the PCF v1.xx days, these static networks have a smaller host address space assigned. With NSX-T, you will find that building for many dynamically assigned networks is a better strategy than a few, very large networks as in the past. ● Infrastructure - 192.168.1.0/24 ● Deployment - 192.168.2.0/24 ● Services - 192.168.3.0/24 The Services network is handy for use with Ops Manager tiles that you might add in addition to PAS. You can think of the Services network as the “everything else not PAS” network. Some of these tiles can call for additional network capacity to grow into on demand.