VMware Tanzu Kubernetes Grid VMware Tanzu Kubernetes Grid 1.3 VMware Tanzu Kubernetes Grid You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com © Copyright 2021 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 2 Contents 1 VMware Tanzu Kubernetes Grid 1.3 Documentation 12 Tanzu Kubernetes Grid Architecture 12 Use the Tanzu Kubernetes Grid Documentation 13 Intended Audience 14 2 Tanzu Kubernetes Grid Concepts 15 Management Cluster 15 Tanzu Kubernetes Clusters 16 Tanzu Kubernetes Cluster Plans 16 Shared and In-Cluster Services 16 Tanzu Kubernetes Grid Instance 16 Bootstrap Machine 16 Tanzu Kubernetes Grid Installer 16 Tanzu Kubernetes Grid and Cluster Upgrades 17 3 Install the Tanzu CLI and Other Tools 18 Prerequisites 19 Download and Unpack the Tanzu CLI and kubectl 20 Install the Tanzu CLI 21 Install the Tanzu CLI Plugins 22 Tanzu CLI Help 23 Install kubectl 23 What to Do Next 24 Install the Carvel Tools 25 Install ytt 25 Install kapp 26 Install kbld 27 Install imgpkg 28 Tanzu CLI Command Reference 29 Table of Equivalents 32 Tanzu CLI Configuration File Variable Reference 33 Common Variables for All Infrastructure Providers 33 vSphere 44 Amazon EC2 49 Microsoft Azure 51 Customizing Clusters, Plans, and Extensions with ytt Overlays 54 Clusters and Cluster Plans 54 Extensions and Shared Services 55 VMware, Inc. 3 VMware Tanzu Kubernetes Grid 4 Deploying Management Clusters 57 Overview 57 Installer UI vs. CLI 58 Platforms 58 Configuring the Management Cluster 59 What Happens When You Create a Management Cluster 59 Core Add-ons 60 Prepare to Deploy Management Clusters 61 Prepare to Deploy Management Clusters to vSphere 61 Prepare to Deploy Management Clusters to Amazon EC2 69 Prepare to Deploy Management Clusters to Microsoft Azure 82 Enabling Identity Management in Tanzu Kubernetes Grid 88 Deploying Tanzu Kubernetes Grid in an Internet-Restricted Environment 93 Install VMware NSX Advanced Load Balancer on a vSphere Distributed Switch 104 Prepare a vSphere Management as a Service Infrastructure 118 Deploy Management Clusters with the Installer Interface 122 Prerequisites 122 Set the TKG_BOM_CUSTOM_IMAGE_TAG 123 Start the Installer Interface 124 Configure the Infrastructure Provider 126 Configure the Management Cluster Settings 132 (vSphere Only) Configure VMware NSX Advanced Load Balancer 135 Configure Metadata 137 (vSphere Only) Configure Resources 139 Configure the Kubernetes Network and Proxies 139 Configure Identity Management 141 (vSphere Only) Select the Base OS Image 144 Register with Tanzu Mission Control 145 Finalize the Deployment 145 What to Do Next 148 Deploy Management Clusters from a Configuration File 149 Prerequisites 149 Create the Cluster Configuration File 151 (v1.3.1 Only) Set the TKG_BOM_CUSTOM_IMAGE_TAG 152 Run the tanzu management-cluster create Command 152 What to Do Next 154 Create a Management Cluster Configuration File 154 Configure Identity Management After Management Cluster Deployment 178 Prerequisites 179 Connect kubectl to the Management Cluster 179 Check the Status of an OIDC Identity Management Service 180 VMware, Inc. 4 VMware Tanzu Kubernetes Grid Check the Status of an LDAP Identity Management Service 182 Provide the Callback URI to the OIDC Provider 183 Generate a kubeconfig to Allow Authenticated Users to Connect to the Management Cluster 184 Create a Role Binding on the Management Cluster 187 Examine the Management Cluster Deployment 192 Management Cluster Networking 192 Configure DHCP Reservations for the Control Plane Nodes (vSphere Only) 192 Verify the Deployment of the Management Cluster 192 Retrieve Management Cluster kubeconfig 194 What to Do Next 195 5 Deploying Tanzu Kubernetes Clusters 196 About Tanzu Kubernetes Clusters 197 Tanzu Kubernetes Clusters, kubectl, and kubeconfig 197 Using the Tanzu CLI to Create and Manage Clusters in vSphere with Tanzu 198 Deploy Tanzu Kubernetes Clusters 198 Prerequisites for Cluster Deployment 198 Create a Tanzu Kubernetes Cluster Configuration File 199 Deploy a Tanzu Kubernetes Cluster with Minimum Configuration 199 Deploy a Cluster with Different Numbers of Control Plane and Worker Nodes 201 Configure Common Settings 203 Deploy a Cluster in a Specific Namespace 203 Create Tanzu Kubernetes Cluster Manifest Files 203 Advanced Configuration of Tanzu Kubernetes Clusters 203 What to Do Next 204 Deploy Tanzu Kubernetes Clusters to vSphere 204 Tanzu Kubernetes Cluster Template 204 Deploy a Cluster with a Custom OVA Image 207 Configure DHCP Reservations for the Control Plane Nodes 208 What to Do Next 208 Use the Tanzu CLI with a vSphere with Tanzu Supervisor Cluster 209 Prerequisites 209 Step 1: Add the Supervisor Cluster 209 Deploy Tanzu Kubernetes Clusters to Amazon EC2 213 Tanzu Kubernetes Cluster Template 213 Tanzu Kubernetes Cluster Plans and Node Distribution across AZs 216 Deploy a Cluster that Shares a VPC and NAT Gateway(s) with the Management Cluster 217 Deploy a Cluster to an Existing VPC and Add Subnet Tags 218 Deploy a Prod Cluster from a Dev Management Cluster 219 What to Do Next 219 Deploy Tanzu Kubernetes Clusters to Azure 220 VMware, Inc. 5 VMware Tanzu Kubernetes Grid Create a Network Security Group for Each Cluster 220 Azure Private Clusters 220 Tanzu Kubernetes Cluster Template 221 What to Do Next 223 Deploy Tanzu Kubernetes Clusters with Different Kubernetes Versions 223 List Available Versions 224 List Available Upgrades 224 How Tanzu Kubernetes Grid Updates Kubernetes Versions 224 Deploy a Cluster with a Non-Default Kubernetes Version 225 Deploy a Cluster with an Alternate OS or Custom Machine Image 226 Customize Tanzu Kubernetes Cluster Networking 226 Deploy a Cluster with a Non-Default CNI 226 Deploy Pods with Routable, No-NAT IP Addresses (NSX-T) 228 Create Persistent Volumes with Storage Classes 232 Overview: PersistentVolume, PersistentVolumeClaim, and StorageClass 232 Supported Storage Types 232 Default Storage Classes 233 Set Up CNS and Create a Storage Policy (vSphere) 234 Create a Custom Storage Class 235 Use a Custom Storage Class in a Cluster 236 Enable Offline Volume Expansion for vSphere CSI (vSphere 7) 236 Configure Tanzu Kubernetes Plans and Clusters 240 Where Cluster Configuration Values Come From 240 Files to Edit, Files to Leave Alone 241 Configuration Precedence Order 243 ytt Overlays 244 6 Managing Cluster Lifecycles 248 Manage Your Management Clusters 249 List Management Clusters and Change Context 249 See Management Cluster Details 249 Management Clusters, kubectl, and kubeconfig 250 Management Clusters and Their Configuration Files 250 Add Existing Management Clusters to Your Tanzu CLI 251 Delete Management Clusters from Your Tanzu CLI Configuration 252 Scale Management Clusters 252 Update Management Cluster Credentials (vSphere) 253 Manage Participation in CEIP 253 Create Namespaces in the Management Cluster 253 Delete Management Clusters 254 What to Do Next 255 VMware, Inc. 6 VMware Tanzu Kubernetes Grid Managing Participation in CEIP 255 Opt In or Opt Out of the VMware CEIP 256 Add Entitlement Account Number and Environment Type to Telemetry Profile 257 Identify the Entitlement Account Number 258 Update the Management Cluster 260 Enable Identity Management After Management Cluster Deployment 261 Overview 261 Obtain Your Identity Provider Details 261 Generate a Kubernetes Secret for the Pinniped Add-on 261 Check the Status of the Identity Management Service 264 (OIDC Only) Provide the Callback URI to the OIDC Provider 264 Generate a Non-Admin kubeconfig 264 Create Role Bindings for Your Management Cluster Users 264 Enable Identity Management in Workload Clusters 264 Connect to and Examine Tanzu Kubernetes Clusters 266 Obtain Lists of Deployed Tanzu Kubernetes Clusters 266 Export Tanzu Kubernetes Cluster Details to a File 267 Retrieve Tanzu Kubernetes Cluster kubeconfig 268 Authenticate Connections to a Workload Cluster 270 Configure a Role Binding on a Workload Cluster 272 Examine the Deployed Cluster 273 Access a Workload Cluster as a Standard User 276 Scale Tanzu Kubernetes Clusters 278 Scale Worker Nodes with Cluster Autoscaler 278 Scale a Cluster Horizontally With the Tanzu CLI 279 Scale a Cluster Vertically With kubectl 279 Update and Troubleshoot Core Add-On Configuration 280 Default Core Add-On Configuration 280 Updating and Troubleshooting Core Add-on Configuration 281 Tanzu Kubernetes Cluster Secrets 287 Update Management and Workload Cluster Credentials (vSphere) 287 Update Workload Cluster Credentials (vSphere) 287 Trust Custom CA Certificates on Cluster Nodes 288 Configure Machine Health Checks for Tanzu Kubernetes Clusters 289 About MachineHealthCheck 289 Create or Update a MachineHealthCheck 290 Retrieve a MachineHealthCheck 291 Delete a MachineHealthCheck 291 Back Up and Restore Clusters 291 Setup Overview 291 Install the Velero CLI 292 VMware, Inc. 7 VMware Tanzu Kubernetes Grid Set Up a Storage Provider 293 Deploy Velero Server to Clusters 294 vSphere Backup and Restore 296 AWS Backup and Restore 296 Azure Backup and Restore 297 Delete Tanzu Kubernetes Clusters 298 Step One: List Clusters 298 Step Two: Delete Volumes and Services 298 Step Three: Delete Cluster 300 7 Deploying and Managing Extensions and Shared Services 301 Locations and Dependencies 301 Preparing to Deploy the Extensions 302 Download and Unpack the Tanzu Kubernetes Grid Extensions Bundle 302 Install Cert Manager on Workload Clusters 303 Create a Shared Services Cluster