A Complete Introduction to Monitoring Kubernetes with New Relic the Fundamentals You Need to Know to Effectively Monitor Kubernetes Deployments Table of Contents
Total Page:16
File Type:pdf, Size:1020Kb
White Paper A Complete Introduction to Monitoring Kubernetes with New Relic The fundamentals you need to know to effectively monitor Kubernetes deployments Table of Contents Introduction 03 Monitoring Kubernetes with New Relic 04 Getting started 04 Exploring clusters with the Kubernetes cluster explorer 05 Benefits of monitoring with the cluster explorer 06 Kubernetes Observability 07 Visualizing services 07 Monitoring cluster health and capacity 07 Correlating Kubernetes events with cluster health 12 Integrating with APM data 13 Integrating with Prometheus metrics 15 Using Prometheus data in New Relic 15 Monitoring logs in context 16 Investigating end-user experience 17 Scaling Kubernetes with Success: A Real-World Example 18 02 A Complete Introduction to Monitoring Kubernetes with New Relic Introduction Before Kubernetes took over the world, cluster adminis- • Automatic scheduling of pods can cause capacity trators, DevOps engineers, application developers, and issues, especially if you’re not monitoring resource operations teams had to perform many manual tasks availability. in order to schedule, deploy, and manage their contain- erized applications. The rise of the Kubernetes con- In effect, while Kubernetes solves old problems, it can tainer orchestration platform has altered many of these also create new ones. Specifically, adopting containers responsibilities. and container orchestration requires teams to rethink and adapt their monitoring strategies to account for the Kubernetes makes it easy to deploy and operate applica- new infrastructure layers introduced in a distributed tions in a microservice architecture. It does so by creating Kubernetes environment. an abstraction layer on top of a group of hosts, so that development teams can deploy their applications and let With that in mind, we designed this guide to highlight the Kubernetes manage: fundamentals of what you need to know to effectively monitor Kubernetes deployments with New Relic. This • Controlling resource consumption by application guide outlines some best practices for monitoring Kuber- or team netes in general, and provides detailed advice for how to • Evenly spreading application load across a host do so with the New Relic platform. infrastructure Whether you’re a cluster admin, an application developer, • Automatically load balancing requests across the an infrastructure engineer, or DevOps practitioner work- different instances of an application ing on the Kubernetes platform, by the end of this guide, • Monitoring resource consumption and resource limits you should be able to use New Relic to monitor the health to automatically stop applications from consuming too and capacity of Kubernetes components and resources, many resources and restarting the applications again correlate events in Kubernetes with contextual insights to help you troubleshoot issues, understand how to moni- • Moving an application instance from one host to tor applications running in your cluster, and know how to another if there is a shortage of resources in a host, or track end-user experience from those apps. if the host dies • Automatically leveraging additional resources made available when a new host is added to the cluster • Easily performing canary deployments and rollback But such capabilities also give teams new things to worry about. For example: • There are a lot more layers to monitor. • The ephemeral and dynamic nature of Kubernetes makes it a lot more complex to troubleshoot. 03 A Complete Introduction to Monitoring Kubernetes with New Relic Monitoring Kubernetes with New Relic In addition to providing visibility into operational data— Getting started such as the number of resources used and namespaces To get started monitoring Kubernetes with New Relic, per cluster and per pod—an important part of monitor- you’ll need to activate the Kubernetes integration by ing Kubernetes is having the ability to see the relation- deploying the newrelic-infra agent onto your Kubernetes ships between objects in a cluster using Kubernetes’ cluster. The New Relic Kubernetes integration brings in built-in labeling system. system-level metrics, allowing you to view, troubleshoot, New Relic integrates with Kubernetes in a number of ways: and alert on the most important parts of your cluster. Use the integration’s out-of-the-box dashboard to inspect • Amazon Elastic Container Service for Kubernetes a single container, or scale up to see large, multi-cluster (Amazon EKS): Amazon EKS provides Kubernetes as deployments across different Kubernetes entities, includ- a managed service on AWS. It helps make it easier to ing nodes, pods, namespaces, and containers. deploy, manage, and scale containerized applications on Kubernetes. 1. New Relic uses kube-state-metrics—a simple service • Google Kubernetes Engine (GKE): GKE provides an that listens to the Kubernetes API server and generates environment for deploying, managing, and scaling metrics—to gather information about the state of your containerized applications using Google-supplied Kubernetes objects. Install kube-state-metrics in infrastructure. your cluster: • Microsoft Azure Kubernetes Service (AKS): AKS curl -o kube-state-metrics-1.7.2.zip manages your hosted Kubernetes environment, https://codeload.github.com/kubernetes/ making it easier to deploy and manage containerized kube-state-metrics/zip/release-1.7.2 && applications without container orchestration unzip kube-state-metrics-1.7.2.zip && oc expertise. It also eliminates the burden of ongoing apply -f kube-state-metrics-release-1.7.2/ operations and maintenance by provisioning, kubernetes upgrading, and scaling resources on demand, without 2. Download the Kubernetes integration config ur - taking your applications offline. ation file: • Red Hat OpenShift: OpenShift provides developers curl -O with an integrated development environment (IDE) for https://download.newrelic.com/infrastructure_ building and deploying Docker-formatted containers, agent/integrations/kubernetes/newrelic- and then managing them with Kubernetes. infrastructure-k8s-lat e st.ya m l • Pivotal Container Service (PKS): Available as part of 3. In the configuration file, add yourNew Relic license key Pivotal Cloud Foundry or as a standalone product, PKS and a cluster name to identify your Kubernetes cluster. provides the infrastructure and resources to reliably Both values are required. Be sure to update <YOUR_ deploy and run containerized workloads across private LICENSE_KEY> with your license key and <YOUR_ and public clouds. CLUSTER_NAME> with the name of your cluster. 04 A Complete Introduction to Monitoring Kubernetes with New Relic env: ments, nodes, pods, containers, and applications. With - name: NRIA_LICENSE_KEY the cluster explorer, you will be able to easily retrieve the value: <YOUR_LICENSE_KEY> data and metadata of these elements, and understand - name: CLUSTER_NAME how they are related. value: <YOUR_CLUSTER_NAME> From the Kubernetes cluster explorer, you can: 4. Add any additional configuration, as documented in 1. Select the cluster you want to explore the Kubernetes integration instructions. 2. Filter by namespace or deployment 5. To confirm that kube-state-metrics is installed, run this command: 3. Select specific pods or nodes for status details kubectl get pods --all-namespaces | grep The cluster explorer has two main parts: kube-state-metrics 1. A visual display of the status of a cluster, up to 24 nodes. 6. To create the DaemonSet, run this command: Within the visual display, the cluster explorer shows kubectl create -f newrelic-infrastructure the nodes that have the most issues in a series of four -k8s-lat e st.ya m l concentric rings: 7. Confirm that the DaemonSet has been created • The outer ring shows the nodes of the cluster, with successfully by looking for newrelic-infra in the results each node displaying CPU, memory, and storage generated by this command: performance metrics. kubectl get daemonsets • The next innermost ring displays the distribution and status of the non-alerting pods associated with 8. Go to one.newrelic.com, and select the Kubernetes that node. cluster explorer launcher. • The third innermost ring displays the pods on alert Exploring clusters with the and that may have health issues even if they are Kubernetes cluster explorer still running. • Finally, the innermost ring displays pods that are pending or that Kubernetes is unable to run. 2. You can select any pod to see its details, such as its namespace, deployment, containers, alert status, CPU usage, memory usage, and more. 3. The cluster explorer node table displays all the nodes of the selected cluster/namespace/deployment, and can be sorted according to node name, node status, pod, pod status, container, CPU% vs. Limit and MEM% vs. Limit. New Relic’s Kubernetes cluster explorer provides a multi-dimensional representation of a Kubernetes cluster from which you can explore your namespaces, deploy- 05 A Complete Introduction to Monitoring Kubernetes with New Relic Benefits of monitoring with the cluster explorer The cluster explorer expands the Kubernetes monitor- ing capabilities already built into the New Relic platform. Use the cluster explorer’s advanced capabilities to filter, sort, and search for Kubernetes entities, so you can better understand the relationships and dependencies within an environment. The default data visualizations of your clus- ter provides a fast and intuitive path to getting answers and understanding their Kubernetes environments, so you can contain the complexity associated