Packet Capture and Analysis in Hybrid Environments Lawrence Miller Inside the Guide • Discover why packet capture and analysis of on- premises and cloud applications in modern hybrid environments is so challenging • Learn how to analyze packets at the Transport and Application Layers to uncover the root cause of issues • Explore how putting the right tools in the hands of the right people can help make your entire IT team more effective THE GORILLA GUIDE TO... Packet Capture and Analysis in Hybrid Environments Express Edition By Lawrence Miller Copyright © 2020 by ActualTech Media All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the publisher except for the use of brief quotations in a book review. Printed in the United States of America. ACTUALTECH MEDIA 6650 Rivers Ave Ste 105 #22489 North Charleston, SC 29406-4829 www.actualtechmedia.com PUBLISHER’S ACKNOWLEDGEMENTS EDITOR Keith Ward, ActualTech Media PROJECT MANAGER Wendy Hernandez, ActualTech Media EXECUTIVE EDITOR James Green, ActualTech Media LAYOUT AND DESIGN Olivia Thomson, ActualTech Media WITH SPECIAL CONTRIBUTIONS FROM Paul R. Dietz, Riverbed Stephen Creel, Riverbed Heidi Gabrielson, Riverbed Kowshik Bhat, Riverbed TABLE OF CONTENTS Introduction 7 Chapter 1: A Primer on Packets 8 Looking at Packets and Packet-Switched Networks 8 Understanding the Open Systems Interconnection (OSI) Model and Packet Encapsulation 11 Addressing Modern Packet Capture and Analysis Challenges 13 Chapter 2: A Packet’s Tale: The Case of the Slow Web App 16 The Game Is Afoot 16 Examining Packets at the Transport Layer 17 Analyzing Packets at the Application Layer 24 Putting It All Together: Time to Find Out Who the Real Villain Is 33 Chapter 3: Force Multipliers: Making Your Smart People More Effective 35 World-Class Packet Analysis 35 Beyond Capturing Packets 40 Performing Magic with AppResponse 40 Delivering Better Business Outcomes 43 Get More Production and Less Frustration 45 CALLOUTS USED IN THIS BOOK In the School House callout, you’ll gain insight into topics that may be outside the main subject but are still important. This is a special place where you can learn a bit more about ancillary topics presented in the book. When we have a great thought, we express them through a series of grunts in the Bright Idea section. Takes you into the deep, dark depths of a particular topic. Discusses items of strategic interest to business leaders. ICONS USED IN THIS BOOK DEFINITION Defines a word, phrase, or concept. KNOWLEDGE CHECK Tests your knowledge of what you’ve read. PAY ATTENTION We want to make sure you see this! GPS We’ll help you navigate your knowledge to the right place. WATCH OUT! Make sure you read this so you don’t make a critical error! TIP A helpful piece of advice based on what you’ve read. INTRODUCTION Today’s complex application architectures span hy- brid environments consisting of multiple public and private clouds, on-premises data centers hosting hun- dreds (and even thousands) of microservices, contain- ers, and virtual machines—all connected across dis- parate wide-area network (WAN) and internet links. Identifying and troubleshooting performance issues without complete visibility across the entire architec- ture and using siloed tools and data causes longer res- olution times, breached service-level agreements, and miscommunication and finger pointing across differ- ent IT teams. Ultimately, this untenable situation leads to lost user productivity and frustration that negatively impacts the business and the bottom line. The Gorilla Guide To…® (Express Edition) Packet Capture and Analysis will help you understand why identifying and resolving application and network per- formance issues has become so challenging for even the best and brightest of your IT teams. You’ll learn how to take a more evidence-based approach to diagnosing and analyzing performance issues across the entire stack and how the right packet capture and analysis tools can help your IT teams work together more effectively to resolve performance issues faster. CHAPTER 1 A Primer on Packets Software may be “eating the world,” but it’s the net- work that connects software applications to services, data, and users. In this chapter, we’ll look at the chal- lenges of packet capture and analysis in a hybrid world consisting of on-premises data centers, public/private clouds, and remote and mobile users. Later chapters will take us through a sample exercise in troubleshoot- ing a network, and describe some standard tools in packet and transaction analysis through services of- fered by Riverbed. Looking at Packets and Packet- Switched Networks Before the advent of packet-switched networks, cir- cuit-switched networks provided connectivity, lit- erally, from point A to point B. The principle of a cir- cuit-switched network was captured in Mayberry’s switchboard operator Sarah on The Andy Griffith Show, manually connecting calls, say, between Sheriff Andy Taylor and Floyd’s barbershop. Each phone line (circuit) connected exactly two endpoints (phones). To connect to a different endpoint (for example, if Andy wanted to call Aunt Bee), Sarah would need to break Andy’s con- nection to Floyd and switch it to Aunt Bee. Although this is a greatly simplified example of circuit-switched net- works, it illustrates the basic concept and some of the challenges associated with circuit-switched networks, including high costs (to maintain multiple point-to- point connections) and limited resiliency (each circuit is a single point of failure). The first packet-switched network, ARPANET, was originally developed for the U.S. Department of Defense in 1969, specifically to address the issue of resiliency in circuit-switched networks. In a packet-switched net- work, endpoints (such as servers and client worksta- tions) can communicate over numerous communica- tions links, typically forming a mesh of available links. This provides numerous ways to connect point A to point B. Depending on your business and technical re- quirements, different links may be dynamically selected for individual transmissions based on factors such as: • Speed • Latency • Cost PACKET CAPTURE AND ANALYSIS IN HYBRID ENVIRONMENTS 9 • Security • Utilization • Availability Better still, in a packet-switched network the traffic is segmented into packets (consisting of a header and payload) that don’t necessarily have to take the same network path from point A to point B. The header infor- mation in the individual packets allows the endpoints to assemble the packets in the correct order when they’re received. An internet Protocol (IP) header contains im- portant information used in packet capture and analy- sis, including: • Version • IP Header Length • Type of Service • Total Length • 16-bit Identification • Flags • Fragment Offset • Time to Live • Protocol PACKET CAPTURE AND ANALYSIS IN HYBRID ENVIRONMENTS 10 • Header Checksum • Source and Destination IP Address • Options (if any) Understanding the Open Systems Interconnection (OSI) Model and Packet Encapsulation The OSI model helps network engineers understand and troubleshoot complex networking issues by separating the network into simpler functional components. This model consists of seven layers that describe how sys- tems and applications communicate and interoperate on a packet-switched network (see Figure 1). Data packets in the OSI model are broadly referred to as protocol data units (PDUs). The OSI model promotes interoperability by only requiring a layer to be able to communicate with the layer directly above and below it. This is accomplished through a process known as en- capsulation. Each layer adds a header (and a footer, too, just in the case of Layer 2) with delivery instructions and information about the data, which is called the pay- load. A PDU at the Physical Layer (Layer 1) is known as a bit or symbol. When the Layer 1 PDU is passed up to the Datalink Layer (Layer 2), the Datalink Layer treats the entire PDU (header and payload) as its own payload. PACKET CAPTURE AND ANALYSIS IN HYBRID ENVIRONMENTS 11 Humancomputer interaction layer, where applications can access the APPLICATION LAYER 7 network services nsures that data is in a usable format and is where data encryption PRESENTATION LAYER 6 occurs Maintains connections and is responsible for controlling ports and SESSION LAYER 5 sessions rovides transparent, reliable data transport and endtoend TRANSPORT LAYER 4 transmission control Decides which physical path the data NETWORK LAYER 3 will take Handles pointtopoint DATALINK LAYER 2 synchroniation and error control Transmits raw bit stream over the physical medium PHYSICAL LAYER 1 Figure 1: The OSI model Layer 2 adds its own header and layer, thus creating a Datalink Layer PDU, known as a frame. This process is repeated at the Network Layer (Layer 3), which adds a header to create the Network Layer PDU, known as a packet. The encapsulation process continues at the PACKET CAPTURE AND ANALYSIS IN HYBRID ENVIRONMENTS 12 Transport Layer (Layer 4), where the PDU is known as a segment or datagram, all the way to the Application Layer (Layer 7). In the upper layers of the OSI model (Layers 5 through 7), PDUs are simply known as PDUs. On the re- ceiving end, the individual headers are stripped off, one by one, at each layer (effectively reversing the process) as a PDU passes down the application stack. In this Guide, we primarily focus on the following layers: • Layer 3: The internet Protocol (IP) • Layer 4: Mostly TCP and UDP • Layer 7: Applications, especially Web apps Addressing Modern Packet Capture and Analysis Challenges To gain visibility into the raw data (packets) travers- ing enterprises’ systems and networks, network engi- neers, infrastructure and operations (I&O) teams, and application owners have traditionally used network performance management (NPM) and application per- formance management (APM) tools. The analyses these tools provide can help optimize performance and trou- bleshoot issues.