Load Balancing Topics

1. What is load balancing? 2. Load balancing techniques 3. Load balancing strategies 4. Sessions 5. Elastic load balancing What is load balancing?

“load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload.”

─Wikipedia Load Balanced Servers

Internet

Server 1 Server 2 … Server N Load Balancing + Failover Load Balancing + Failover + Cloud • Round robin DNS • DNS global server load balancing • Layer 4 load balancer • Web switch • Reverse proxy

Load balancing techniques Round Robin DNS Configure multiple A records for single name – Return next IP address from circular list – Some servers may try to return closest IP address. Problems – Clients cache addresses. – No awareness of availability status of addresses. Round Robin DNS Entry

> dig google.com

;; ANSWER SECTION: google.com. 21 IN A 74.125.225.18 google.com. 21 IN A 74.125.225.19 google.com. 21 IN A 74.125.225.20 google.com. 21 IN A 74.125.225.16 google.com. 21 IN A 74.125.225.17

DNS Global Server Load Balancing

1. Client makes a DNS query to ISP DNS server. 2. ISP DNS server looks up DNS records. 3. GSLB receives DNS query. 4. GSLB algorithm examines site health and latency to determine which site best. 5. GSLB returns IP address to client of site that is available and has lowest latency.

Problem: DNS servers and clients cache responses for long periods of time. Application Presentation Layer 4 Load Balancing Switch Session Transport Network Network Address Translation (NAT) for LB Data Link – One external IP address Physical – Switch maps incoming connections to one of the pool of servers. – Switch maps outgoing connection source IP address to IP address of the switch. Additional features – Better load balancing algorithms – Detection of availability status of servers

Application Presentation Web Switch Session Transport Network AKA content switch or application switch Data Link Physical

Balances load at application layer – Multiple load balancing algorithms – Handles SSL for servers – May also perform compression – May also add firewall features http://www.f5.com/pdf/products/big-ip-platforms-ds.pdf Reverse Proxy

• Proxy server has network identity • Forwards requests to backend web servers • Features can include caching, compression, SSL Open Source Load Balancers

Perlbal – used by LiveJournal, TypePad

HAProxy – used inside hw + sw load balancers

Varnish – HTTP accelerator, caching + balancing

Pound – Security-focused HTTP accelerator

mod_proxy_balance – Apache module

NGinx – lightweight, high performance web proxy • Round robin • Least connections/time • Predictive • Random • Weighted strategies

Load balancing strategies Round Robin

Server 1

Server 5 Server 2

Server 4 Server 3 Least Connections / Response Time

Send requests to server with the least number of connections or lowest response time. – Good for balancing between requests with different requirements or servers with different performance levels. – Problems can arise with multiple load balancers making decisions in parallel. – Weights can be added for manual tweaking. Predictive

Round robin or least connections with additional heuristics to compensate for information staleness issue arising from many short rapid transactions. Random

• Either select server at random or combine with resource-based algorithm to deal with information staleness problems. • Weighted random adds manual constant to probability of choosing particular servers. – A dual-core isn’t twice as fast as a single CPU. – Requires trial and error experimentation. Sessions The Problem

User interactions with a server have a certain amount of state. – Authentication – Current stage of transaction – Shopping carts, etc. How can the application preserve state when the load balancer sends next request to a different server? Session Stickiness

Stickiness requires session-aware load balancer – Ensures that future requests from same session always go to same server. Problems – Lack of failover since requests tied to single server – Difficult to allocate resources effectively since sessions vary tremendously in duration and size Client-side State

Why not store all state in client cookies? – Insufficient client storage (ameliorated by HTML5) – Insecure (can’t trust client to control prices, id) Solution – Store frequently accessed low security data in client cookies. – Store rarely accessed data in central backend storage and take performance hit to access when necessary. Elastic load balancing

Elastic Load Balancing

Distributes traffic across EC2 instances – Instances can be located across multiple AZs – Supports any TCP based protocol – Monitors instance health and will not distribute traffic to unhealthy instances – Supports SSL termination – Supports session stickiness – Provides metrics to CloudWatch

ELB Algorithms

1. Weighted round robin – Sends new request to instance handling smallest number of requests. – Round robin choice if multiple instances have same smallest number of requests. – Can configure health checks to prevent ELB from sending requests to unresponsive instances. 2. Session sticky – Weighted round robin, but all requests that are part of a session go to the same server. ELB Lifecycle

1. Create an ELB – List of AZs – Parameters for health check – List of listeners 2. Add instances to ELB – By instance ID – ELB will track status: InService, OutOfService 3. Advertise public DNS name of ELB 4. Modify number of instances to match traffic – Or setup CloudWatch/AutoScale to handle for you 5. Delete ELB when unneeded ELB, CloudWatch, and AutoScale CloudWatch

Monitoring service for EC2 – CPU utilization, Data transfer, Storage usage Pricing – Basic Monitoring with 5 minute granularity free – Detailed Monitoring (1 minute) for 1.5₵ per hour – 10₵ per alarm after first 10 alarms CloudWatch Terminology A namespace represents a source of data AWS/EC2 AWS/ELB A measure is a raw, observed data value One minute’s worth of observation A unit is an attribute of a measure Seconds, %, bytes, bits, counts, bytes/s, bits/s A dimension is a refined view of a type of data AvailabilityZone, ImageType, InstanceID, … A metric is a stored, processed measure A statistic is a computed attribute of a metric Minimum, maximum, average, sum AutoScaling

AutoScaling Group – Set of EC2 instances that should scale together Triggers – Scale on CloudWatch alerts – Scale on time-based schedule – Fixed number of healthy instances Examples – Add 3 instances if CPU > 50% – Remove 3 instances if CPU < 10% AutoScaling Setup

1. Create an ELB 2. Create AutoScaling launch configuration – ID of AMI to be launched – Instance type – Key pair to authenticate to instances – List of EC2 security groups for instances 3. Create an AutoScaling Group 4. Create a trigger for the group AutoScaling Operation

1. CloudWatch metrics specified in AutoScaling group’s trigger are retrieved at specified times. 2. Metrics checked against trigger thresholds – If metrics larger than UpperThreshold and the number of instances less than MaxSize, a scale- out event is initiated, launching new instances. – If metrics are smaller than LowerThreshold and number of instances is greater than MinSize, a scale-in event is initiated, terminated excess instances. Key Points

1. Load balancing distributes transactions across multiple servers, CPUs, links, or storage devices 2. Techniques – Round robin DNS (configuration) – DNS-based GSLB (software + configuration) – Layer 4 switch (hardware) – Web switch (hardware) – Reverse proxy (software) 3. Algorithms – Round robin – Weighted: least connections/fastest response time – Predictive heuristics – Random 4. Session stickiness References

1. Jeff Barr, Host Your Web Site in the Cloud: Amazon Web Services Made Easy, Sitepoint, 2010. 2. Theo Schlossnagle, Scalable Internet Architectures, Sams Publishing, 2007. 3. Willy Tarreau, “Making Applications Scalable with Load Balancing,” http://1wt.eu/articles/2006_lb/, 2006. 4. Pete Tenereillo, Why DNS Based Global Server Load Balancing (GSLB) Doesn’t Work, http://www.tenereillo.com/GSLBPageOfShame.htm, 2004.