The Pennsylvania State University

The Graduate School

School of Science, Engineering, and Technology

CAFFEINE: CONGESTION AVOIDANCE FOR FAIRNESS & EFFICIENCY IN NETWORK ENTITIES

A Thesis in

Computer Science

by

Pattanapoom Phinjirapong

©2015 Pattanapoom Phinjirapong

Submitted in Partial Fulfillment of the Requirements for the Degree of

Master of Science

August 2015 ii

The thesis of Pattanapoom Phinjirapong was reviewed and approved* by the following:

Jeremy J. Blum Associate Professor of Computer Science Thesis Advisor

Thang N. Bui Associate Professor of Computer Science Chair, Mathematics and Computer Science Programs

Sukmoon Chang Associate Professor of Computer Science

Omar El Ariss Assistant Professor of Computer Science

Linda M. Null Associate Professor of Computer Science Associate Chair, Mathematics and Computer Science Programs

*Signatures are on file in the Graduate School.

iii

Abstract Software Defined Networking (SDN) enables the separation of the data forwarding plane and the control plane in network switches and routers. The control plane functionality is provided by a centralized network operating system. By moving this functionality to software, a range of new capabilities are possible in networks. This thesis describes and evaluates a new algorithm, called Congestion Avoidance For Fairness & Efficiency In Network Entities (CAFFEINE) that uses SDN provided capabilities.

CAFFEINE provides dynamic routing in network topology for long-lived network traffic with goals of fully utilizing network capacity and maintaining fairness among traffic classes through Quality of Service (QoS) guarantees. CAFFEINE is designed to spread these long-lived traffic flows throughout an organization’s network. To achieve the goals, CAFFEINE periodically polls network devices in order to monitor the current network state. CAFFEINE routes traffic using a modified widest path algorithm, with path length constraints, to find a path in the network with maximum available bandwidth, and then it sets QoS guarantees on that path.

From a service view, CAFFEINE consists of a network monitoring service, a rate limiting service, a queue management service, and a flexible routing service. The network monitoring service is responsible for polling the state of the network from the controller in SDN and calculating aggregate information. The flexible routing service periodically receives the aggregated network state information and re-calculates the new routes. The rate limiting service also receives that network state information and uses this state to set a minimum guaranteed bandwidth and a maximum allowed bandwidth for every class of traffic on each output port of every router in the topology. The queue management service then sets rules in the network switches to assign the new flow to a particular path in the network.

The evaluation of CAFFEINE is simulated on virtual networks using Mininet, virtualization software that can simulate hosts and switches. The testing compares CAFFEINE against a conventional routing scheme that uses shortest path routing. Multiple tests, which represent variety of networks, show that CAFFEINE is able to utilize network spare capacity and can prioritize network flows from preconfigured rules. CAFFINE allows network administrators to categorize different flows to different priorities for weighted fairness and gain better throughput. iv

Table of Contents List of Figures ...... v List of Tables ...... vi List of Algorithms ...... vii Chapter 1: Introduction ...... 1 Chapter 2: Preliminaries ...... 4 2.1 Introduction ...... 4 2.2 Shortest Path Uniform Routing ...... 4 2.3 Pre-SDN Alternate Approaches ...... 5 2.4 SDN Approaches ...... 5 Chapter 3: Methodology ...... 10 3.1 Overview of CAFFEINE ...... 10 3.2 CAFFEINE Network Services ...... 11 3.2.1 Network Monitoring Service ...... 13 3.2.2 Rate Limiting Service ...... 14 3.2.3 Flexible Routing Service ...... 21 3.2.4 Queue Management Service ...... 23 Chapter 4: Evaluation ...... 24 4.1 Simulation System ...... 24 4.1.1 Assumptions ...... 24 4.1.2 Simulation Details ...... 24 4.1.3 Fairness Testing ...... 26 4.1.4 Utilization Testing ...... 30 4.1.5 Variable Link Capacity Testing ...... 35 Chapter 5: Conclusions and Future Work ...... 39 5.1 Evaluation of CAFFEINE ...... 39 5.2 Future Work ...... 39 References ...... 41

v

List of Figures Figure 1: Data flow diagram of CAFFEINE ...... 12 Figure 2: Example scenario of weighted fairness ...... 18 Figure 3: Example scenario for minimizing wasted effort ...... 19 Figure 4: The topology of fairness testing ...... 26 Figure 5: Comparison graph of downloaded data in fairness testing ...... 27 Figure 6: Comparison graph of total downloaded data in fairness testing ...... 27 Figure 7: Comparison of average bandwidth in fairness testing ...... 28 Figure 8: The topology of utilization testing ...... 30 Figure 9: Comparison of possible paths for h1, h2 and h3 ...... 30 Figure 10: Comparison of possible paths for h4, h5 and h6 ...... 31 Figure 11: Comparison of possible paths for h7, h8 and h9 ...... 31 Figure 12: Comparison of downloaded data in utilization testing ...... 32 Figure 13: Comparison of total downloaded data in utilization testing ...... 33 Figure 14: Comparison of average bandwidth in utilization testing ...... 34 Figure 15: The topology of variable link capacity testing ...... 35 Figure 16: Comparison of downloaded data in variable link capacity testing ...... 36 Figure 17: Comparison of total downloaded data in variable link capacity testing ...... 37 Figure 18: Comparison of average bandwidth in variable link capacity testing ...... 38

vi

List of Tables Table 1: Example work of rate limiting service ...... 20 Table 2: Downloaded data in fairness testing ...... 26 Table 3: Average bandwidth in fairness testing ...... 28 Table 4: Downloaded data in utilization testing ...... 32 Table 5: Average bandwidth in utilization testing ...... 34 Table 6: Downloaded data in variable link capacity testing ...... 36 Table 7: Average bandwidth in variable link capacity testing ...... 38

vii

List of Algorithms Algorithm 1: Rate Limiting Algorithm ...... 15 Algorithm 2: Counting Function ...... 16 Algorithm 3: Total Weight Calculating Function ...... 16 Algorithm 4: Min-bandwidth Calculating Function ...... 17 Algorithm 5: Max-bandwidth Determining Function ...... 18 Algorithm 6: Widest Path Algorithm ...... 22 Algorithm 7: Path Determination Algorithm ...... 23 1

Chapter 1: Introduction Convention networking lets switches and routers work on their own. Routing algorithms and information exchanging are performed autonomously by these devices. They can decide how to forward data packets to destinations based on the exchanged information. However their software is bound to the firmware of manufacturers and discourages researchers from developing new algorithms and new protocols.

With the development of SDN (Software Defined Networking), network management is becoming more flexible. The separation of the data and control planes enables the centralized collection and analysis of information from each switch, which can be used to optimize network performance in novel ways.

In recent years, SDN has moved rapidly from research projects to commercial implementations. For example, even before commercial SDN devices were available, Google developed custom hardware and custom software to support its data center backbone (Jain, et al. 2013). The resulting SDN- based backbone enabled link utilization to increase to almost 100 percent, with a 66 percent cost savings over traditional layer 2 and layer 3 technologies. Facebook has also developed a custom SDN switch, called Wedge, using the Linux FBoss operating system (Facebook 2014). This project is included in the open source community, Open Compute Project, which focuses on technologies for server, storage, and data center hardware.

This thesis proposes an SDN-based algorithm suitable for networks with traffic having priority levels and long-lived network traffic. The algorithm uses OpenFlow-provided functionality to implement both dynamic routing and traffic shaping with goals of improving network throughput and ensure fairness among traffic classes. The routing protocol presented in this thesis uses a widest-path algorithm in an attempt to spread traffic throughout the network to improve utilization of available resources. The goal of traffic shaping is to provide fairness among traffic classes via Quality of Service (QoS) guarantees that allocate bandwidth using a prioritized multilevel max-min fairness algorithm.

A traffic flow is defined as a one-way stream of data packets from a source computer to a destination computer. In addition, priority represents the need of bandwidth on different types of traffic flows. In order to maximize network throughput and minimize latency for traffic flows, an 2

algorithm called Congestion Avoidance For Fairness & Efficiency In Network Entities (CAFFEINE) is designed with the details in section 3.1.

The traffic engineering protocol presented here uses two principal OpenFlow mechanisms in order to achieve these goals. The first mechanism is to select less congested routes for new flows. The CAFFEINE protocol periodically collects the current state of the network in a centralized network controller. When a new flow reaches a boundary router, CAFFEINE uses this knowledge of the network state to choose a less congested route, providing better throughput at the client and server than a fixed routing approach.

In addition, a second mechanism causes all network flows to be processed by logical queues inside network switches. By assigning queues to the flows, the system is able to provide an upper bound limit and lower bound guarantee on the bandwidth the network flows can use. The lower bound guarantee is set in order to provide fairness among different network traffic classes. The upper bound is set to use spare capacity while minimizing the likelihood of the wasted effort due to dropped packets.

Many types of network traffic have bursty characteristics. By setting a minimum bandwidth for each logical queue, flows are guaranteed this minimum bandwidth, even in the face of bursty competing traffic from other sources, and they avoid bandwidth starvation. In addition, each logical queue has a maximum bandwidth set. If the maximum bandwidth limits are set to the minimum guaranteed bandwidth, then flows would not be able to use spare capacity in the network. Without a maximum bandwidth limit, if a downstream link is congested, then using spare upstream capacity could be wasted because these packets might ultimately be dropped. CAFFEINE attempts to set the maximum bandwidth limit such that a network flow can use spare capacity on a link as long as it is unlikely to lead to congestion on downstream links.

In order to evaluate the effectiveness of CAFFEINE, this thesis simulates its use in a packet switch under enterprise network environment, in which flows between applications have different characteristics in terms of bandwidth needed and congestion sensitivity. In order to provide a scalable solution, instead of handling each particular flow uniquely, priority classes are introduced to group similar network flows together. Each group is then managed separately. For example, media streaming on Real-time Transport Protocol (RTP) is classified as high priority (Schulzrinne, et al. 2003). On the other hand, file downloading on File Transfer Protocol (FTP) is classified as lower priority traffic (Postel and Joyce 1985). This is because FTP is not as sensitive to bandwidth 3

stability as RTP. The rationale behind this classification is that web traffic could be viewed as external, customer driven traffic, with more restrictive latency requirements.

From the results of evaluation, CAFFEINE satisfies both the goal of full utilization and fairness. In terms of full utilization, CAFFEINE benefits from spare capacity that is ignored by conventional routing schemes. The throughput is improved by between 19% and 28% in a range of scenarios. In terms of fairness, CAFFEINE limits the bandwidth of flows according to preconfigured priorities. Thus, it can manage the priority of flows as expected. 4

Chapter 2: Preliminaries

2.1 Introduction A routing algorithm finds the path through a network to deliver data from sender to receiver, when the sender cannot reach the receiver directly. Initial routing algorithms were built into router firmware often following the shortest path algorithm, where the path costs can be defined based on the number of hops or congestion on links. Later, router hardware evolved to provide some level of improvement in throughput and quality of service/fairness guarantees. The throughput objective has a clear motivation, but the weighted fairness goal does not always imply every host should get the same amount of shared bandwidth. Rather, there are different classes of customers and different applications of networking that require different priorities.

2.2 Shortest Path Uniform Routing Initially, routing algorithms followed preconfigured rules that were set manually by an administrator, an approach that suffered from scalability challenges. The challenges were solved in part by splitting a network into components called autonomous systems (AS’s). Typically the routers in an AS run two types of routing algorithms, one for packets that are staying within the AS (intra-AS routing), and one for packets that are destined for another AS (inter-AS routing). There are two major intra-AS routing protocols and one major inter-AS routing protocol that are widely used.

Routing Information Protocol (RIP) is an intra-AS routing algorithm that works by letting routers periodically exchange distance vectors between neighbor routers in the same AS (Hedrick 1988). The exchanged distance vectors contain the number of hops needed to get to each destination via the neighbor. When a packet arrives for a destination, the router forwards it to the neighbor that is the closest to the destination subnet based on these distance vectors.

Open Shortest Path First (OSPF) is another commonly used intra-AS routing algorithm. In OSPF, each router exchanges link-state information with every router in the same AS (Moy 1998). This provides a complete view of the topology for each router. The exchanged information is then processed distributively at each router using Dijkstra’s shortest path algorithm to find a shortest- path tree to all subnets from each router (Dijkstra 1959). In addition, unlike RIP, OSPF can minimize either the number of hops or some other cost value. This flexibility allows the administrators to discourage the selection of paths that pass through low-bandwidth links or encourage the use of high-bandwidth links. OSPF is also capable of using multiple same-cost paths. 5

These two approaches, RIP and OSPF, were designed based on routing by destination only, and focus on only one metric, shortest path, which effectively limits the possible paths taken by packets for a given destination.

Border Gateway Protocol (BGP) can work for intra-AS as Internal BGP and inter-AS as External BGP (Rekhter, Li and Hares 2006). BGP is widely used by service providers nowadays because of its ability to allow a boundary switch, a switch that connects ASs together, to exchange information for determining a routing and reduce the works of exchanging information between switches of different ASs. Contrary to the concept of AS, BGP still needs to be configured manually in order to bridge between ASs.

2.3 Pre-SDN Alternate Approaches Since the forwarding in previous approaches is based on the destination IP address, packets destined for a single destination typically follow a single path in the network. To spread traffic throughout the network, Multiprotocol Label Switching (MPLS) (Rosen, Viswanathan and Callon 2001) was designed as an enhancement to conventional routing. MPLS adds an additional label field to packet which is then used for routing purposes. Not only can this approach speed up the look up time required to determine the output port on a router, it can also be used to create a virtual circuit that allows packets to travel on multiple routes that lead to the same destination.

The flexibility in routing that comes from the approaches such as MPLS allows for more efficient network management because it enables routing based on the current network traffic (Wang and Wang 1999). These approaches can improve network utilization over the approaches that do not consider current network state. Wang and Wang investigated a routing algorithm that used heuristics to spread the traffic over the network. With the methods used to identify bottleneck links and bottleneck demands, their algorithm minimizes congestion and maximizes the potential for traffic growth.

2.4 SDN Approaches Researchers have come with the idea of Software Defined Networking (SDN) to separate the control plane and the data plane from switches, rather than put them together on switch/router firmware. The control plane is a part of switch/router architecture that determines the network map and decides the route path for the incoming data packet. In addition, the data plane, also called the forwarding plane, is a part that forwards a data packet based on the decision of the control plane. Since the control plane is no longer dependent on the switch manufacturers, SDN encourages 6

researchers to develop new algorithms and protocols for networking by moving the software to the central controller, which is programmable. It also enables centralization of complex algorithms in software rather than requiring the implementation of distributed algorithms in network hardware. This concept reduces complexity in the process of design and leads to faster development of new routing algorithms. A routing algorithm, which is in the control plane, is installed on the central controller rather than on the switches as in the conventional approach. So the switches don’t know how to handle a data packet by themselves. When the first data packet of a new flow reaches the first switch, which is connected with the source machine, this switch needs to ask the central controller how to handle this traffic flow. Depending on the decision of the central controller, the data packet from this flow might be dropped for security purpose or routed to a different destination for load balancing purpose. Under normal circumstances, the central controller determines a path needed to forward the data packets to the destination and gives directions to every switch along the path. The following data packet is forwarded with the action given from the central controller without further input from the switches. So the overhead delay is occurred to the first data packet of a flow only and does not cost too much idle time for traffic flows. OpenFlow has gained popularity among researchers more than other SDN standard implementations.

SDN has enabled a wide range of new approaches to networking. For example, researchers have conducted an early experiment of SDN network in a campus environment at Stanford University (Casado, et al. 2009). The resulting network routing algorithms were policy aware and enforced a strong binding between a packet and its origin. Similarly, the 4D project also utilized a centralized controller for its network management (Greenberg, et al. 2005). The project demonstrated the ease with which new policies and protocols can be realized in an SDN network. Moreover, the tests conducted demonstrated the scalability of SDN approaches. In this network, a single controller without replication handled up to 6000 hosts without significantly increasing latency.

SDN has also been used to spread network traffic over multiple paths in a network. Google developed and implemented an SDN-based approach for the network backbone that links its data centers (Jain, et al. 2013). This SDN implementation uses a combination of customized applications, customized hardware, multi-path routing, flow group controlling, priority classes of data, flexible path processing, and both centralized and independent traffic engineering in order to improve utilization and avoid packet loss. Normally, the WAN links, which were made before the SDN approach, were provisioned such that they experienced 30% to 40% of average utilization to prevent packet loss. This over-provisioning comes with high cost for infrastructures. 7

In the SDN approach, Google classified data into classes, distinguished by characteristics of its application owners. Some classes are more sensitive to delays than others. The SDN approach provides the capability of adapting low priority traffic to use the available bandwidth, without adversely affecting high priority and latency sensitive traffic. This approach has driven link utilization up to near 100% for many of their links and increased overall average utilization to 70%.

CAFFEINE borrows some inspiration from the Google SDN approach, particularly in that it groups flows into classes and performs multiple path routing. However, rather than basing the approach on complex customized hardware and software, which would incur significantly higher cost, CAFFEINE uses existing SDN hardware, including Open vSwitch with its standard features in OpenFlow. The result is a potentially design that has a cost/performance tradeoff compared to the idea of Google’s approach.

Multiple path routing can be significantly more effective than conventional single path routing in the face of unexpected traffic changes (Zhang, et al. 2014). Through the use of conventional routing based on the shortest path to a destination and a flexible augmented routing for a minority of traffic, Zhang et al. were able to improve the utilization and make a set of rules to use less TCAM (ternary content addressable memory) resources, the expensive, but extremely fast memory used in routers. Normally, explicit routing is complex and requires large TCAM, but the authors combine destination-based routing with few explicit routing entries to gain advantage on both high speed memory and abundant memory which is slower. CAFFEINE follows this approach in terms of flexible routing, but it uses only the available bandwidth to find paths, and provides weighted fairness guarantees.

The flexibility of software-based routing in SDN has enabled a range of other applications, including in-network load balancing capabilities (Wang, Butnariu, and Rexford 2011). The significant challenge for this work was the development of forwarding rules that perform the load balancing while minimizing the space of rules needed to be installed in the limited TCAM memory using a combination of Microflow rules and wildcard rules. Microflow rules are the rules that match one specific flow, i.e., one four-tuple consisting of source IP address, destination IP address, source port, and destination port. On the other hand, wild card rules refer to a rule that has wild cards that match on any values. By using more wild card rules, the authors’ approach allows more of the rules to be resident in fast TCAM memory because they are less complex and take less space. Since OpenFlow supports neither hash-based routing nor wild-card matching on the higher-order bits in IP addresses, the authors developed, instead, an algorithm that uses small number of wild-card 8

rules and some Microflow rules while partitioning entire client IP address space for load balancing purposes. CAFFEINE proposes a similar approach to manage a large number of flows by grouping flows into classes.

QoS guarantees can both reserve a resource for a flow and limit the amount of resources that a flow takes. SDN-based QoS approaches have investigated both the provision of QoS guarantees and prevent any particular traffic flow from causing resource starvation to other requestors.

Combining throughput and QoS guarantees present a particular challenge. Since fairness and throughput are conflicting objectives, Google’s approach attempted to maximize a weighted combination of these two factors (Danna, Mandal, and Singh 2012). The authors used a linear programming-based approach to solve the resulting optimization problem. Nace and Pioro described the notion in detail of max-min fairness (MMF) and gave direction for MMF algorithms along with some of its applications in routing (Nace and Pioro 2009). An MMF allocation concept works in the way that bandwidth is allocated to all requestors, which is fair in the sense that the only way to increase a requestor’s allocation would be to increase the allocation of all requestors with an equal allocation. The MMF algorithm proceeds in rounds. In the first round, resources on bottleneck links are divided equally among requestors. In subsequent rounds, if there is spare capacity, the capacity on bottleneck links is divided equally among requestors who still have not had their demand satisfied. Then, the left-over capacity from the second round is processed in the same manner in the third round and so on until no more resource are left to be shared or every requestor has met its capacity.

Other SDN-based QoS approaches provide the insights into how CAFFEINE could be integrated into a production SDN network. Jeong et al. proposed a QoS-aware Network Operating System (QNOX) (Jeong, Kim, and Kim 2012). QNOX extends Network Operating System (NOX), by implementing four new modules on top of NOX. These modules include a Service Element, Control Element, Management Element, and Cognitive Knowledge Element. CAFFEINE uses a similar approach in terms of organizing the network monitoring, route discovery, and QoS provision tasks into different modules.

Since the single controller solutions are limited in their scalability, many SDN approaches are ultimately realized in a hierarchical manner in order to manage larger networks. Bhattacharya et al., for example, proposed a hierarchical architecture with two levels of controllers, i.e., network providers and service providers (Bhattacharya and Das 2013). This paper inspires the future 9

improvement of CAFFEINE for scalability purpose. The future improvements of CAFFEINE should be designed in such a way that they can adapt to a hierarchical approach. This concept could be implemented by allowing CAFFEINEs on different ASes to communicate with each other and let multiple egress switches be destinations for routing that has a destination outside an AS.

10

Chapter 3: Methodology

3.1 Overview of CAFFEINE This CAFFEINE algorithm is designed to improve the performance of a network. The following assumptions, design guidelines, and goals are considered during an implementation of CAFFEINE.

 Assumptions: 1. Traffic flows are not equally spread over every pair of hosts in a network, but tend to have few destinations. This assumption is drawn from the behavior of users in an organization that uses typical services, i.e., email, web, messaging, video call and database. These services are located either in their organization servers or outside their company’s network. Thus the destination switches of these traffic flows are at either their data center or their egress switch, a switch that connects company’s network to the internet. These users have a small chance to create traffic directly between their computers. 2. Network administrators have knowledge about priorities of traffic flows in their networks to set preconfigured rules about traffic classes’ priority weights. This assumption is included because Queue Management Service needs the priority weights to assign the proper queue for each traffic flow. However, in the case that all traffic flows are considered equally important, they can all be set to the only flow class with any given positive weight to gain advantage from the flexible routing service.  Design guidelines: 1. CAFFEINE should minimize wasted effort on the data packet that will be dropped at downstream. Each data packet that is sent along the path always takes a portion of bandwidth from each link through which it passes. The data packets that will be dropped waste those portions and reduce throughput of other flows. By looking for the bottleneck link in the topology, CAFFEINE sets up a maximum bandwidth on the queues to prevent from wasted effort. This guideline is considered in order to support the goal of increasing throughput. 2. CAFFEINE should minimize the impact of new traffic flows on existing traffic flows. A starvation on existing flows can cause all downloaded data of those flows to be wasted and significantly decrease throughput of the network. This guideline is considered in order to support the goal of increasing throughput. 11

3. CAFFEINE should allow network flows to fully utilize spare network capacity when possible. Most of the conventional routing algorithms consider only shortest path. Some of them can encourage/discourage the use of high/low capacity links but none of them consider the current available bandwidth in the network. They overlook spare capacity that can be used for better improvement. CAFFEINE considers every possible path. 4. CAFFEINE should focus on long-lived traffic flows. Short-live traffic flows are not issues for users in an organization because it already takes a short time to finish, but most long-lived traffic flows will always be issues for productivity in an organization  Goals of CAFFEINE: 1. Increase throughput of the network in an organization: This is a common goal of most routing algorithms. The more delivered data result in fast work and better productivity. 2. Provide weighted fairness among different flow classes: This goal is considered because CAFFEINE takes advantages over every possible route in an organization. Some traffic flows might gain most of the bandwidth and interrupt other traffics. So this goal will help CAFFEINE to maintain good productivity in organization by offering proper guarantee bandwidths based on interclass fairness.

3.2 CAFFEINE Network Services To support the goals and follow guidelines listed in the previous section, CAFFEINE is comprised of four services: Network Monitoring Service, Rate Limiting Service, Flexible Routing Service, and Queue Management Service. As shown in Figure 1, most of these services are external applications that use Network Operating System services provided via a RESTful API by the Floodlight Controller. Although they are implemented as external applications because they can be done quickly for testing purpose, but they should all be implemented as services that are invoked by Floodlight Controller for better performance. In addition, CAFFEINE extends the Floodlight controller by implementing a custom module that provides flexible routing service, which exposes a RESTful API to implement the required routing algorithms. In this implementation, CAFFEINE uses forwarding rules separately for each traffic flow, so it can use each forwarding rule to represent each traffic flow.

12

Figure 1: Data flow diagram of CAFFEINE

The Network Monitoring Service periodically polls the Floodlight controller to capture the current network state in terms of a current bandwidth usage on each link. This service aggregates the usage statistics and forwards this data to the Flexible Routing Service and the Rate Limiting Service.

The Rate Limiting Service uses available features in Open vSwitch, a switch compliant with the OpenFlow standard, to support CAFFEINE’s goals. An Open vSwitch allows for the setting of a minimum and maximum bandwidth on a per queue basis. The minimum bandwidth is a minimum guaranteed bandwidth given to flows using a particular queue. The maximum bandwidth is a maximum allowed bandwidth that will not be exceeded by flows on a queue. The minimum bandwidth allows for the definition of a per class Quality of Service (QOS) to support CAFFEINE’s weighted fairness goal. In addition, the design of the algorithm in the Rate Limiting Service follows the minimizing impact of new flows guideline by limiting the changes needed to minimum bandwidth when new flows are introduced. The minimizing wasted effort guideline is followed via the use of maximum bandwidth limit. This limit is set based on available downstream bandwidth.

The Flexible Routing Service uses data received from the Network Monitoring service to select the widest path for new network flows. Instead of using a shortest path algorithm, this service supports the throughput goal by spreading traffic throughout the network. In addition, it follows the minimizing impact of new flows guideline by favoring paths with spare capacity over congested paths. Introducing a new flow into a less congested path is less likely to affect other flows using links on this path. In addition, this approach follows the minimizing wasted effort guideline by avoiding congested links. 13

The Queue Management Service manages the bandwidth limits for queues in the network. When a new flow is introduced to the network, the Queue Management Service queries the Flexible Routing Service for the current widest path. After determining the corresponding priority of a new connection flow, the service sets forwarding rules and queue bandwidth limits for switches on the path for the new flow. In addition, the service periodically adjusts the queue configurations based on the current network state. This service supports the weighted fairness goal.

3.2.1 Network Monitoring Service The Network Monitoring Service periodically queries the central controller, which polls the needed data from switches, for the state of the network. In particular, CAFFEINE requires a measure of the average spare capacity on each link for the Flexible Routing Service, which finds a widest path based on spare capacity. Since the current version of OpenFlow does not report average available bandwidth, this service needs to calculate the aggregate information based on available data by itself, allowing the remaining services of CAFFEINE to use the aggregate information.

OpenFlow does report an aggregate amount of data as a number of bytes that has matched each forwarding rule in a switch. Since these rules specify per port forwarding, CAFFIENE stores this information from the one previous call and estimate the current spare capacity on each link on a switch, based on the following equations:

bw  c  cb Equation 1 Equation 1 is used by Network Monitoring Service to estimated current available bandwidth in the network, so the other services of CAFFEINE can make their decisions based on this estimated data. bw: Current available bandwidth in a link c: Capacity of a link cb: Estimated current consumed bandwidth in a link

cb  avgBW( fr) fr Equation 2 Equation 2 is used to estimate the current consumed bandwidth in a link by summing the average consumed bandwidth on every forwarding rule. Noted that CAFFEINE uses each forwarding rule for a particular traffic flow, thus each forwarding rule represents its corresponding traffic flow. avgBW(fr): An average bandwidth consumed on forwarding rule fr fr: A forwarding rule that is installed into switch by central controller 14

df (n)  df (n 1) avgBW( fr)  af (n)  af (n 1) Equation 3 Equation 3 is used to calculate an average consumed bandwidth on a forwarding rule. Since each forwarding rule contains several counters that track status of the forwarding rule itself, CAFFEINE can asks the central controller to poll this information to estimate current state of network. df(n): The total transmitted data on a forwarding rule fr that is retrieved in the latest polling. This data is counted in a forwarding rule as a number of transmitted bytes. df(n-1): The total transmitted data on a forwarding rule fr that was retrieved in previous polling. af(n): The age of a forwarding rule fr that is retrieved in the latest polling. This data is counted in a forwarding rule as a total time since its installation. af(n-1): The age of a forwarding rule fr that was retrieved in the previous polling.

3.2.2 Rate Limiting Service The Rate Limiting Service runs alongside the Network Monitoring Service. Based on the route provided by the Flexible Routing Service, the traffic flows are assigned to queues in each of the switches. The Rate Limiting Service then adjusts the upper bound and lower bound for each queue based on the number of flows assigned to the queues reported by the Network Monitoring Service. The upper bound and lower bound on queues are set in order to support the following goals: fairness, minimizing impact of new flows on existing flows, and minimizing wasted effort due to the transmission of packets that will be dropped at a later switch.

This service determines the lower bound by calculating the total weight in each queue from the pre- configured priority weight of that queue. The total weight is simply the result of multiplying weight by the number of flows in a queue. Then each queue gets a portion of the capacity bandwidth on that output port based on the ratio of their weights. For example, suppose there are two flows on the same output port which has 1Mb/s capacity. One flow belongs to queue A, which has a weight of 1, and another flow belongs to queue B, which has a weight of 2. Then queue A should get 0.33 Mb/s lower bound and queue B should get 0.66 Mb/s lower bound. The action of the service is described as the following pseudocode.

Rate Limiting Algorithm 15

input: preconfigured priority weights, capacity of each link, polling information about forwarding rules in switches output: min-bandwidths and max-bandwidths to be set at each queue in each link

reset counter for every flow to 0 remove every flow in max_possible_bandwidth remove every stored path in flow_route

//begin aggregating network state on every queue, port, switch for each switch sw in topology for each output port p in switch sw call func countFlowsInPort(p) reset total_weight(p) for p to 0 reset queue_weight for every queue in port p to 0 call func calculateTotalWeightInPort(p) for each queue q in configured rules call func calculateMinBandwidth(q) call func determineMaxBandwidth(q, sw, p) endfor each queue endfor each port endfor each switch //finish aggregating network state //begin setting up lower bound and upper bound on each queue for each switch : sw for each port : p for each queue : q tmp_lower_bound = lower_bound_map of q at sw,p tmp_upper_bound = 0 for each flow_key fk in flow_route of q at p, sw //looking for the maximum upper bound of flows in this queue if tmp_upper_bound < max_possible_bandwidth of fk tmp_upper_bound = max_possible_bandwidth of fk endif endfor each flow if tmp_upper_bound < linkspeed at p, sw tmp_upper_bound = linkspeed at p, sw set tmp_lower_bound, tmp_upper_bound to q at p, sw endfor each queue endfor each port endfor each switch //finish setting up lower bound and upper bound on every switch

Algorithm 1: Rate Limiting Algorithm 16

Counting Function input: polling information about forwarding rules in switches output: number of traffic flow in each queue in each link This function counts the number of flows in each class in order to determine the bandwidth ratios between flow classes.

func countFlowsInPort(p) for each flow f in output port p determine a key flow_key for the flow with the combination of destination of f, source of f, etc. set flow_key in the flow_route for queue of f at p, sw to indicate that this combination belongs to this flow if max_possible_bandwidth for the flow with flow_key is not existed then initialize max_possible_bandwidth = 0 for a flow that use key as flow_key endif increase a counter for queue of flow f at sw, p by1 endfor each flow return

Algorithm 2: Counting Function

Total Weight Calculating Function input: number of traffic flow in each queue in each link output: total weight of each queue in each link.

This function uses the number of flows in each class to calculate the total weight in each link.

func calculateTotalWeightInPort(p) for each queue q in topology calculate queue_weight(q) by multiply its counter in port p with priority_weight of q add queue_weight to the total_weight of this output port endfor each queue return

Algorithm 3: Total Weight Calculating Function

17

Min-bandwidth Calculating Function input: total weight of each queue in a link output: min-bandwidth of each queue in a link This function calculates min-bandwidth of each queue by its weight ratio compared to the total weight. However, a queue that has none of the current flows is set to half of the capacity to prevent the problem of flow initiation during a gap of updating. An update gap occurs between each round of polling by Network Monitoring Service.

func calculateMinBandwidth(q) if total_weight > 0(check this to prevent the problem of divide by zero) if queue_weight(q) > 0 set lower_bound_map for q at p, sw to (queue_weight(q) * linkspeed at p, sw) / total_weight endif else //set it to half of the link speed to prevent the unreachable //destination problem during a gap of re-adjusting minimum bandwidth set lower_bound_map for q at p, sw = (linkspeed at p, sw)/2 endelse endif else //set it to half of the link speed to prevent the unreachable //destination //problem during a gap of re-adjusting minimum bandwidth set lower_bound_map for q at p, sw = (linkspeed at p, sw)/2 endelse return

Algorithm 4: Min-bandwidth Calculating Function

Max-Bandwidth Determining Function input: min-bandwidth of each queue in a link output: potential candidate of max-bandwidth in a flow This function determines max-bandwidth by choosing the highest min-bandwidth along the total path of a flow. 18

func determineMaxBandwidth(q, sw, p) //check if there is new higher possible bandwidth for q at sw, p for each flow_key fk that has flow_route at sw, p, q if max_possible_bandwidth of fk < lowerbound for q at p, sw set max_possible_bandwidth of fk = lowerbound for q at p, sw endif endfor each flow return

Algorithm 5: Max-bandwidth Determining Function

In the case that multiple flows have to share a common link, the lower bound and upper bound of the queues allow Open vSwitch to use its available mechanisms to provide fairness. The current version of Open vSwitch supports linux-htb (Hierarchical Token Bucket) and linux-hfsc (Hierarchical Fair Service Curve).

Linux-htb is based on the Token Bucket algorithm, which controls the rate and burstiness of a traffic flow by simulating a bucket in which tokens are added at a fixed rate (Valenzuela, et al. 2004). A flow can transmit a data packet when there are tokens equivalent to the amount of data to be sent.

Linux-hfsc is based on a nonlinear service-curve, which is a nondecreasing function (Stoica, Zhang, and Eugene Ng 2000). It can allocate a fairly shared bandwidth and can also limit the initial delay of a data stream, which is important to VoIP or games that are sensitive to delay.

Figure 2: Example scenario of weighted fairness 19

By relying on these Open vSwitch features, the rate limiting service can guarantee that each queue has a proportional ratio of bandwidth with respect to its priority and the amount of flow it contains. Since this Rate Limiting Service periodically recalculates available bandwidth based on newly updated network state, the fairness mechanism can maintain weighted fairness better than a convention static allocation. In an example scenario in Figure 2, there are two flows, class A and class B, whose weights are 3 and 1 respectively in link L. Therefore, class A will get 75% of capacity while class B gets 25% of capacity. Even if there are two new flows in class B, this ratio would remain in a conventional static setting. The Rate Limiting Service, though, changes this ratio to 50%-50% because class B would have a total weight of 3 (1 + 1 + 1) equal to a total weight of class A. Thus it supports the weighted fairness goal. Since this service does not split a single flow to use multiple paths, which might cause trouble in the order of delivered packet, this service cannot always guarantee the exact bandwidth ratio when it tries to use spare capacity on the other paths by sending different flows into different paths.

The fairness mechanism in Open vSwitch also guarantees that each queue gets the proper reserved bandwidth based on its priority. The new flows that are introduced into the network do not affect the existing flows significantly. No matter how bursty of each the new flows is, the fairness mechanism guarantees shared bandwidth to the existing flows and prevents them from starvation. Thus this service follows the guideline of minimizing the impact of new flows.

Figure 3: Example scenario for minimizing wasted effort

With the upper bound of the queue, this Rate Limiting Service is expected to guarantee that each flow does not transport data at any point in the topology at a rate that is greater than the maximum bandwidth ratio it gets on any particular link in its path. In the example scenario in Figure 3, 20

suppose hosts h1 and h3 make a connection to server sv1 while host h2 makes a connection to server sv2. The three connections, h1-sv1, h2-sv2, h3-sv1, are on different queues with the priority weights of 1, 3, and 1 respectively. The calculated minimum guaranteed bandwidths from the Rate Limiting Service are as described in Table 1.

Table 1: Example work of rate limiting service

capacity flow priority priority min-bandwidth max-bandwidth link (Mb/s) (source-destination) class weight (Mb/s) (Mb/s) h1-sv1 A 1 1 2 L3 4 h2-sv2 B 3 3 4 h1-sv1 A 1 2 2 L5 4 h3-sv1 C 1 2 2 L6 4 h2-sv2 B 3 4 4

As shows in Table 1, there are two flows in different classes in link L3 with the weights of 1 and 3. Under the work of Rate Limiting Service, these flows in link L3 get min-bandwidth as 1 Mb/s and 3 Mb/s respectively. The min-bandwidths are calculated with the following equation:

Equation 4 -

The min-bandwidths on other links are calculated using the same process. On the other hand, max- bandwidths of each queue are determined by the highest possible min-bandwidth along the total path of each flow in its queue. For example, flow h1-sv1 received min-bandwidths as 1 Mb/s on link L3 and 2 Mb/s on link L5. So its max-bandwidth is set to 2 Mb/s on every link it passes.

In the absence of other controls, while all flows are active, link L5 is saturated. If flow h2-sv2 completes, flow h1-sv1 would try to increase its transmission rate, since there is more bandwidth available on linke L3. Ultimately, this would cause packets to be dropped on the queue for link L5. CAFFEINE tries to avoid this situation by setting the max-bandwidth of flow h1-sv1 to only 2 Mb/s, determined as mentioned in the previous paragraph, to avoid wasted effort that will be dropped eventually. Thus the Rate Limiting Service supports the goal of minimizing wasted effort.

21

3.2.3 Flexible Routing Service The Flexible Routing Service consists of two parts. The first part periodically calculates a next-hop map based on a widest path algorithm and updates it. The second part is invoked by the Queue Management Service when a new flow reaches a boundary router.

Widest Path Algorithm input: current available bandwidth of each link that is calculated from Equation 1, Equation 2 and Equation 3 output: forward map that gives paths for each source-destination pair This first part uses a greedy approach to solve the widest path problem.

Path Determination Algorithm input: forwarding map from Widest Path Algorithm, source switch, destination switch output: a total path from source switch to destination switch This second part of the routing service uses the next hop map produced by the first part to determine the route to be followed.

These two algorithms are shown in the following pseudocodes on the pages 22 and 23.

This Flexible Routing Service follows both the maximizing throughput guideline and minimizing impact of new flows guideline. By choosing the widest path, the new path attempts to maximize the throughput for this new flow and does not load the shortest path exclusively. Moreover, by using the widest paths, CAFFEINE tends to avoid congested paths. If the new flows are assigned to pass through the saturated links, they would cause more severe congestion and have a larger impact on the existing flows.

22

reset next hop map for each switch root in topology initialize flag seen to false for every switch initialize bandwidth bw to -1 for every switch initialize priority queue nodeQ to empty queue put root into nodeQ with cost = infinity while nodeQ is not empty get switch node and cost c by dequeue highest cost node from nodeQ //tie is broken by choosing the prior one if node exists in seen continue endif add node into seen for each link l in node let source switch src = source of l //assuming node is a destination if src exists in seen continue endif let avai_bw = out going available bandwidth of l //which is calculated from Network Monitoring Service if c < avai_bw // check if prior link has less bandwidth than current link l avai_bw = c endif put the following forwarding information src->node into next hop map by using root as destination and src as source enqueue src into nodeQ with cost avai_bw endfor each link endwhile endfor each switch return next hop map

Algorithm 6: Widest Path Algorithm

23

Initialize dst as destination switch of flow Initialize src as source switch of flow Initialize route as empty set select next hop map which has dst as destination from the first part while src != dst get src->node from next hop map by search from source as src put forwarding information src->node into route src = node endwhile return route that contains all forward information in a total route

Algorithm 7: Path Determination Algorithm

3.2.4 Queue Management Service This service works like a reception desk of the Floodlight Network Operating System. The first packet of every new flow is sent from an edge switch to the controller and processed by this service first. This service determines the priority of the new flow by looking into preconfigured rules, finds a corresponding queue, and then asks the Routing Service for a new route. Once it gets the result for the new path, it converts the route and the corresponding queue into flow-mod messages. These messages are sent to the Open vSwitch switches on the route to assign the flow to a queue in the switches’ forwarding table. After the Queue Management Service assigns a flow to a queue, the Rate Limiting Service then determines the new bandwidth allocation for the queues, as described in the earlier section. 24

Chapter 4: Evaluation This chapter gives an explanation of how the evaluation of CAFFEINE is conducted and analyzes the results. The simulations of different network configurations are created in Mininet, an application that allows a user to create virtual network by simulating virtual hosts and switches. In addition, Floodlight, which is an implementation of SDN central controller based on OpenFlow standard, is used as a central controller. The controller is extended with new services and CAFFEINE applications are created.

4.1 Simulation System

4.1.1 Assumptions The following assumptions are made in the Mininet’s network simulation to evaluate CAFFEINE:

1. Most of the traffic flows are elastic, in the sense that these traffic flows try to get as much bandwidth as possible under the constraints of transport layer congestion control mechanisms, until the message has been transmitted. 2. Most of the traffic flows in the network pass through a small number of initial switches. This assumption is supported by scenarios in which users tend to use services on an organization’s servers or services on the internet outside company. Therefore most of the initial switches for flows are likely to be gateway switches that bridge the organization network to the Internet outside or the switch at a server room in the organization. 3. The priority of traffic flow classes, which emphasizes the bandwidth needed for flow classes, can be determined and set up a priori, allowing for the development of preconfigured weights to be used for the queue management service. In the case where all traffic has the same priority, CAFFEINE still works towards the goal of improving throughput, while striving to achieve an equal distribution of allocated bandwidth across all flows. 4. The topology of the network allows for multiple possible paths rather than a conventional tree structure that has only one possible route. 5. Open vSwitch, which is capable of using OpenFlow version 1.0 standard and supporting either the linux-htb or linux-hsfc fairness mechanisms, are used as switches in the network.

4.1.2 Simulation Details CAFFEINE is evaluated under three distinct network topologies for three different tests. Each topology is described in their sections under 4.1.3, 4.1.4 and 4.1.5. Each topology consists of 9 client 25

machines, namely from h1 to h9, and 3 servers, namely sv1, sv2, and sv3. Each client machine continuously makes a request to a file of size 2 MB, 4 MB, and 6 MB with the wget command and repeats these requests for fifteen minutes. The performance is measured by an average of 25 repetitions. h1, h4, and h7 made requests to sv1. h2, h5, and h8 made requests to sv2. h3, h6, and h9 made requests to sv3. Their priority weights are assigned differently on each topology and are described in their sections under 4.1.3, 4.1.4 and 4.1.5. The larger these weights, the more bandwidth their flows need. An implementation is created with Python scripts that communicate with a controller through RESTful APIs to measure the ability of CAFFEINE rather than the actual product that should be implemented as services on the central controller. A major limiatation of this implementation is its inability to work with complex rules and other criteria for routing. This limitation is balanced by the simplicity in the protocol that arises from the use of destination as the routing criteria. Each topology is simulated under 3 different routing configurations:

1. Without CAFFEINE: This configuration uses the default routing from Floodlight controller. The routing algorithm is the shortest path that works in the same manner as OSPF without the multiple paths feature. Unlike OSPF, though, routing calculations and control logic are performed in software at the network controller, which creates forwarding rules that are sent to switches. The selected shortest route in this algorithm is always the same unless there are changes in topology. The default routing does not provide any Quality of Service guarantees, and thus does not support varying traffic priorities. 2. Naive CAFFEINE: This configuration uses CAFFEINE to make a decision of routing and picking a route once per file request. This approach has significant routing and queue setup message overhead. This configuration represents the use of CAFFEINE with short-lived flows in network. 3. Revised CAFFEINE: This configuration works similarly to the previous setting, but reduces the amount of flow-mod messages. It adds/removes forwarding rules only at the start and the end of each round. Therefore, the three requests of 2MB, 4MB, and 6MB files are forced to use the same route as if they are one single long-lived flow. However, the Queue Management Service and Network Monitoring Service still work periodically with the same frequency. This configuration represents the use of CAFFEINE with long-lived flows in network. 26

4.1.3 Fairness Testing

Figure 4: The topology of fairness testing

As shown in Figure 4, the first topology tested is designed to force traffic flows to use a shared link in order to display the capability of fairness. The priority weights are assigned based on the destination of the traffic flows, sv1 as 1.0, sv2 as 3.0, and sv3 as 4.0. The Table 2 displays the result from fairness testing.

Table 2: Downloaded data in fairness testing

downloaded data (MB) priority source destination without naive revised weight CAFFEINE CAFFEINE CAFFEINE h1 sv1 1.0 96.48 49.52 43.68 h2 sv2 3.0 96.48 120.48 115.92 h3 sv3 4.0 96.48 151.68 144.96 h4 sv1 1.0 96.00 47.76 41.68 h5 sv2 3.0 96.00 107.44 109.68 h6 sv3 1.0 97.12 133.28 143.76 h7 sv1 1.0 98.88 44.72 39.28 h8 sv2 3.0 97.12 104.32 106.80 h9 sv3 4.0 97.92 130.08 139.12 Total 872.48 889.28 884.88

Table 2 shows CAFFEINE’s ability to shape traffic flows. The total downloaded data implies different amounts of data downloaded for each routing algorithm. Since, without CAFFEINE, configuration cannot be set up with weighted fairness, all hosts receive the same amount of bandwidth because of the mechanism in transport layer. Meanwhile, CAFFEINE configurations, that 27

are capable of handling weighted fairness between traffic classes, can shape their traffics to follow the priority weights that are configured.

Megabytes (MB) Downloaded data in fairness testing 160 140 120 Without CAFFEINE 100 Naive CAFFEINE 80 Revised CAFFEINE 60 40 20 0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 5: Comparison graph of downloaded data in fairness testing

Figure 5 shows the graphical comparison among routing configurations. It is easily seen that CAFFEINE can shape traffic flows to follow their priority weights.

Total downloaded data in fairness testing

Megabytes (MB)

1000 Without CAFFEINE 800 Naive CAFFEINE 600 Revised CAFFEINE

400

200

0

Figure 6: Comparison graph of total downloaded data in fairness testing

Figure 6 shows the graphical comparison among routing configurations. In this scenario that has no multiple paths that might leave spare capacity in conventional routings, CAFFEINE cannot improve throughput to overcome conventional routings. 28

Table 3: Average bandwidth in fairness testing

without CAFFEINE naive CAFFEINE revised CAFFEINE priority average average average average average average source destination weight bandwidth bandwidth bandwidth bandwidth bandwidth bandwidth (KB/s) ratio (KB/s) ratio (KB/s) ratio h1 sv1 1.0 112.54 1.00 58.17 1.00 52.81 1.00 h2 sv2 3.0 111.41 0.99 147.79 2.54 137.57 2.60 h3 sv3 4.0 111.09 0.99 189.37 3.26 176.40 3.34 h4 sv1 1.0 111.24 1.00 58.39 1.00 50.17 1.00 h5 sv2 3.0 110.85 1.00 150.98 2.59 134.99 2.69 h6 sv3 4.0 111.39 1.00 195.98 3.36 176.59 3.52 h7 sv1 1.0 112.96 1.00 55.06 1.00 48.52 1.00 h8 sv2 3.0 112.32 0.99 147.23 2.67 131.62 2.67 h9 sv3 4.0 111.58 0.99 191.31 3.47 173.56 3.47

Kilobytes per second Average bandwidth of flows in fairness testing (KB/s) 200

150 Without CAFFEINE Naive CAFFEINE 100 Revised CAFFEINE

50

0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 7: Comparison of average bandwidth in fairness testing 29

Table 3 shows the average bandwidths that are logged during invocation of the wget command through the period of 15 minutes. These results imply that CAFFEINE can shape traffic flows and guarantee bandwidths based on their priority classes.

Figure 7 shows the graphical interpretation of Table 3. It is easier to see that traffic flows under conventional routing, without CAFFEINE, are converged to get equal bandwidths, but CAFFEINE configurations can provide weighted fairness between traffic classes.

The results on the first configuration show that the bandwidth is equally distributed among flows under the default scenario without CAFFEINE, and unequally distributed based on their priority classes under the CAFFEINE scenarios. There is no significant improvement in total downloaded data under CAFFEINE because this topology is set up to have a single bottleneck link on which all flows must travel. However, the ratio of the downloaded data between flows reflects different priorities, and shows a significant difference between queues with different priority weights. Note that there is approximately a 10 to 20% difference between the ratios of downloaded data and the ratios of the priorities. There are reasons behind this difference. First, some flows are active when others have not yet started or have already completed. Second, a default min-bandwidth for a queue that doesn’t contain any flows yet is set to a half of a link capacity in CAFFEINE; this default min-bandwidth allows new flows to initiate connections during a gap of update. These two reasons skew the ratios of downloaded data to be different from their priority weights. The results from these scenarios support the contention that CAFFEINE provides support for weighted fairness goals. The next scenarios show that under multiple paths routing, the bandwidth ratios are skewed even more due to the mechanisms in CAFFEINE that help flows to use the additional capacity. 30

4.1.4 Utilization Testing

Figure 8: The topology of utilization testing

As shown in Figure 8, the next scenario is designed to test CAFFEINE’s ability to use multiple path routing, under the widest path route discovery. Therefore, this topology is provided with high capacity links that are not in the shortest path of h4 to h9. The priority weights are set to 1, 1.5, and 2 for destinations sv1, sv2, and sv3 respectively.

Figure 9: Comparison of possible paths for h1, h2 and h3

31

Figure 10: Comparison of possible paths for h4, h5 and h6

Figure 11: Comparison of possible paths for h7, h8 and h9

Figures 9, 10, and 11 show the comparisons between conventional routings and CAFFEINE. While conventional routings consider only the shortest path that has minimum number of hops, CAFFEINE considers every possible path and takes the current available bandwidth into its consideration to pick the widest path.

32

Table 4: Downloaded data in utilization testing

downloaded data (MB) priority source destination without naive revised weight CAFFEINE CAFFEINE CAFFEINE h1 sv1 1.0 160.80 187.68 184.64 h2 sv2 1.5 158.88 200.16 189.20 h3 sv3 2.0 160.32 209.76 215.84 h4 sv1 1.0 132.48 144.00 154.80 h5 sv2 1.5 132.00 156.48 171.28 h6 sv3 2.0 132.00 160.8 188.32 h7 sv1 1.0 131.04 142.08 163.04 h8 sv2 1.5 131.04 152.64 175.44 h9 sv3 2.0 131.04 158.88 185.44 total 1269.60 1512.48 1628.00

The results in Table 4 on the default routing, without CAFFEINE, show a limitation of the conventional routing. It limits routing paths to only a single shortest path. While there is spare capacity in the dashed line link of Figure 8, the conventional routing algorithm ignores them. Only hosts h1, h2, and h3 that have their shortest path through the dash link can get benefit from the high capacity link, while other hosts get no benefit from these links. The comparison of results in Figure 12 shows that CAFFEINE allows h4 to h9 to take advantage of spare capacity and achieves throughput improvement over conventional routing. On the other hand, the ratio of downloaded data is skewed from their priority weights more than result in fairness testing because the traffic flows go through different path and not always shared the same link.

Megabytes (MB) Downloaded data in utilization testing

250

200 Without CAFFEINE 150 Naive CAFFEINE Revised CAFFEINE 100

50

0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 12: Comparison of downloaded data in utilization testing 33

Total downloaded data in utilization testing

Megabytes (MB)

1800 1600 Without CAFFEINE 1400 Naive CAFFEINE 1200 Revised CAFFEINE 1000 800 600 400 200 0

Figure 13: Comparison of total downloaded data in utilization testing

Figure 13 shows that CAFFEINE can use spare capacity in the network to improve throughput. Compared to Figure 6 in which CAFFEINE gets no throughput improvement, this also confirms that CAFFEINE does need a topology that has multiple paths in the network to improve the overall throughput of the network.

In Table 5, the average bandwidths, which are logged by the wget command during the 15 minutes tests, also show that CAFFEINE allows the network to gains benefit from spare capacity. This benefit is available to every host in this scenario as its bandwidth and throughput increase over the conventional routing.

Figure 14 shows the graphical comparison between different configurations. It can be seen easily that CAFFEINE allows h4 to h9 to gain benefit from high capacity links as well as increase overall throughput.

However it needs to be noted that these average bandwidths might not be straight forward in terms of showing throughput because other flows can gain more bandwidth during the overhead of flow setup that cause new flows to be idle. The wget command also doesn’t log when it is idly waiting for the traffic flows to start their connections. Additionally, Figure 14 also shows that only h1, h2, and h3 gain benefit from the dash line link, which has greater capacity and lies in their shortest path, in conventional routing, while CAFFEINE allows other hosts to gain benefit from all high capacity links in the network. 34

Table 5: Average bandwidth in utilization testing

without CAFFEINE naive CAFFEINE revised CAFFEINE priority average average average average average average source destination weight bandwidth bandwidth bandwidth bandwidth bandwidth bandwidth (KB/s) ratio (KB/s) ratio (KB/s) ratio

h1 sv1 1.0 184.50 1.00 242.03 1.00 214.60 1.00 h2 sv2 1.5 183.88 1.00 258.51 1.07 224.57 1.02 h3 sv3 2.0 184.30 1.00 274.09 1.12 256.87 1.17 h4 sv1 1.0 150.25 1.00 223.38 1.00 199.18 1.00 h5 sv2 1.5 149.84 1.00 249.54 1.09 219.48 1.11 h6 sv3 2.0 149.82 1.00 258.17 1.12 243.87 1.22 h7 sv1 1.0 147.89 1.00 228.86 1.00 213.12 1.00 h8 sv2 1.5 148.59 1.00 251.03 1.07 233.18 1.08 h9 sv3 2.0 148.21 1.00 264.95 1.12 250.68 1.14

Kilobytes per second Average bandwidth of flows in utilization testing (KB/s) 300

250

200 Without CAFFEINE Naive CAFFEINE 150 Revised CAFFEINE 100

50

0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 14: Comparison of average bandwidth in utilization testing 35

In this utilization testing, naive CAFFEINE can raise throughput by up to 19.13% while revised CAFFEINE can raise throughput by up to 28.23% according to the results in Table 4 and Figure 13. Applying a one-tailed t-test to these results yields p-values of 1.36 * 10-9 and 1.64 * 10-13, indicating that the improvement in throughput caused by CAFFEINE is statistically significant under both short lived and long lived flows. In addition, the ratio of downloaded data is skewed more than the result in fairness testing because CAFFEINE spread flows over multiple paths, but comes with better throughput improvement. Note that while multipath routing is expected to significantly improve overall throughput, the time required for path setup would have an adverse effect on throughput. The impact of this additional overhead explains the difference in the average throughput for the two sets of CAFFEINE configurations.

4.1.5 Variable Link Capacity Testing

Figure 15: The topology of variable link capacity testing

As shown in Figure 15, the last scenario is designed to reflect the general topology in an organization with high capacity links at the center of the network. The priority weights are all set to the same value, indicating that all flows are considered equally important as they are treated in most organizations.

36

Table 6: Downloaded data in variable link capacity testing

downloaded data (MB) priority source destination without naive revised weight CAFFEINE CAFFEINE CAFFEINE h1 sv1 1 203.04 227.52 249.12 h2 sv2 1 203.52 221.28 253.44 h3 sv3 1 204.00 224.64 265.44 h4 sv1 1 203.52 178.56 238.56 h5 sv2 1 204.00 179.52 239.52 h6 sv3 1 204.00 178.08 239.04 h7 sv1 1 204.00 168.48 228.00 h8 sv2 1 203.52 170.4 227.04 h9 sv3 1 203.52 169.92 234.24 Total 1833.12 1718.4 2174.4

Table 6 shows a result of the simulation that naive CAFFEINE, which is expected to gain improvement, can lead to poorer performance as the total downloaded data is less than the conventional routing. However revised CAFFEINE, which is designed to reduce the flow path setup overhead, still gains better improvement over the conventional routing. Figures 16 and 17 show the graphical comparisons between three configurations in Table 6.

Downloaded data in variable link capacity testing Megabytes (MB) 300

250

200 Without CAFFEINE Naive CAFFEINE 150 Revised CAFFEINE 100

50

0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 16: Comparison of downloaded data in variable link capacity testing

As seen in Figure 16, even though the priority weights are all set equally to 1, the multiple flows from one source switch can take over some portion of bandwidth from other source switches. The downloaded data from different source switches can be different while flows from the same source switch tend to have an equal amount of shared bandwidth. 37

Total downloaded data in variable link capacity testing

Megabytes (MB)

2500 Without CAFFEINE 2000 Naive CAFFEINE 1500 Revised CAFFEINE

1000

500

0

Figure 17: Comparison of total downloaded data in variable link capacity testing Figure 17 shows the graphical comparison from data in Table 6. Notice that naive CAFFEINE leads to poorer performance than conventional routing in this variable link capacity testing. Meanwhile revised CAFFEINE still achieves throughput improvement over conventional routing.

Table 7 and Figure 18 show similar results to Table 6 and Figure 16: that under CAFFEINE, the flows from different source switches can take portion of bandwidth on any links in topology. This leads to the different of bandwidths from hosts under different source switches, but flows from the same source switch still tend to get equal bandwidths. Note that the average bandwidth of naive CAFFEINE seems to be higher than without CAFFEINE, but it is because the naive CAFFEINE has too much overhead that cause flows to be idle during their path setup. When traffic flows are idle, the other flows can gain more bandwidth and result in higher average bandwidth in the log of the wget command.

The results of the first configuration without CAFFEINE show a regular overview of a general network in large organizations, but the interesting thing is that it outperforms CAFFEINE with short-lived flows. This confirms that the flow-mod messages do incur significant overhead and cost under some scenarios. While naive CAFFEINE, that asks for a new route on every new flow, is outperformed by the conventional routing that simply uses the same forwarding rules for the total testing period, revised CAFFEINE can improve and get better throughput than conventional routing by up to 18.62% improvement. Revised CAFFEINE confirms that CAFFEINE works well on long- lived flows. Applying a one-tailed t-test statistic to the default routing and CAFFEINE with long- lived flows yields a p-value of 2.58 * 10-20 indicating a statistically significant improvement under CAFFEINE. 38

Table 7: Average bandwidth in variable link capacity testing

without CAFFEINE naive CAFFEINE revised CAFFEINE priority average average average average average average source destination weight bandwidth bandwidth bandwidth bandwidth bandwidth bandwidth (KB/s) ratio (KB/s) ratio (KB/s) ratio h1 sv1 1 231.37 1.00 307.29 1.00 298.51 1.00 h2 sv2 1 231.42 1.00 303.05 0.97 305.03 1.02 h3 sv3 1 231.36 1.00 304.13 0.99 320.21 1.07 h4 sv1 1 232.35 1.00 299.71 1.00 312.9 1.00 h5 sv2 1 231.87 1.00 303.99 1.01 314.55 1.00 h6 sv3 1 231.86 1.00 302.86 1.00 315.88 1.00 h7 sv1 1 232.15 1.00 281.53 1.00 308.28 1.00 h8 sv2 1 232.55 1.00 285.49 1.01 307.05 1.00 h9 sv3 1 231.82 1.00 288.65 1.01 315.78 1.03

Kilobytes per second Average bandwidth of flows in variable link capacity testing (KB/s) 350

300

250 Without CAFFEINE 200 Naive CAFFEINE

150 Revised CAFFEINE

100

50

0 flow h1-sv1 h2-sv2 h3-sv3 h4-sv1 h5-sv2 h6-sv3 h7-sv1 h8-sv2 h9-sv3

Figure 18: Comparison of average bandwidth in variable link capacity testing 39

Chapter 5: Conclusions and Future Work

5.1 Evaluation of CAFFEINE The goal for CAFFEINE is to improve the performance of software defined networks in terms of throughput and weighted fairness. The results in 4.1.5 show that using CAFFEINE with short-lived flows might lead to poorer performance due to the control message overhead. On the other hand, using CAFFEINE with long-live flows provided improvements in throughput in all scenarios tested.

Moreover, the disadvantage of cost in flows setting up can be mitigated by features of the OpenFlow standard that have not been evaluated in current testing. The OpenFlow standard includes the notion of wildcard matching and idle timeouts on the forwarding rules. By aggregating flows through wildcard routing, a collection of short-lived flows could be aggregated into a long-lived one. As long as there are sufficient short-lived flows within the time-out period, new control messages to set up the routes would not be needed. Moreover, these timeouts would reduce the number of control messages needed for CAFFEINE to clean up installed route on switches.

The evaluation tests also support the argument that CAFFEINE can provide for better network performance in topologies with multiple routes, especially in a general case where there are high capacity links as the center of network. The general testing scenario contained similar characteristics to internal corporate networks or data center networks. CAFFEINE also provided for effective support for fairness goals, particularly in the case of a bottleneck link.

5.2 Future Work Future works for the CAFFEINE algorithm that we intend to include are:

Hierarchical Control: CAFFEINE works with a single controller for small, flat topologies. Extending the CAFFEINE approach to a hierarchical one would enable the application of this approach in larger networks. For scalability purpose, CAFFEINEs in different hierarchies should be able to communicate with each other and provide information regarding the possible ingress/egress switches of their own AS. allowing them to determine the widest path for routing.

Flows aggregation: as mentioned above, combining multiple short-lived flows to get a long-lived aggregated flow could extend scenarios in which CAFFEINE would provide good results. The use of idle timeout and wildcard rules in OpenFlow could be integrated into CAFFEINE for this purpose. 40

Routings that take advantage of memory types: Not all in-switch memory is created equal. Future work will investigate the optimization of the use of high-speed TCAM memory. Moreover, not all routes must be dynamic. Instead, CAFFEINE will investigate the use of a combination of default static routes and dynamic routes. A dynamic route can be constructed from dynamic routing on some switches and static routing on the other switches. This approach would reduce overhead caused by the flow-mod messages.

Suitable interval for Network Monitoring Service and Queue Manaage Service: These two services work by communicating with switches in network. However there is a tradeoff for gaining information that cost a portion of bandwidth to be used. A refined investigation on proper interval for these services should be analyzed in future works of CAFFEINE. 41

References Bhattacharya, Bivas, and Debabrata Das. "SDN based Architecture for QoS Enabled Services Across Networks with Dynamic Service Level Agreement." In Advanced Networks and Telecommuncations Systems (ANTS), 1-6. Kattankulathur: IEEE, 2013.

Casado, Martin, Michael J. Freedman, Justin Pettit, Jianying Luo, Nick McKeown, and Scott Shenker. "Rethinking Enterprise Network Control." In Networking, IEEE/ACM Transactions, 1270-1283. IEEE, 2009.

Danna, Emilie, Subhasree Mandal, and Arjun Singh. "A practical algorithm for balancing the max min fairness and throughput objectives in traffic engineering." In INFOCOM, 846-854. Orlando, FL: IEEE, 2012.

Dijkstra, Edsger Wybe. "A note on two problems in connexion with graphs." In Numerische Mathematik 1, 269-271. Springer-Verlag, 1959.

Facebook. Facebook. June 18, 2014. https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the next-steps-toward-a-disaggregated-network/ (accessed June 1, 2015).

Gelberger, Alexander, Niv Yemini, and Ran Giladi. "Performance Analysis of Software-Defined Networking (SDN)." In Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), 389-393. San Francisco, CA: IEEE, 2013.

Greenberg, Albert, et al. "A clean slate 4D approach to network control and management." In ACM SIGCOMM Computer Communication Review, 41-54. New York, NY: ACM, 2005.

Hedrick, Charles. "Routing Information Protocol." In RFC1058. IETF, 1988.

Jain, Sushant, et al. "B4: Experience with a Globally-Deployed Software Defined WAN." In SIGCOMM '13, 3-14. New York, NY, USA: ACM, 2013.

Jeong, Kwangtae, Jinwook Kim, and Young-Tak Kim. "QoS-aware Network Operating System for Software Defined Networking with Generalized OpenFlows." In Network Operations and Management Symposium (NOMS), 1167-1174. Maui, HI: IEEE, 2012.

Moy, John. "OSPF Version 2." In RFC2328. IETF, 1998.

Nace, Dritan, and Michal Pioro. "Max-min fairness and its applications to routing and load-balancing in communication networks: a tutorial." In Communications Surveys & Tutorials, 5-17. IEEE, 2009.

Postel, Jon, and Reynolds Joyce. "FILE TRANSFER PROTOCOL (FTP)." In RFC959. IETF, 1985.

Rekhter, Yakov, Tony Li, and Susan Hares. "A Border Gateway Protocol 4 (BGP-4)." In RFC4271. IETF, 2006. 42

Rosen, Eric C., Arun Viswanathan, and Ross Callon. "Multiprotocol Label Switching Architecture." In RFC3031. IETF, 2001.

Schulzrinne, Henning, Stephen L. Casner, Ron Frederick, and Van Jacobson. "RTP: A Transport Protocol for Real-Time Applications." In RFC3550. IETF, 2003.

Stoica, Ion, Hui Zhang, and Tze Sing Eugene Ng. "A hierarchical fair service curve algorithm for link sharing, real-time, and priority services." In ACM SIGCOMM '97 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 249-262. New York, NY: ACM, 1997.

Valenzuela, Jose Luis, A. Monleon, I. San Esteban, Marc Portoles, and Oriol Sallent. "A Hierarchical Token Bucket Algorithm to Enhance QoS in IEEE 802.11: Proposal, Implementation and Evaluation." In Vehicular Technology Conference, 2659 - 2662. Los Angeles, CA: IEEE, 2004.

Wang, Richard, Dana Butnariu, and Jennifer Rexford. "OpenFlow-based server load balancing gone wild." In the 11th USENIX conference on Hot topics in management of internet, cloud, and enterprise networks and services, 12-12. Berkeley, CA: USENIX Association, 2011.

Wang, Yufei, and Zheng Wang. "Explicit routing algorithms for Internet traffic engineering." In Eight International Conference on Computer Communicationis and Networks, 582-588. Boston, MA: IEEE, 1999.

Zhang, Junjie, Kang Xi, Min Luo, and H. Jonathan Chao. "Load Balancing for Multiple Traffic Matrices Using SDN Hybrid Routing." In High Performance Switching and Routing (HPSR), 44-49. Vancouver, BC: IEEE, 2014.