Simulating the Impact of Qoe on Per-‐Service Group Hsd Bandwidth

SIMULATING THE IMPACT OF QOE ON PER-SERVICE GROUP HSD BANDWIDTH

CAPACITY REQUIREMENTS

TOM CLOONAN, CTO, NETWORK SOLUTIONS JIM ALLEN, LEAD SOFTWARE ENGINEER, BROADBAND GROUP MIKE EMMENDORFER, SENIOR DIRECTOR OF SOLUTION ARCHITECTURE AND STRATEGY, OFFICE OF THE CTO

TABLE OF CONTENTS

INTRODUCTION TO TRAFFIC ENGINEERING ...... 3 QUALITY OF EXPERIENCE BASICS ...... 7 THE NEW TRAFFIC ENGINEERING FORMULA ...... 9 VALIDATING THE NEW TRAFFIC ENGINEERING FORMULA ...... 14 TWO OBSERVED WEAKNESSES OF THE NEW TRAFFIC ENGINEERING FORMULA ...... 39 SELECTING APPROPRIATE TMAX & BHPG VALUES GIVEN AN ADVERTISED BILLBOARD BANDWIDTH ...... 41 TYPICAL DESIGN SCENARIO ...... 44 CONCLUSIONS ...... 46 RELATED READINGS ...... 46 REFERENCES ...... 46 ABBREVIATIONS & ACRONYMS ...... 48

INTRODUCTION TO TRAFFIC ENGINEERING

“How much bandwidth capacity is needed to keep the subscribers in our service groups happy?”

That question has been asked by Multiple System Operators (MSOs) throughout the years of High-Speed Data (HSD) deployments, and traffic engineers have developed many tools to help answer the question. Different MSOs have employed different traffic engineering algorithms and used different traffic engineering models throughout the years. This paper will propose and study a new traffic engineering algorithm that may be useful as MSOs move forward into the future.

A service group is an important concept within traffic engineering. In this paper, it will be loosely defined as a group of subscribers who share common bandwidth resources. In the Cable Industry, a service group is oftentimes comprised of the subscribers connected to a single fiber node or the subscribers connected to a group of fiber nodes. As traffic engineers of the past created their important estimates of required bandwidth capacity for a High-Speed Data (HSD) service group, they usually paid attention to four important parameters:

• the Maximum Sustained Traffic Rate (Tmax) levels offered to the subscribers • the Advertised Billboard Bandwidth levels • the Busy-Hour Performance Goal • the Average Per-Subscriber Traffic (Tavg) level consumed by subscribers

The “Maximum Sustained Traffic Rate” (Tmax) level is the maximum bandwidth permitted for a particular subscriber service flow within a DOCSIS system. Tmax is a maximum bandwidth level that is managed by the traffic scheduling algorithms within the Cable Modem Termination Systems (CMTSs).

This Tmax value is closely associated with the MSO’s “Advertised Billboard Bandwidth” (ABB) value, which is the bandwidth level that is advertised by MSO marketing teams as the maximum bandwidth that a subscriber may be likely to observe when they subscribe to a particular Service Level Agreement (SLA) level.

The “Busy-Hour Performance Goal” (BHPG) value is the bandwidth level that the MSO traffic engineer attempts to guarantee to any subscriber performing a Performance monitoring test during the busy-hour period. This BHPG value is a Quality of Experience goal or Quality of Experience (QoE) threshold that the MSO attempts to reach. Whether they reach that goal or not is determined by many factors, including the amount of bandwidth capacity provided to a particular service group, the number of subscribers

sharing the bandwidth capacity in the service group, the activity levels of those subscribers.

There is oftentimes a close association between a subscriber’s Tmax value and the subscriber’s Advertised Billboard Bandwidth (ABB) value. The ratio of the Tmax value to the Advertised Billboard Bandwidth value is an important quantity that must be determined by every MSO when they begin deploying DOCSIS HSD services. Within this paper, we will call this ratio the “Cushion Ratio” (Rc), where:

Rc = Tmax/(Advertised Billboard Bandwidth) (1)

Throughout much of this paper, we will assume that the Tmax value is usually set by the MSO to be ~20% higher than the Advertised Billboard Bandwidth value. This leads to a Cushion Ratio of Rc = 1.2. However, MSOs are known to use other Cushion Ratios of Rc = 1.1 or Rc = 1.0. Cushion Ratios of 1.0 are not typically recommended. The use of Cushion Ratios which are greater than 1.0 are highly recommended by the authors, because it will be shown this will often decrease system costs.

There is oftentimes a close association between a subscriber’s Advertised Billboard Bandwidth (ABB) value and the MSO’s Busy-Hour Performance Goal (BHPG) value. The ratio of Busy-Hour Performance Goal to Advertised Billboard Bandwidth is an interesting value that we will call the “Benevolence Ratio” (Rb), where:

Rb = (Busy-Hour Performance Goal)/(Advertised Billboard Bandwidth) (1)

MSOs can choose to set the Busy-Hour Performance Goal value to be equal to the Advertised Billboard Bandwidth value (Rb = 1.0). In this case, they are instructing their traffic engineers to make an attempt at ensuring that subscribers will experience Performance monitor scores that match their Advertised Billboard Bandwidth levels at all times- even during the congested busy-hour periods of time. However, it will result in higher costs to provide the extra bandwidth capacity within the network design.

To save money, MSOs can also take a different approach by setting Rb < 1.0, whereby they provide a service that offers Performance monitor scores that match the Advertised Billboard Bandwidth levels only during periods of low congestion, but that do not match Advertised Billboard Bandwidth levels during periods of high congestion.

For simplicity, within most of the rest of this paper, we will assume that the MSO is setting Rb = 1.0.

Since MSOs typically offer more than one SLA level to their subscriber base, there is a different Tmax value associated with each SLA level. The highest SLA level offers the highest Advertised Billboard Bandwidth value and the highest Tmax value. We will refer

to this highest of all Tmax values as the Tmax_max value within this paper. The Advertised Billboard Bandwidth corresponding to that highest Tmax value is typically the highest Advertised Billboard Bandwidth value. We will refer to this as the ABB_max value. The Busy-Hour Performance Goal corresponding to that highest ABB value is typically the highest Busy-Hour Performance Goal value- we will refer to this as the BHPG_max value. To illustrate this concept, consider a hypothetical MSO who offers three Service Level Agreement levels (Bronze, Silver, and Gold). Assume that these Service Level Agreement levels are defined as shown below:

• Bronze SLA o Advertised Billboard BW = 50 Mbps o Tmax = 60 Mbps (Rc = 1.2) o Busy-Hour Performance Goal = 50 Mbps (Rb = 1.0) • Silver SLA o Advertised Billboard BW = 100 Mbps o Tmax = 120 Mbps (Rc = 1.2) o Busy-Hour Performance Goal = 100 Mbps (Rb = 1.0) • Gold SLA o Advertised Billboard BW = 165 Mbps o Tmax = 200 Mbps (Rc = 1.2) o Busy-Hour Performance Goal =165 Mbps (Rb = 1.0)

For this hypothetical MSO, the Tmax_max value would be given by 200 Mbps. Its associated maximum Advertised Billboard Bandwidth (ABB_max) would be given by 200 Mbps/1.2 = ~165 Mbps. Its associated maximum Busy-Hour Performance Goal (BHPG_max) would also be given by ~165 Mbps. This 165 Mbps value would be the target value that the MSO should endeavor to provide to their Gold-level subscribers whenever those subscribers are trying to perform large-scale TCP file transfers or performance monitoring tests in the busy-hour. Since the MSO’s DOCSIS network is designed and sized to be shared by many transient users, statistics are used by traffic engineers to determine exactly the right amount of bandwidth capacity required within a service group to support the traffic of the many subscribers. These statistical analyses make assumptions about the likely levels of concurrency (simultaneous activity) between the many subscribers. However, there is always the possibility that the actual concurrency levels might be greater than the assumed concurrency levels, and when that happens, congestion is likely to result and it may lead to performance levels that are lower than this Advertised Billboard Bandwidth.

The Average Per-Subscriber Traffic Rate (Tavg) level is not a CMTS configuration setting (like Tmax) or a marketing value (like ABB) or a QoE goal (like BHPG). Instead, it is the actual bandwidth consumed by subscribers. Within this paper, we will define it to be the

average amount of bandwidth consumed by a single subscriber during busy-hour operation. It can be measured by an MSO using the following technique:

1) Identify a typical service group. Ideally, this measurement should be done on an uncongested service group to keep from biasing the results with measurements from congested modems. 2) Count the total number of subscribers (Nsub) within the service group. The Nsub value includes the subscribers that are active as well as subscribers that are idle 3) Measure the number of bytes (B) passed into the service group within a given window of time (W), where W is measured in seconds. Do this during the busy-hour period of time, which is typically from 8pm to 9pm at night 4) Calculate the average amount of bandwidth consumed by a subscriber using the formula: Tavg = 8*B/(W*Nsub) (2)

The fundamental goal of traffic engineering is to determine the “minimum” amount of bandwidth capacity (C) required to maintain “adequately high” QoE levels for all subscribers who are sharing the bandwidth capacity within a particular service group. Designing for the “minimum” amount of bandwidth capacity helps to control costs. Ensuring that all subscribers maintain an “adequately high” QoE level helps to reduce subscriber churn.

This is a challenging task, because it is oftentimes difficult to determine when the QoE levels are adequate. Consider the hypothetical MSO described above with the three HSD ABB levels given by Bronze = 50 Mbps, Silver = 100 Mbps, and Gold = 165 Mbps. As a result, the Tmax_max value is 200 Mbps. Assume that a particular HSD service group consists of 500 homes passed and that a 50% take-rate in the HSD service results in Nsub = 250 subscribers sharing the bandwidth capacity (C) within that service group. Out of those 250 subscribers, assume that 175 use Bronze SLAs, 50 use Silver SLAs, and 25 use Gold SLAs. Assume that the average bandwidth per subscriber is measured during the busy-hour to be Tavg = 500 kbps. The task of the traffic engineer is to determine the minimum bandwidth capacity (C) that will keep those 250 subscribers happy by ensuring that their QoE levels are high enough.

Traffic engineering activities of the past have oftentimes used rules-of-thumb to help define required bandwidth capacities. A typical rule-of-thumb from the past might have offered bandwidth capacity (C) greater than the (Nsub)*(Tavg) value. So in the above hypothetical case, this would imply offering a bandwidth capacity of C = (250)*(500 kbps) = 125 Mbps. Another rule-of-thumb might have offered bandwidth capacity into a service group that is equal to 2x the Tmax_max value. In the above hypothetical case, this would imply offering a bandwidth capacity of C = 2x(200 Mbps) = 400 Mbps. Yet

another rule-of-thumb might have doubled the bandwidth capacity whenever the (Nsub)*(Tavg) value exceeded 70% of the current bandwidth capacity.

All of the rule-of-thumb approaches above yield different results. But which rule-of- thumb approach is correct? Which rule-of-thumb approach yields the ideal bandwidth capacity level that ensures reasonable QoE levels. That is not clear from the analyses above.

The problem with the rule-of-thumb approaches of the past is that they did not tie the bandwidth capacity to predicted QoE levels. As a result, they did not easily permit the MSOs to adjust their HFC plant bandwidth capacity to consciously increase or decrease their subscriber QoE levels. In fact, none of these rule-of-thumb formulae even had any inputs that allowed the MSO to specify the desired QoE level, so these simple formulae may not be adequate today or in the future. As a result, this paper will propose the inclusion of a distinct QoE parameter within the traffic engineering formula to better characterize QoE performance levels. QUALITY OF EXPERIENCE BASICS

QoE is a fundamental measure of the user’s satisfaction level. Unfortunately, QoE monitoring is becoming more difficult over time, because the nature of the traffic propagating over networks has changed quite often throughout history. In recent years, these changes have been taking place on at least four different fronts:

1) There is a much larger mix of traffic types (Web-browsing, IP video, VoIP, Peer- to-peer downloads, gaming, etc.) that are now making up the aggregated bandwidth within typical DOCSIS networks. 2) Each of the different traffic types has its own unique and different traffic requirements and sensitivities. Some of the traffic types are sensitive to changes in packet stream bandwidth (ex: Web-browsing, IP video, Peer-to-peer downloads) while others are sensitive to changes in packet delay or jitter or packet loss (ex: VoIP, gaming). 3) Maximum Sustained Traffic Rates (Tmax values) for cable modem subscribers now range over much larger values (for different SLAs) than they did in the past. 4) The arrival of Adaptive Bit-Rate IP video traffic on the DOCSIS network introduces a new traffic type that can quickly expand to fill unused bandwidth capacity and that can also be compressed during periods of congestion. This new breed of traffic makes it more difficult to determine QoE levels. [Cl2013]

Thus, Monitoring of the QoE level for each of the different traffic types and different SLAs requires that special attention be paid to the appropriate metrics. In this paper, we will focus on the QoE levels of three particular traffic types that we have deemed to be

extremely important in ensuring that subscribers will be satisfied with their HSD service in the future. Those three traffic types are:

A. IP video (comprises ~60% of network bandwidth today) B. Web-browsing (comprises ~15% of network bandwidth today) [Sa2014] C. Performance monitoring scores (which many subscribers use to ascertain their overall service level- offered by services such as SamKnows and DSLReports and MSOs themselves)

For IP video activities, we assume that the large jitter buffers designed into most of today’s Adaptive Bit-Rate (ABR) delivery systems help guard against most issues resulting from minimal packet delay or jitter or loss. Instead, we focus on the average bandwidth metric for determining the QoE of this service type. Decreased average bandwidth levels can cause the ABR algorithms to throttle the resolution of these video streams. We will assume that many subscribers who will feed their ABR IP video feeds to their large-screen TVs may complain about their QoE levels if this throttling occurs and precludes them from viewing their videos in the highest resolution formats of their time. As a result, we will assume that IP video will have the following QoE requirements that will vary over time, and we will study it at three distinct times:

• 2014 IP Video QoE requirements: 1080p video using H.264 compression; Minimum bandwidth = ~5 Mbps • 2018 IP Video QoE requirements: 4K video using HEVC compression; Minimum bandwidth = ~20 Mbps • 2021 IP Video QoE requirements: 8K video using HEVC compression; Minimum bandwidth = ~75 Mbps

For Web-browsing activities, we will assume that the download times are the dominant metrics that will ultimately determine good QoE levels. We assume that subscribers will prefer download times to be less than 4 seconds to yield acceptable QoE levels. [Pr2012] These download times can be influenced by many factors, but they are most likely dominated by the bandwidth available to each subscriber on the network. Average Web-pages of 2014 are approximately 1.6 Mbytes long, and they have been growing at a rate of ~25% per year. [Ca2014] We will assume that the Web-pages will continue to grow at that rate into the future. We will also assume that Web-page read times will remain at ~22 seconds. As a result, we will assume that Web-browsing will have the following QoE requirements over time:

• 2014 Web-browsing QoE requirements: 13 Mbit page must be downloaded in 1.5 seconds with 22 second windows between successive Web page reads • 2018 Web-browsing QoE requirements: 32 Mbit page downloaded in 1.5 seconds with 22 second windows between successive Web page reads

• 2021 Web-browsing QoE requirements: 62 Mbit page downloaded in 1.5 seconds with 22 second windows between successive Web page reads

For Performance monitoring activities (such as those provided by SamKnows and DSLReports), many different parameters are oftentimes tested. However, the one parameter that likely dominates the subscriber’s perception of their service level is the result from the downstream bandwidth test. We will assume that this test performs large-file transfers using HTTP downloads. We will also assume that the low-bandwidths associated with the Slow-Start state of TCP connections are avoided using various techniques during the test. We will also assume a typical Advertised Billboard Bandwidth of 165 Mbps exists today, and that a 50% CAGR exists for that bandwidth. As a result, we will assume that Performance monitoring tests will have the following QoE requirements over time:

• 2014 Performance monitoring QoE requirements: Advertised Billboard BW=165 Mbps; Tmax_max=200 Mbps; Avg HTTP download BW in Perf. Monitoring test must be at least 165 Mbps • 2018 Performance monitoring QoE requirements: Advertised Billboard BW=850 Mbps; Tmax_max=1.0 Gbps; Avg HTTP download BW in Perf. Monitoring test must be at least 850 Mbps • 2021 Performance monitoring QoE requirements: Advertised Billboard BW=2.7 Gbps; Tmax_max=3.3 Gbps; Avg HTTP download BW in Perf. Monitoring test must be at least 2.7 Gbps

The focus of the next section will be on the development of a new traffic engineering formula that will give MSOs the ability to specify the QoE levels for their subscribers as they architect the required bandwidth capacities for their future HFC networks. THE NEW TRAFFIC ENGINEERING FORMULA

The work within this paper was initially spawned by efforts from some of the authors to define a new traffic engineering formula that would provide more accuracy than the formulae used in the past. Early results of that work were outlined in a previous work. [Em2014]

When the authors of the current paper continued with follow-on work to the previous paper, they attempted to simplify the original formula. They also began to develop a useful (and simpler) formula that would permit MSO traffic engineers to specify the required DOCSIS HSD bandwidth capacity (C) for a service group along with the ability to also specify a specific QoE level. The authors also wanted to validate the accuracy of the new formula using simulation techniques.

From the beginning, it was clear that the new formula would need to include several variables related to the DOCSIS network. These variables included:

1) The number of subscribers (Nsub) within the service group. As the number of subscribers is increased, it should be apparent that the amount of shared bandwidth capacity (C) should be increased as well. 2) The average busy-hour bandwidth per subscriber (Tavg). As the average bandwidth per subscriber is increased, it should be apparent that the amount of shared bandwidth capacity (C) should be increased as well. 3) The Maximum Sustained Traffic Rate (Tmax) offered by the MSO. As the Tmax values are increased, it should be apparent that the amount of bandwidth capacity (C) should be increased as well.

One other important variable was desired within this formula. That variable was a parameter that somehow specified the desired Quality of Experience level that subscribers would ultimately experience. The authors decided to use a single constant called the K value to specify the QoE level. The idea behind the K value was that setting of the K value to 0 would create a low QoE level, and setting of the K value to a higher number would create a higher QoE level.

The new formula developed for this paper specifies the required bandwidth capacity (C) for a service group, and it is given by THE NEW TRAFFIC ENGINEERING FORMULA (BASED ON Tmax_max):

C >= (Nsub*Tavg) + (K*Tmax_max), (3)

where: C is the required bandwidth capacity for the service group Nsub is the total number of subscribers within the service group Tavg is the average bandwidth consumed by a subscriber during the busy-hour K is the QoE constant (larger values of K yield higher QoE levels)… where 0 <= K <= infinity Tmax_max is the highest Tmax offered by the MSO.

This formula is a simple two-term equation. Its simplicity is part of its beauty. The first term (Nsub*Tavg) allocates bandwidth capacity to ensure that the aggregate average bandwidth generated by the Nsub subscribers can be adequately carried by the service group’s bandwidth capacity. The author’s view this first term as the “DC component” of traffic that tends to exist as a continuous flow of traffic during the busy-hour period.

There are obviously bursty fluctuations that will occur (the “AC component” of traffic) which can force the instantaneous traffic levels to both fall below and rise above the DC traffic level. Fluctuations below the DC traffic level can occur whenever there is a

temporary lull in packet transmissions. Fluctuations above the DC traffic level can occur whenever there is an increase in the concurrency level (the number of simultaneously transmitting or receiving subscribers). The second term (K*Tmax_max) is added in an attempt to increase the probability that all subscribers- including those with the highest Tmax values- will experience good QoE levels for most of the fluctuations that go above the DC traffic level.

The second term in the formula (K*Tmax_max) has an adjustable parameter defined by the K value. This parameter allows the MSO to increase the K value and add bandwidth capacity headroom that helps provide better QoE to their subscribers within a service group. In addition, the entire second term is scaled to be proportional to the Tmax_max value, which is the maximum Tmax value that is being offered to subscribers. The constant of proportionality is the K value, which can be set to any value from zero to infinity.

(Note: In the previous paper [Em2014], the K value was specified by the product of two values (hr*s), where hr was a headroom value and s was a simultaneous transmission period. However, in this paper, we are proposing to replace those two values by the single K value, which is ultimately proportional to the desired QoE level. A change in the K value results in a corresponding change within the QoE levels experienced by the subscribers who are sharing the service group bandwidth capacity (C). Lower K values yield lower QoE levels, and higher K values yield higher QoE levels).

If the MSO chooses to set the K value to its minimum value of zero, then the second term in the above formula is forced to be zero and only the first term is used to specify the required bandwidth capacity (C) of the service group. In this case, costs (and bandwidth capacities) are minimized, because the required bandwidth capacity will only be equal to the expected average bandwidth in the busy-hour, and the resulting QoE levels will be fairly low. As a result, bursts of bandwidth exceeding the (Nsub*Tavg) value will be throttled by the available bandwidth capacity, causing a reduction in file download speeds. In addition, any subscriber who monitors their bandwidth levels during the busy-hour period would undoubtedly find that their bandwidth levels are far below their Advertised Billboard Bandwidth levels. As a result, low QoE levels would result.

On the other hand, if the MSO chooses to set the K value to a very large value, then the second term in the above formula would become quite large and the resulting bandwidth capacity (C) of the service group would also become quite large. This would result in much higher system costs to supply the required bandwidth capacity, but it would also ensure that the bandwidth capacity could easily support the bursts of bandwidth that periodically exceed the (Nsub*Tavg) value. In addition, any subscriber who monitors their bandwidth levels during the busy-hour period would undoubtedly find that their bandwidth levels are very close to their Advertised Billboard Bandwidth

levels. As a result, high QoE levels would result. This tradeoff between costs and subscriber QoE levels is a common theme that challenges traffic engineers on a daily basis.

In previous papers [Cl2013] [Em2014], a similar formula assumed (without proof) that a K value of ~1.0 would yield acceptable and adequate QoE results. Using that simplifying assumption, one could predict the HSD bandwidth capacities needed in the future by extrapolating the Nsub, Tmax_max, and Tavg values out into the future using trends of the past. As an example, one might assume that the Nsub values remain fixed until node-splits occur, at which point the Nsub value for each service group is cut in half. One might also assume that the Tmax_max value (which for some MSOs is ~200 Mbps today) might grow at a 50% CAGR (as predicted by the Nielson Law). One might also assume that the Tavg value (which for some MSOs is ~500 kbps today) might also grow at a 50% CAGR. In the previous papers, formulae were developed to calculate the Tmax_max and Tavg values as a function of time. These formulae are stated in a slightly modified form below. The formula for Tmax_max at a time Y years from now (assuming a 50% CAGR) is given by:

Tmax_max = (200 Mbps)*(1.5)Y (4a)

and the formula for Tavg at a time Y years from now (assuming a 50% CAGR) is given by:

Tavg = (500 kbps)*(1.5)Y. (4b)

Using these formulae (which apply the above 50% CAGR assumptions), one can plot the predicted Tavg and Tmax_max values as a function of time (in years) as shown in Figure 1. Tavg is shown in black and Tmax_max is shown in blue. It is important to note that this figure is a semi-log plot. In addition to Tavg and Tmax_max curves, the figure also displays the yellow and maroon and purple curves for (Nsub*Tavg). This is the important first term in formula (3), and it represents the average aggregate bandwidth generated by the accumulated traffic for all subscribers within the service group during the busy- hour period. Each of the yellow (or maroon or purple) curves in the figure represents the average bandwidth generated for a service group of a different size, and each adjacent curve in the figure is associated with a service group that is one node-split away from the service group for the curve directly above it.

Downstream BW as a function of time (w/ ~50% Annual Growth Rate) ~100 Gbps in 2030 100G 512 Subs/SG (2x 512 HHP Nodes) 256 Subs/SG (512 HHP Nodes) D3.1 DS Limit = 10.8 Gbps (1200 MHz) 128 Subs/SG (256 HHP Nodes) 1.2 GHz of DOCSIS 3.1 Chans. 10G 64 Subs/SG (128 HHP Nodes) 116 DOCSIS 3.0 Chans. D3.0 DS Limit = 4.9 Gbps (750 MHz) 32 Subs/SG (64 HHP Nodes) 64 DOCSIS 3.0 Chans. 16 Subs/SG (32 HHP Nodes) 32 DOCSIS 3.0 Chans. Avg BW/SG 16 DOCSIS 3.0 Chans. 1G 8 DOCSIS 3.0 Chans. 4 DOCSIS 3.0 Chans. 2 DOCSIS 3.0 Chans. 100M 1 DOCSIS 3.0 Chan. 10M 16000 Subs/SG (Many Nodes) Avg BW/SG ~332 Mbps 8000 Subs/SG (Many Nodes) ~30 in 2030 4000 Subs/SG (Many Nodes) 1M 2000 Subs/SG (Many Nodes) Mbps 1000 Subs/SG in Avg BW/sub 512 Subs/SG 2010 100k Avg BW/SG ~150 kbps ~100 kbps in 1997 Max BW/sub in 2010 10k

DS BW for Modems (bps)DS BW for Modems 1k ~500 bps Nielsen’s Law in 1997 Tmax 300 bps 100 in 1982 Consumption- Consumption Tmax- Oriented Period + Tmax- Oriented 10 (Tmax < 1 Ch & Oriented Period Period Tmax < Avg SG (Tmax =~ Avg SG (Tmax >> Avg SG Consumption) Consumption) Consumption) 1 Year 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022 2026 2030

The Era The Era The Era The Era of of of of Dial-Up Cable 3.0 3.1 Modems Modems Modems Modems Figure 1: DOCSIS HSD Downstream Traffic Engineering Predictions for the Future with CAGR=50%

Then using formula (3), one can also plot the minimum required bandwidth capacity (C) for any service group of size Nsub as a function of time. This is done in Figure 2, where the yellow and maroon and purple plots now show the required bandwidth capacity. As was illustrated in [Cl2014], an interesting phenomena unveils itself within the figure. Node-splits can only impact one of the variables within formula (3)- they can only change the Nsub value. Under normal circumstances, a node-split will cause the Nsub value to be halved. Thus, a node-split will obviously reduce the magnitude of the first term (Nsub*Tavg) within formula (3). However, nodes-splits have no effect on the second term (K*Tmax_max) within formula (3). As a result, after a large number of node-splits have caused the first term in the formula to be reduced to very small and negligible values, the required bandwidth capacity (C) specified by formula (3) will ultimately be dominated by the second term (K*Tmax_max), and further node-splits will cease to have any significant effect on the required bandwidth capacity (C). Eventually, even a small service group with Nsub = 1 will still require a bandwidth capacity of at least the Tmax value for that single subscriber, so the Tmax_max level becomes a floor function for the required bandwidth capacity (C). This implies that node-splits will eventually lose much of their value as a tool for reducing required bandwidth capacities within service groups. Fortunately, that day is still 3 or 4 node-splits away from today, which should carry many MSOs to the 2020 or 2025 time-frame as illustrated within the figure.

Downstream BW as a function of time (w/ ~50% Annual Growth Rate) Agg BW/SG 512 Subs/SG (2x 512 HHP Nodes) split to 64 subs/SG 64to split split to 256to subs/SG split 128to subs/SG split - - - 256 Subs/SG (512 HHP Nodes) 128… 64… 32 100G 16 Subs/SG (32 HHP Nodes) Node Node Node split to 64 subs/SG 64to split split to 256to subs/SG split 128to subs/SG split - - - D3.1 DS Limit = 10.8 Gbps (1200 MHz) ~100 1.2 GHz of DOCSIS 3.1 Chans. Node 10G Node Node Gbps 116 DOCSIS 3.0 Chans. D3.0 DS Limit = 4.9 Gbps (750 MHz) 64 DOCSIS 3.0 Chans. in 32 DOCSIS 3.0 Chans. 16 DOCSIS 3.0 Chans. 1G 2030 8 DOCSIS 3.0 Chans. 4 DOCSIS 3.0 Chans. 2 DOCSIS 3.0 Chans. 100M 1 DOCSIS 3.0 Chan. 10M 16000 Subs/SG (Many Nodes) Agg BW/SG ~332 Mbps 8000 Subs/SG (Many Nodes) ~30 in 2030 4000 Subs/SG (Many Nodes) 1M 2000 Subs/SG (Many Nodes) Mbps 1000 Subs/SG Avg BW/sub 512 Subs/SG in Proposed Human Factors Formula 2010 100k Agg BW/SG ~150 kbps for Quality of Experience (QoE): ~100 kbps in 1997 Max BW/sub in 2010 10k Required Service Group Capacity = K* Max BW/sub + Num_subs * Avg BW/sub DS BW for Modems (bps)DS BW for Modems 1k ~500 bps Nielsen’s Law in 1997 Tmax 300 bps (K=1 may yield acceptable QoE) 100 in 1982 Consumption- Consumption Tmax- Oriented Period + Tmax- Oriented 10 (Tmax < 1 Ch & Oriented Period Period Tmax < Avg SG (Tmax =~ Avg SG (Tmax >> Avg SG Consumption) Consumption) Consumption) 1 Year 1982 1986 1990 1994 1998 2002 2006 2010 2014 2018 2022 2026 2030

The Era The Era The Era The Era of of of of Dial-Up Cable 3.0 3.1 Modems Modems Modems Modems Figure 2: DOCSIS HSD Required Downstream Bandwidth Capacity with 50% CAGR

It should be clear that the plots in Figure 2 could prove to be invaluable tools to MSOs as they begin to explore plans to modify their HFC spectra to accommodate HSD traffic growth going forward into the future. However, before these plots can be trusted, one must first provide some level of proof that the selected value of K=1 within formula (3) will indeed yield desirable QoE levels. Proving that fact using simulations is the fundamental goal of the next section in this document. VALIDATING THE NEW TRAFFIC ENGINEERING FORMULA

Our hypothesis is that formula (3) is a valid formula for calculating the minimum bandwidth capacity needed within a service group. We also need to determine appropriate values of K within that formula to make the formula useful.

Validating that a formula (such as formula (3)) is valid is a challenging task. The state- space associated with the formula is enormous. An exhaustive analysis of the entire state-space and a full proof is probably not possible given the many variations of traffic types that exist and the many different QoE metrics that need to be monitored and the many different mixes of traffic type blends that can exist and the many different service group sizes that can exist and the many different conditions that can exist within those service groups. In addition, all of those conditions will obviously change over time as

new applications and new traffic types evolve, so proving that formula (3) is valid today and that it will also be valid in the future is an even bigger challenge.

As a result, we have instead decided to use simulations to assist us in at least providing some degree of confidence that formula (3) is a reasonably valid formula to use for most of the typical MSO service group scenarios. We will also attempt to offer some guidelines that MSOs can use in selecting appropriate K values to ensure particular QoE levels to their subscribers.

In order to accomplish this goal, a Java-based simulation environment was created that included several important sub-systems. The simulation environment permitted us to interconnect server objects, Cable Modem Termination System (CMTS) objects, and Cable Modem (CM) objects in various ways to create different types and sizes of service groups as shown in Figure 3. These various arrangements of objects are known as Network Models. In all Network Models used within this paper, the Upstream bandwidth was selected to be large enough to not impact the performance of the Cable Modem clients.

FTP Server … FTP Server Test Server

down (9 Gbps) RTT = 10 msec up (2 Gbps)

CMTS

docsis (BW Mbps)

atdma

cm_user cm_test

FTP Traffic: Tmax = 25 Mbps Speed Test Traffic: Tmax = 25 Mbps N Concurrent DS TCP Sessions 1 DS TCP Sessions 6.5 MB Blocks => 385 Kbps Avg. Rate 320 MB Block Uniform 10 - 30 sec Spacing 60 sec Intervals

Figure 3: Example Network Model Within The Simulation Environment

Most of the clients within our simulation environment were assumed to be involved in one of the following three Internet activities: IP Video streaming, Web-browsing, or Performance monitor testing. These were the three activities identified in the QoE section of this paper as having profound effects on a subscriber’s perception of his QoE level.

Since all three of these activities usually involve the use of TCP-based HTTP downloads, the models for the three user types were quite similar. However, we were able to customize parameters within each of the models to better simulate each of the particular activities. Thus, IP Video streaming models performed many successive HTTP GETs of small-sized files (which represent the IP Video segment files) with only ~2 second gaps between successive HTTP GETs. Web-browsing models performed HTTP GETs of medium-sized files (which represented the HTML files associated with Web pages), but there was a randomized and somewhat lengthy window of time (~20 seconds) placed between successive HTTP GETs to simulate the subscriber’s viewing of the Web page. Performance monitor testing models performed HTTP GETs of large-sized files and ensured that Congestion Windows were initialized to high levels to simulate the types of tests performed by Performance monitor tools. Different mixes of these models were permitted to create many types of traffic on the service group. This traffic is known as a Traffic Model. Analytical data was collected within the simulation environment to permit us to quickly identify the QoE levels for each of the different Network Models and Traffic Models as we changed the blend of users and traffic types.

To analyze the state-space across a large number of years, we opted to perform simulation runs for three different epochs in time: 2014, 2018, and 2021. We therefore had to create predictions on what a “typical” service group and traffic blend might look like at each of those different points in time. (Note: We held the Nsub value fixed in these predictions. The Nsub values will undoubtedly be scaled down as a result of bandwidth growth and associated node-splits, but we will explore that later in this paper). The details surrounding the predictions for each of these epochs in time are given in Table 1.

In upcoming sections of the paper, we will create traffic blends that exactly mimic the blends shown in the table. However, in this section, we will create more generalized service group models that approximate the behavior of the blends shown in the table using a single idealized customer pool, where each of the Cable Modem objects within the customer pool pulls down an HTTP file of random size with a randomized delay between successive HTTP GETs (similar to Web-browsing). Early experiments showed that this particular model stressed the system more than other models, so the results are (in essence) a worst-case scenario. We have a belief that using this model as background traffic, the results of the single Performance monitor test within the simulator will go a long way to characterize the performance of the various traffic types, so we will lean heavily on these observations of the QoE levels for the Performance monitoring tests within these initial simulation runs. Thus, exactly one Cable Modem object must be set up to perform Performance monitor testing within the middle of all of the other HTTP downloads that are occurring as a result of the activities of the other Cable Modem objects within the simulation run.

Year 2014 2018 2021 Tavg 500 kbps 2.5 Mbps 8.4 Mbps Tmax_max 200 Mbps 1 Gbps 3.3 Gbps Advertised Billboard Bandwidth (ABB_max) 165 Mbps (Rc=1.2) 850 Mbps (Rc=1.2) 2.7 Gbps (Rc=1.2) Busy-Hour Performance Goal (BHPG_max) 165 Mbps (Rb=1.0) 850 Mbps (Rb=1.0) 2.7 Gbps (Rb=1.0) Nsub 256 256 256 (Nsub*Tavg) 128 Mbps 640 Mbps 2.1 Gbps (K*Tmax_max) if K = 1 200 Mbps 1 Gbps 3.3 Gbps (Nsub*Tavg)+(K*Tmax_max) if K = 1 328 Mbps 1.6 Gbps 5.4 Gbps (9 DOCSIS DSs) (45 DOCSIS DSs) (150 DOCSIS DSs)

# Active Subs/Service Group (20% concurrency) 50 50 50 IP Video Segment Time 2 sec 2 sec 2 sec IP Video Segment File Size 1.25 Mbytes 5 MBytes 18.8 Mbytes IP Video Average BW 5 Mbps 20 Mbps 75 Mbps

Web-browsing Page Size 1.6 Mbytes 4.0 Mbytes 7.8 Mbytes Web-browsing Avg Inter-Page Time 22 seconds 22 seconds 22 seconds Table 1: Predictions on Future Service Group Parameters

Armed with the simulation tool and the predictions of Table 1, we proceeded to explore the validity of formula (3) in many different service group environments. Our basic methodology for our simulations followed the analytical approach outlined below: 1. For the particular year of interest (2014, 2018, or 2021), create a simulation environment with a Parameter Set based on the predictions listed in Table 1 2. Construct a Network Model and a Traffic Model for this Parameter Set… include exactly one Cable Modem object in the model that will be implementing Performance monitor tests in the midst of all of the other Cable Modem object activities 3. Change the bandwidth capacity within the service group using formula (3) and iterating on the K value from 0.3 to 2.0 (which seem like reasonable values for K) a) Assign the bandwidth capacity predicted by the formula (3) hypothesis for this particular value of K b) Simulate 20 minutes of Real-time Traffic loads (including the Performance monitor tests) c) Plot the Performance monitor test measurements versus the value of K 4. Record the “minimum K value” for which the Performance monitor test exceeds the desired QoE Threshold (defined by the Busy-Hour Performance Goal)

Repeat Steps 1 – 4 for Various Combinations of Parameter Sets, trying to change one parameter at a time to measure the sensitivity of the “minimum K value” (determined in Step 4) to these parameter changes. In a certain sense, we are therefore trying to determine the partial derivative of the “minimum K value” relative to each of the parameters.

The figures below show a subset of the outputs from simulation runs that were performed as we explored the large state-space associated with different service group

environments. In all of these runs, we attempted to fill the service group bandwidth capacity with HTTP traffic that created an average bandwidth level given by (Nsub*Tavg), however the instantaneous data rate was allowed to vary as it would in the real world with different concurrency levels as a function of time. In each run, we assume that 20% of the subs are “active” (i.e.- involved in some transactions- whether waiting to perform an HTTP transfer or performing an HTTP transfer). In each run, the Rc value was set to be 1.2 and the Rb value was 1.0, so the Advertised Billboard Bandwidth and the Busy-Hour Performance Goal was always set to be at a level that is at 83% of the Tmax value.

In each figure, we were exploring a different Network Model. In each figure, the service group bandwidth capacity within that Network Model was varied as the K value was varied, and the resulting performance metrics were extracted and plotted as a function of the K value.

A summary of the figures and results for the 2014 simulations are given below:

2014 Baseline Reference 200 8

150 6 Tmax_max

100 4 sec

Mbps Perf Goal

50 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4a: 2014 Simulation Results with Baseline Reference

• In all cases, Bandwidth Capacity (as a function of K) = Nsub*Tavg + K*Tmax

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 325 Mbps Nsub*Tavg + 1*Tmax Minimum K value (to match 1.15 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 355 Mbps Nsub*Tavg + K*Tmax

2014 (2*Nsub) 200 8

150 6 Tmax_max

100 4 sec

Mbps Perf Goal

50 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4b: 2014 Simulation Results with Nsub Doubled

• Same as 2014 Baseline Reference, but with subscribers doubled (Nsub = 500… 100 active)

Parameter Value Comment Nsub 500 100 Active Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 450 Mbps Minimum K value (to match 1.3 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 510 Mbps

2014 (2*Tavg) 200 8

150 6 Tmax_max

100 4 sec

Mbps Perf Goal

50 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4c: 2014 Simulation Results with Tavg Doubled

• Same as 2014 Baseline Reference, but with Tavg doubled (Tavg = 1 Mbps)

Parameter Value Comment Nsub 250 50 Active Tavg 1 Mbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 450 Mbps Minimum K value (to match 1.3 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 510 Mbps

2014 (2*Tmax) 400 8

300 6 Tmax_max

200 4 sec

Mbps Perf Goal

100 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4d: 2014 Simulation Results with Tmax Doubled

• Same as 2014 Baseline Reference, but with Tmax doubled (Tmax = 400 Mbps)

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 400 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 525 Mbps Minimum K value (to match 1.0 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 525 Mbps

2014 (4*Interval) 200 8

150 6 Tmax_max

100 4 sec

Mbps Perf Goal

50 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4e: 2014 Simulation Results with File Size & HTTP Interval Quadrupled

• Same as 2014 Baseline Reference, but with HTTP file size quadrupled and GET Interval quadrupled (file size = 25 Mbytes, Interval = 80 seconds)

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 25 MB Avg HTTP GET Interval 80 sec BW Cap. (for K=1) 325 Mbps Minimum K value (to match 1.3 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 422 Mbps

2014 (Bronze) 200 8

150 6 Tmax_max

100 4 sec

Mbps Perf Goal

50 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 4f: 2014 Simulation Results with 50% of Subs Having Lower Tmax

• Same as 2014 Baseline Reference, but with 90% of subscribers having a lower Tmax of 50 Mbps

Parameter Value Comment Nsub 250 50 Active (45 Bronze, 5 Gold) Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 325 Mbps Minimum K value (to match 1.0 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 325 Mbps

A summary of the figures and plots and results for the 2018 simulations are given below:

2018 Reference 1000 8

750 6 Tmax_max

500 4 sec

Mbps Perf Goal

250 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5a: 2018 Simulation Results with Baseline Reference

• In all cases, Bandwidth Capacity (as a function of K) = Nsub*Tavg + K*Tmax_max

Parameter Value Comment Nsub 250 50 Active Tavg 2.5 Mbps Tmax 1 Gbps Avg HTTP File Size 31.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 1.6 Gbps Minimum K value (to match 1.15 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 1.8 Gbps

2018 (2*Nsub) 1000 16

750 12 Tmax_max

500 8 sec

Mbps Perf Goal

250 4 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5b: 2018 Simulation Results with Nsub Doubled

• Same as 2018 Baseline Reference, but with subscribers doubled (Nsub = 500… 100 active)

Parameter Value Comment Nsub 500 100 Active Tavg 2.5 Mbps Tmax 1 Gbps Avg HTTP File Size 31.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 2.2 Gbps Minimum K value (to match 1.4 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 3.1 Gbps

2018 (2*Tavg) 1000 16

750 12 Tmax_max

500 8 sec

Mbps Perf Goal

250 4 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5c: 2018 Simulation Results with Tavg Doubled

• Same as 2018 Baseline Reference, but with Tavg doubled (Tavg = 5 Mbps)

Parameter Value Comment Nsub 250 50 Active Tavg 5 Mbps Tmax 1 Gbps Avg HTTP File Size 31.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 2.2 Gbps Minimum K value (to match 1.6 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 3.5 Gbps

2018 (2*Tmax) 2000 8

1500 6 Tmax_max

1000 4 sec

Mbps Perf Goal

500 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5d: 2018 Simulation Results with Tmax Doubled

• Same as 2018 Baseline Reference, but with Tmax doubled (Tmax = 2 Gbps)

Parameter Value Comment Nsub 250 50 Active Tavg 2.5 Mbps Tmax 2 Gbps Avg HTTP File Size 31.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 2.6 Gbps Minimum K value (to match 0.9 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 2.3 Gbps

2018 (4*Interval) 1000 10

8 750 6 Tmax_max

500 sec Mbps 4 Perf Goal 250 Perf Mon Test 2 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5e: 2018 Simulation Results with File Size & HTTP Interval Quadrupled

• Same as 2018 Baseline Reference, but with HTTP file size quadrupled and GET Interval quadrupled (file size = 125 Mbytes, Interval = 80 seconds)

• “Minimum K value” to surpass Busy-Hour Performance Goal = 0.85

Parameter Value Comment Nsub 250 50 Active Tavg 2.5 Mbps Tmax 1 Gbps Avg HTTP File Size 125 MB Avg HTTP GET Interval 80 sec BW Cap. (for K=1) 1.6 Gbps Minimum K value (to match 0.9 Web Resp > 1.5 sec Perf Goal) (but very big Web pages) BW Cap. (to match Perf Goal) 1.4 Gbps

2018 (Bronze) 1000 8

750 6 Tmax_max

500 4 sec

Mbps Perf Goal

250 2 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 5f: 2018 Simulation Results with 50% of Subs Having Lower Tmax

• Same as 2018 Baseline Reference, but with 50% of subscribers having a lower Tmax of 250 Mbps

Parameter Value Comment Nsub 250 50 Active (45 Bronze, 5 Gold) Tavg 2.5 Mbps Tmax 1 Gbps Avg HTTP File Size 31.25 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 1.6 Gbps Minimum K value (to match 1.0 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 1.6 Gbps

A summary of the figures and plots and results for the 2021 simulations are given below:

2021 Reference 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6a: 2021 Simulation Results with Baseline Reference

• In all cases, Bandwidth Capacity (as a function of K) = Nsub*Tavg + K*Tmax_max

Parameter Value Comment Nsub 250 50 Active Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 5.4 Gbps Minimum K value (to match 1.1 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 5.9 Gbps

2021 (2*Nsub) 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6b: 2021 Simulation Results with Nsub Doubled

• Same as 2021 Baseline Reference, but with subscribers doubled (Nsub = 500… 100 active)

• “Minimum K value” to surpass Busy-Hour Performance Goal = 0.9

Parameter Value Comment Nsub 500 100 Active Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 7.5 Gbps Minimum K value (to match 1.2 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 9 Gbps

2021 (2*Tavg) 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6c: 2021 Simulation Results with Tavg Doubled

• Same as 2021 Baseline Reference, but with Tavg doubled (Tavg = 16.8 Mbps)

Parameter Value Comment Nsub 250 50 Active Tavg 16.8 Mbps Tmax 3.3 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 7.5 Gbps Minimum K value (to match 1.4 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 10.5 Gbps

2021 (1.5*Tmax) 5000 7 6 4000 5 3000 4 Tmax_max sec Mbps 2000 3 Perf Goal 2 Perf Mon Test 1000 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6d: 2021 Simulation Results with Tmax 50% Higher

• Same as 2021 Baseline Reference, but with Tmax doubled (Tmax = 5 Gbps)

Parameter Value Comment Nsub 250 50 Active Tavg 8.4 Mbps Tmax 5 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 7.1 Gbps Minimum K value (to match 0.7 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 5.0 Gbps

2021 (3*Interval) 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6e: 2021 Simulation Results with File Size & HTTP Interval Tripled

• Same as 2021 Baseline Reference, but with HTTP file size quadrupled and GET Interval quadrupled (file size = 420 Mbytes, Interval = 80 seconds)

Parameter Value Comment Nsub 250 50 Active Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 315 MB Avg HTTP GET Interval 60 sec BW Cap. (for K=1) 5.4 Gbps Minimum K value (to match 1.1 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 5.9 Gbps

2021 (Bronze) 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 6f: 2021 Simulation Results with 50% of Subs Having Lower Tmax

• Same as 2021 Baseline Reference, but with 50% of subscribers having a lower Tmax of 825 Mbps

Parameter Value Comment Nsub 250 50 Active (45 Bronze, 5 Gold) Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec BW Cap. (for K=1) 5.4 Gbps Minimum K value (to match 0.9 Web Resp < 1.5 sec Perf Goal) BW Cap. (to match Perf Goal) 4.9 Gbps

Interestingly, the minimum required value of K always fell in the range from 0.7 to 1.5 for all tests performed. However, most of these simulations assumed a worst-case situation in which all subscribers used the Gold SLA and were performing large HTTP downloads.

If we consider only the simulations which either contained some subscribers with a lower SLA (Figures 4f, 5f & 6f) or with some subscribers watching IP video streams with smaller file sizes (Figures 7a, 7b, & 7c), then the resulting values of K all fall within the very small range between 0.9 and 1.1. We believe that these cases more accurately represent real-world scenarios. The lower required K values probably result from the

fact that these subscriber mixes are more easily able to yield bandwidth to the Performance monitoring test (due to typical CMTS scheduling algorithms that fairly throttle traffic during congestion).

While this is not a conclusive proof that K can always be set to a value in that range to yield good QoE results for subscribers, it does give us a fair degree of confidence that K values near 1.0 may be adequate for many real-world MSO scenarios in the coming years. As traffic engineers tend to be cautious individuals, the authors would actually recommend adding a little buffer to the range of acceptable K values found in the simulations above. In particular, it might be wise for cautious traffic engineers to set the minimum required value of K to values in the 1.0 to 1.2 range. (Note: The authors have been setting the minimum required K value equal to 1.0 for their recent traffic engineering analyses).

To further check this hypothesis, we created another set of simulations that mix together various types of traffic that even more closely match the predicted (blended) traffic conditions defined in Table 1. The results of those simulation runs are shown in Figure 7a (for 2014), in Figure 7b (for 2018), and in Figure 7c (for 2021). It is promising to note that the results from those simulation runs also yield minimum required K values in the same range.

2014 Mix 200 2

150 Tmax_max

100 1 sec

Mbps Perf Goal

50 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 7a: 2014 Simulation Results with Blended Traffic

Parameter Value Comment Nsub 250 50 Active (22 Video, 28 Web) Video Bitrate 5 Mbps Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 1.6 MB Avg HTTP GET Interval 22 sec Bandwidth (for K=1) 325 Mbps Minimum K value 1.1 Web Resp < 1.5 sec, No ABR throttling Bandwidth (for K=Perf Goal) 345 Mbps

2018 Mix 1000 2

750 Tmax_max

500 1 sec

Mbps Perf Goal

250 Perf Mon Test Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 7b: 2018 Simulation Results with Blended Traffic

Parameter Value Comment Nsub 250 50 Active (30 Video, 20 Web) Video Bitrate 20 Mbps Tavg 2.5 Mbps Tmax 1 Gbps Avg HTTP File Size 4 MB Avg HTTP GET Interval 22 sec Bandwidth (for K=1) 1.6 Gbps Minimum K value 1.0 Web Resp < 1.5 sec, No ABR throttling Bandwidth (for K=Perf Goal) 1.6 Gbps

2021 Mix 3500 2 3000

2500 Tmax_max 2000 Perf Goal

1 sec

Mbps 1500 Rb = .75 1000 Perf Mon Test 500 0 0 Web Resp Time 0 0.5 1 1.5 2 K

Figure 7c: 2021 Simulation Results with Blended Traffic

Parameter Value Comment Nsub 250 50 Active (27 Video, 23 Web) Video Bitrate 75 Mbps Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 7.8 MB Avg HTTP GET Interval 22 sec Bandwidth (for K=1) 5.4 Gbps Minimum K value 1.0 Web Resp < 1.5 sec, No ABR throttling Bandwidth (for K=Perf Goal) 5.0 Gbps

At this point, it may be interesting to note how a change in the Benevolence Ratio (Rb) might impact the required minimum K value. If an MSO decides to reduce their Rb value to be below the ideal value of Rb=1.0, then that change will (by definition) lower the QoE level of the subscriber. For example, consider Figure 7c above. If it is decided that a subscriber executing a Performance monitoring test only needs to see 75% of their Advertised Billboard Bandwidth during the busy-hour period, then the required minimum K value drops to 0.6 (see the horizontal green line). This will obviously yield lower costs in providing the required bandwidth capacity (at the expense of subscriber QoE).

TWO OBSERVED WEAKNESSES OF THE NEW TRAFFIC ENGINEERING FORMULA

While the new traffic engineering formula described in (3) appears to be a useful formula and also appears to be (to some extent) validated by early simulation work, there are at least two potential issues that must be explored.

Issue #1: In all of the simulation runs performed within the previous section, there was always one (and only one) subscriber who was ever performing a Performance monitoring test at any moment in time. This was guaranteed by the fact that the simulation only enabled one subscriber to repeatedly execute the Performance monitoring test.

In the real world, it is possible (although most likely with a low probability) that more than one subscriber may launch a Performance monitoring test at the same time. Consider the simulation results shown in Figure 8, where we have purposely forced two subscribers to simultaneously launch Performance monitoring tests within our normal simulation run. It can be seen that a K value of ~1.0 is no longer adequate to provide both subscribers with good QoE. A higher K value of 1.9 is required in this case, and the cost of the bandwidth capacity would increase.

2014 (2*Perf monitor) 200 8

150 6

Tmax_max

100 4 sec

Mbps Perf Goal

Perf Mon Test 50 2 Web Resp Time

0 0 0 0.5 1 1.5 2 K

Figure 8: 2014 Simulation Results with Two Simultaneous Performance Monitor Tests

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec Bandwidth (for K=1) 325 Mbps Minimum K value 1.9 Web Resp < 1.5 sec Bandwidth (for K=Perf Goal) 505 Mbps

Issue #2: In all of the previous simulation runs, the number of subscribers per service group (Nsub) was always maintained at levels above 250 subscribers. However, many MSOs are considering performing node-splits down to much smaller service group sizes. Figure 9 shows the simulation results for a small Network Model from the 2021 time- frame:

2021 Nsub=50 3500 7 3000 6 2500 5 2000 4 Tmax_max sec

Mbps 1500 3 Perf Goal 1000 2 Perf Mon Test

500 1 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 9: 2021 Simulation Results with Nsub = 50

Parameter Value Comment Nsub 50 10 Active Tavg 8.4 Mbps Tmax 3.3 Gbps Avg HTTP File Size 105 MB Avg HTTP GET Interval 20 sec Bandwidth (for K=1) 3.7 Gbps Minimum K value 0.8 Web Resp < 1.5 sec Bandwidth (for K=Perf Goal) 3.0 Gbps

If we compare Figure 9 with Figures 6a & 6b, we see that the minimum value of K tends to decrease from 1.2 to 0.8 with smaller service group sizes. This is likely due to the fact that, under congestion, the Performance monitoring test tends to receive a share of the available bandwidth that is ~1/Nsub of the total bandwidth (which results in a decreasing percentage of the total bandwidth as Nsub gets larger).

Unfortunately, as the size of the service group gets smaller the overall performance of the network becomes less predictable using statistical methods. For example, if a service group contains only 4 active subscribers, there is a very real chance that all four will be simultaneously attempting to utilize their maximum SLA bandwidth.

Thus, traffic engineers must use caution when working with small service group sizes. SELECTING APPROPRIATE TMAX & BHPG VALUES GIVEN AN ADVERTISED BILLBOARD BANDWIDTH

How does one select appropriate Tmax & BHPG values given an Advertised Billboard Bandwidth value?

Consider an example where the Advertised Billboard Bandwidth is given by 200 Mbps, Nsub = 250, and Tavg = 500 kbps. There are many options from which a traffic engineer can choose.

Option #1: The traffic engineer could let Rc=1.0 and set Tmax_max=200 Mbps and could let Rb=1.0 and also set BHPG=200 Mbps. The results of these settings (Figure 10) show that the minimum required K value would need to be increased to a value of ~2.0, which would increase system costs. The required bandwidth capacity is given by 525 Mbps,

which is quite large. Thus, these are not advisable settings (with Rc=1.0) and should be avoided.

2014 Rc=1.0 300 3

250

200 2 Tmax_max

150 sec

Mbps Perf Goal 100 1 Perf Mon Test 50 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 10: 2014 Simulation Results with Rc=1.0 and Rb=1.0

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 200 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec Bandwidth (for K=1) 325 Mbps Minimum K value 2.0 Web Resp < 1.5 sec Bandwidth (for K=Perf Goal) 525 Mbps

Option #2: The traffic engineer could let Rc=1.1 and set Tmax_max=220 Mbps and could let Rb=1.0 and set BHPG=200 Mbps. The results of these settings (Figure 11) show that the minimum required K value falls back into the “normal” range of 1.2. The required bandwidth capacity is given by 414 Mbps, which is reasonable.

2014 Rc=1.1 300 3

250

200 2 Tmax_max

150 sec

Mbps Perf Goal 100 1 Perf Mon Test 50 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 11: 2014 Simulation Results with Rc=1.1 and Rb=1.0

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 220 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec Bandwidth (for K=1) 345 Mbps Minimum K value 1.2 Web Resp < 1.5 sec Bandwidth (for K=Perf Goal) 414 Mbps

Option #3: The traffic engineer could let Rc=1.5 and set Tmax_max=300 Mbps and could let Rb=1.0 and set BHPG=200 Mbps. These are also reasonable settings. The use of larger Rc values (Figure 12) lower the minimum required K value, which becomes 0.7. The required bandwidth capacity is given by 298 Mbps, which is significantly less than the value found in Options #1 or #2. Thus, these are also reasonable settings. However, the use of the larger Rc value leads to the need for a larger Tmax value, and that may place undesirable requirements on the number of Downstream tuners that the modem must support. As a result, Option #2 may be more preferred.

2014 Rc=1.5 300 3

250

200 2 Tmax_max

150 sec

Mbps Perf Goal 100 1 Perf Mon Test 50 Web Resp Time 0 0 0 0.5 1 1.5 2 K

Figure 12: 2014 Simulation Results with Rc=1.5 and Rb=1.0

Parameter Value Comment Nsub 250 50 Active Tavg 500 Kbps Tmax 300 Mbps Avg HTTP File Size 6.25 MB Avg HTTP GET Interval 20 sec Bandwidth (for K=1) 425 Mbps Minimum K value 0.7 Web Resp < 1.5 sec Bandwidth (for K=Perf Goal) 298 Mbps

TYPICAL DESIGN SCENARIO

In a typical scenario focused on the DOCSIS HSD Downstream bandwidth, an MSO traffic engineer may be given the Advertised Billboard Bandwidths (for each SLA) from the marketing team, and he may be asked to architect the DOCSIS HSD network to support those Advertised Billboard Bandwidths.

We will work through a typical design example (with some new numbers) to illustrate a process that might be followed. Assume that marketing department has defined two SLAs: • Silver SLA o Advertised Billboard BW = 150 Mbps • Gold SLA o Advertised Billboard BW = 300 Mbps

1) Determine the sizes of the service groups (i.e.- the Nsub value). Assume all service groups have Nsub = ~150 subscribers. 2) Measure the Tavg value for the subscribers on the network using formula (2). Let us assume that Tavg = 1 Mbps. 3) Select a Tmax value for each SLA. We have found that Rc values of 1.1 or 1.2 are acceptable to use. From Figures 11, 4a, 5a, & 6a, we find that if Rc=1.1, then K=1.2. If Rc=1.2, then K=~1.1. If the MSO selects Rc=1.1, then K=1.2 and the Tmax values for each SLA are given by: • Silver SLA o Tmax = Rc*ABB = 1.1*(150 Mbps) = 165 Mbps • Gold SLA o Tmax = Rc*ABB = 1.1*(300 Mbps) = 330 Mbps 4) Select a BHPG value for each SLA. We will assume that this MSO desires to have an Rb value of 1.0, so the BHPG values for each SLA are given by: • Silver SLA o BHPG = Rb*ABB = 1.0*(150 Mbps) = 150 Mbps • Gold SLA o BHPG = Rb*ABB = 1.0*(300 Mbps) = 300 Mbps 5) Determine the required bandwidth capacity for the service group. The traffic engineer focuses on the Gold SLA for this. From that SLA, he can instantly define the Tmax_max value to be 330 Mbps. Using formula (3), he can then calculate the minimum required bandwidth capacity to be: C = (Nsub*Tavg) + (K*Tmax_max) = (150 * 1 Mbps) + (1.2* 330 Mbps) = 546 Mbps 6) Calculate the number of required DOCSIS 3.0 Downstream channels (assuming ~36 Mbps of available bandwidth capacity per channel), which is given by: ceiling(546 Mbps/36 Mbps) = 16 DOCSIS Downstream channels.

The relationship between Rc and K values in step (3) leads to an interesting observation. For typical values of Rc such as 1.1 or 1.2, the typical values of K might be ~1.2 or ~1.1, respectively. Thus, the product of K*Rc seems to be nearly constant and is given by ~1.32. That observation can permit us to propose a simplification to the form of formula (3) used in step (5).

C = (Nsub*Tavg) + (K*Tmax_max) (5)

C = (Nsub*Tavg) + (K*Rc*ABB_max) (6)

C = (Nsub*Tavg) + (1.32*ABB_max) (7)

Since ABB_max (and not Tmax_max) is typically specified by the MSO’s marketing team, this simpler formula (7) may be applicable (and useful) whenever Rc is a “typical” value of 1.1 or 1.2 and when Rb = 1.0.

CONCLUSIONS

For many years, DOCSIS traffic engineers have used various rule-of-thumb criteria to determine the bandwidth capacity requirements for their High-Speed Data (HSD) service groups. Interestingly, one very important characteristic of the user experience was missing from all of these metrics- that characteristic is the Quality of Experience (QoE) level for subscribers as they consume different service types.

A new traffic engineering formula is introduced and validated within this paper. It incorporates all of these important criteria (Number of Subscribers, Average Bandwidth consumption levels, Maximum Bandwidth Service Level Agreements, and QoE).

RELATED READINGS • PAPER: Predictions on the Evolution of Access Networks to the Year 2013 and Beyond • PAPER: A Technical Migration Plan for the Evolution to DOCSIS 3.1 • PAPER: Advanced Quality of Experience Monitoring Techniques for a New Generation of Traffic Types Carried by DOCSIS

MEET ONE OF OUR EXPERTS: Tom Cloonan

Tom Cloonan, has been the Chief Technology Officer or Chief Strategy Officer of ARRIS for the past twelve years, responsible for directing the architectural work and future product planning within the company. Prior to that, Tom was the CTO and co-founder of the start-up company Cadant, where he led the architectural work for the development of the C4 CMTS. In a previous life, Tom also worked at Lucent Bell Laboratories on various telecommunications and routing platforms. Tom has a BSEE degree, and MSEE degree, and a Ph.D. in Physics. His research has resulted in over 30 patents, over 50 published papers, and several book chapters.

REFERENCES

(1) [Ca2014] Goran Candrlic, “How big is a page size going to be in 2014?” Website Speed, http://www.globaldots.com/big-page-size-going-2014

(2) [Cl2013] Tom Cloonan, Jim Allen, Tony Cotter, and Ben Widrevitz, “Advanced Quality of Experience Monitoring Techniques for a New Generation of Traffic Types Carried by DOCSIS,” Proceedings, The NCTA Cable Show Spring Technical Forum (June, 2013).

(3) [Cl2014] Tom Cloonan, Mike Emmendorfer, John Ulm, Ayham Al-Banna, and Santhana Chari, “Predictions on the Evolution of Access Networks to the Year 2030 and Beyond,” Proceedings, The NCTA Cable Show Spring Technical Forum (April, 2014).

(4) [Em2014] Mike Emmendorfer and Tom Cloonan, “Nielson’s Law vs. Nielson TV Viewership for Network Capacity Planning,” Proceedings, The NCTA Cable Show Spring Technical Forum (April, 2014).

(5) [Pr2012] Pro Webmasters, “What’s an acceptable avg. page load time?”http://webmasters.stackexchange.com/questions/28220/whats-an- acceptable-avg-page-load-time

(6) [Sa2104] Sandvine Global Internet Phenomenon Report (1H 2014),https://www.sandvine.com/downloads/general/global-internet- phenomena/2014/1h-2014-global-internet-phenomena-report.pdf

ABBREVIATIONS & ACRONYMS

ABB Advertised Billboard Bandwidth ABR Adaptive Bit-Rate B Number of bytes counted in window of time BHPG Busy-Hour Performance Goal C Bandwidth Capacity required by a service group CAGR Compound Annual Growth Rate CM Cable Modem CMTS Cable Modem Termination System FTTH Fiber To The Home HSD High-Speed Data HTML Hyper-Test Mark-up Language HTTP Hyper-Text Transport Protocol K QoE constant for Tmax_max Ka QoE constant for ABB_max MSO Multiple System Operator Nsub Number of subscribers in a service Group QoE Quality of Experience Rc Cushion Ratio Rb Benevolence Ratio SLA Service Level Agreement Tavg Average Per-Subscriber Traffic Level TCP Transmission Control Protocol Tmax Maximum Sustained Traffic Rate W Window of time in seconds Y Number of years

©ARRIS Enterprises, Inc. 2014 All rights reserved. No part of this publication may be reproduced in any form or by any means or used to make any derivative work (such as translation, transformation, or adaptation) without written permission from ARRIS Enterprises, Inc. (“ARRIS”). ARRIS reserves the right to revise this publication and to make changes in content from time to time without obligation on the part of ARRIS to provide notification of such revision or change.