Internet Research: Comments on Formulating the Problem

Gathered by Sally Floyd, with contributions from Deborah Estrin, Greg Minshall, Vern Paxson, Lixia Zhang, and others. January 21, 1998

1 Introduction Development and deployment in the infrastructure is of necessity incremental. This note contains a discussion about formulating the research Explicit examined assumptions are better than implicit un- problem for Internet research. examined ones. The goal of this note is to further the discussion of implicit Changes in the Internet can be unanticipated and uncon- and explicit assumptions in network research. In particular, trolled. this note tries to articulate one such set of assumptions for In- The Internet architecture and scale make requirements for ternet research. Each of these assumptions is shared by some global consistency problematic. subset of the network research community, though perhaps Some research problems have their own natural time none of these assumptions are shared universally. The goal scales. of this paper is not to argue the validity of the assumptions, but to articulate them, to invite discussion of con¯icting or shared sets of assumptions, and to consider the implications 3 Discussion of the assumptions of these assumptions in formulating problems in Internet research. This process would aim for both a greater convergence Robustness is more important than ef®ciency. of underlying assumptions, and a more explicit and examined Robustness has been one of the great strengths of the Inter- discussion of those assumptions. net, integral to its design from the very beginning [Clark88]. In some sense, this note is in the tradition of Shenker et al.©s One of our overriding assumptions is that it is critical not to paper on ºPricing in Computer Networks: Reshaping the Re- subordinate robustness to the goal of more closely approxi- search Agendaº [Shenker96], which is a discussion about for- mating optimal ef®ciency. [This is discussed in more detail in mulating the research problem for the speci®c area of network the section on TCP.] pricing. This note does not address questions about how to de- The Internet is characterized by heterogeneity on many sign good protocols or write good papers; many of these ques- levels. tions are addressed elsewhere [ADVICE, Lampson84]. This note simply addresses the question of how to formulate the The Internet is characterized by heterogeneity on many research problem. levels: router scheduling algorithms and queue management The ®rst part of the paper brie¯y describes some underlying mechanisms, routing protocols, levels of multiplexing, proto- assumptions. The second part discusses implications of these col versions and implementations, etc. There is heterogene- assumptions in formulating research problems in various areas ity of underlying link layers (e.g., point-to-point, multiaccess of Internet research. links, wireless, ATM, FDDI, etc.). Because the Internet is composed of autonomous organizations and internet service providers, each with their own separate policy concerns, there is a heterogeneity of administrative domains and pricing struc- 2 Outlining the underlying assump- tures. There is a heterogeneity in the traf®c mix and in the tions levels of congestion at different times and places. As is discussed later in the paper, this heterogeneity has to be taken A summary of the assumptions: into account in different ways with different areas of research. Robustness is more important than ef®ciency. Complex systems are not best designed from scratch and The Internet is characterized by heterogeneity on many on paper. levels. Our observation is that large complex systems are not best Complex systems are not best designed from scratch and designed on paper as a complete system and then imple- on paper. mented, but are best designed incrementally, with new insights

1 and development growing out of practical experience with ex- this period of change is not a temporary phenomena, but will isting functionally. Our assumption is that this is true for sub- continue to characterize the future Internet. systems such as the Mbone, reliable multicast protocols, or Some research problems have their own natural time integrated services as well as for larger systems such as the scales. global Internet. Some fundamental research problems are remote from con- Development and deployment in the infrastructure is of cerns of actual implementations, but many other research necessity incremental. problems have natural time scales related to the problem un- Unlike the development and deployment of applications, der consideration. Theses time scales might be the time scales which can sometimes spread like wild®re, the development on which decisions will be made either in the marketplace or and deployment of the infrastructure is of necessity incremen- in the IETF, the time scales after which conditions will have tal. This is due to both the scale of the infrastructure and to the changed enough to require reformulating the problem, etc. autonomy of individual components. This restriction to incre- Again, this is discussed further in the section on TCP. mental deployment is problematic for proposed mechanisms The Internet architecture and scale make requirements for (e.g., some scheduling algorithms, routing protocols, and QoS global consistency problematic. mechanisms) that will only be of use when they are ubiqui- A far-¯ung and complex system that requires consistency tously deployed throughout the Internet. or synchronization among all the components is problematic. Although there have been ©¯ag days© in the Internet in the This is particularly true given the scale and heterogeneity of past1, it seems unlikely that this can happen again. The 1983 administrative control in the current Internet. ¯ag day in particular was possible because at the time there was some centralized control of the ARPAnet by a single agency (i.e., DARPA). 4 Some implications Explicit examined assumptions are better than implicit un- These assumptions have several implications about formulat- examined ones. ing research problems. One implication is that starting from We all make implicit assumptions in our research, and such scratch and designing a new self-contained system is not al- implicit assumptions are often wrong. For example, because ways the most effective approach. In many cases it is more my intuition is partly shaped by my experiences with my sim- productive to take a piece of a larger problem, and to try to un- ulator, which until recently had a rather simple view of the derstand it and its relationship to the larger problem. These world, I tend to implicitly assume that TCP connections fol- might be exactly the problems that need attention. And in low ®xed symmetric paths on point-to-point links, with data many cases progress can be made in understanding a problem ¯owing only from the Ásender© to the Áreceiver©, and not vice- even if a complete answer is likely to be elusive and to require versa. The Áreal world© is of course much more complicated, the combined effort of the larger research community. with dynamic and asymmetric routing, multiaccess wireless One way to design for incremental evolution is to start with links, and two-way data traf®c within a single TCP connec- IP and the current Internet. IP lends itself to the addition tion. Implicit assumptions about things such as the nature of of rich new functionality and to the process of evolving to traf®c, the level of congestion, and the requirements of appli- meet new challenges. New transport protocols and applica- cations are in general highly problematic, and are best made tions are added, routers implement new queueing and schedul- as explicitly as possible. ing mechanisms, and new underlying technologies such as Changes in the Internet can be unanticipated and uncon- ATM or wireless are used. Other incremental changes either trolled. planned or already in progress include the incremental deploy- The history of the Internet has been one of unanticipated ment of IPv6, IP multicast, and integrated services. While change. This includes changes in applications (e.g., the web, it often looks attractive to design a new network architecture realtime traf®c) and in the infrastructure (e.g., past changes in from scratch to accomplish a speci®c purpose (e.g., OSI, X.25, routing, current changes to incorporate multicast, integrated ATM, active networks), our inclinations and experience are services, and IPv6). This can be particularly frustrating when with deploying new functionality in the IP network. the Internet changes, while you are in the middle of a research project, in ways that threaten to invalidate your work. As an example, work on integrated services can be dif®cult when on- 5 TCP going changes in the traf®c mix (e.g., the emergence of the web, realtime applications with congestion control and adap- In the next two sections we consider how these assumptions tive playback times, etc.) and pricing structures make it dif- about the nature of the Internet guide our own formulation of ®cult to ®nd ®rm ground on which to stand. We believe that speci®c research problems. This section examines some of the ways that our assumptions have shaped our framing of re- 1There was a Á¯ag day© in the ARPAnet for the transition of the ARPANET search questions on TCP. host protocol from NCP to TCP/IP in January 1, 1983 [Leiner97]. Implicit assumptions:

2 Early research on TCP dynamics (including our own) and more fruitful approach, in my judgement, would be to start focused on behavior with a small number of competing, instead with IP, the IP architecture, and the real world of the unlimited-demand TCP connections in one-way networks Internet. (Or, to rephrase, the only axiom is to start with IP and (e.g., the data traf®c going in one direction, and the acks go- the real world, characterized as it is by heterogeneity, change, ing in the other), in a simple topology with static routing over and the restrictions of incremental deployment.) point-to-point links. This is a ®ne starting point for under- Heterogeneity and implicit assumptions: standing traf®c dynamics in the most simple scenarios, and One implication of our assumptions is that we can© t make does not necessarily re¯ect assumptions that this is a faith- strong assumptions about the likely traf®c mix for ªrealtimeº ful model of real-world environments. However, it has also traf®c. (We use the phrase ªrealtimeº traf®c loosely to refer been necessary to acknowledge more complex environments, to traf®c that needs per-¯ow quality of service guarantees.) including short Web connections, two-way traf®c, rich topolo- The ªrealtimeº traf®c will not necessarily be mostly audio and gies, and dynamic routing. Understanding traf®c dynamics in video, and the fraction of the traf®c that is video will not nec- large complex topologies with a rich traf®c mix remains a dif- essarily be mostly MPEG video. Some of the traf®c will be ®cult and unsolved problem. unicast and some will be multicast. Some of the traf®c will be Robustness and heterogeneity: using end-to-end congestion control and some won© t. Some of A second area of TCP research is the addition of new func- the traf®c will be extremely amenable to aggregation (because tionality or modi®ed congestion control algorithms to TCP. of a Poisson packet arrival process or a small peak-to-mean A central requirement is to maintain TCP©s robustness in the ratio for ON/OFF traf®c) and some will not. Some of the real- presence of packet losses and a wide range of offered load, time applications might want jitter control or strict delay guar- link speeds, packet sizes, and congestion levels. Thus, while antees from the network, but most will not. Any assumptions one goal is to evolve TCP to more effectively use the avail- about the traf®c should be as explicit as possible. And any able bandwidth in a range of scenarios (e.g., short web trans- such assumptions are likely to restrict the resulting design in fers, large high-bandwidth ®le transfers, TCP over satellite and some fashion. other wireless links, TCP over ATM, TCP over integrated ser- One process of de®ning the research problems, based on vices, etc.) this has to be done without sacri®cing effectiveness these assumptions, has proceeded as follows: because the in other environments, and without affecting the robustness of traf®c cannot easily be characterized, measurement-based ad- the congestion control algorithms for the network as a whole. missions control has strong advantages. One approach that This robustness is far more important than full ef®ciency in makes minimal assumptions about traf®c characteristics is to using all of the available bandwidth. assume that ¯ows applying for admissions will characterize Unanticipated change and time scales of research problems: themselves simply by a policed peak rate. Because TCP is a changing target, research problems on This realtime traf®c is also not likely to become the domi- TCP also have their own natural time scales. As an exam- nant traf®c in the Internet. The current Internet is dominated ple, investigations of the unnecessary retransmit timeouts of by best-effort TCP traf®c that does not require strict bounds Reno TCP (1990) were of signi®cant interest a few years ago, on per-packet delay or on packet drop rates. We would conjec- but should become less relevant in a few years when most ture, based in part on the heterogeneity of Internet traf®c, that TCP implementations have begun to incorporate the Selective there will continue to be a large fraction of the overall Internet Acknowledgement option (SACK). The problem of unneces- traf®c that does not require either per-¯ow QoS guarantees or sary retransmit timeouts will be even less relevant when most aggregated differential services. In this case, the admissions routers have implemented intelligent queue management (e.g., control procedure does not have to take on the dif®cult job RED) that avoids many of the bursts of packet drops charac- of trying to use 90% of the link bandwidth for admitted traf- teristic of Drop-Tail gateways. Thus, tailoring the design of ®c while avoiding packet drops or large delays. When only a future protocol (e.g., some form of ATM rate control) to ad- a moderate fraction of the link bandwidth is used by realtime dress the problems of current Reno TCP implementations with traf®c, the admissions control procedure has much more room unnecessary retransmit timeouts is not helpful; such problems to maneuver. of TCP are somewhat transient concerns, and the ®rst-order Thus, in the case of integrated services, taking into account ®x, already in progress, is to ®x the problem in TCP itself. the assumptions of heterogeneity and the requirements for robustness does not necessarily make the research problem less tractable; in some respects it makes the research problem more 6 Integrated services tractable, by avoiding altogether the goal of high bandwidth utilization by a large body of carefully-characterized realtime In this section we consider as an example the particular re- traf®c. search area of providing qualities of service in the Internet. Incremental design: One possibility would be to start with a set of axioms, and The process of incremental design, taking into account to derive an architecture from that set of axioms. An alternate lessons learned from actual deployment, can be illustrated

3 by the ongoing design process for the Mbone tools. Au- of evolution of the global internet from the present reality dio/video/whiteboard tools were ®rst deployed in limited en- to the envisioned future can be viewed not only as an unfor- vironments, even though the tools are not fully-formed, with tunate constraint preventing certain discontinuous jumps, but some major pieces of the functionality (e.g., the congestion also as a source of power, enabling an envisioned future inter- control mechanisms needed for more widespread deployment net to be successful because it builds upon the present reality for these tools to peacefully co-exist as best-effort traf®c) still [Dawkins96, Kauffman95]. in the research stages. Gaining experience with the limited In our own research on adaptive web caching we have taken deployment of the Mbone tools has been one preparation for a two-pronged approach, thinking both about the possible state developing the tools for the next stage. of a global data dissemination architecture of the future, and Similarly, the approach of the Internet research community also about incremental changes that can be make in the current is that an essential next step is to gain experience with a lim- web-caching architecture to advance in that direction. ited deployment of the current framework for integrated services (i.e., RSVP and the guaranteed service and controlled load service templates). This operational experience should 8 Conclusions be critical in evaluating the current framework, and in keeping track of the evolving needs and characteristics of applications. There are plenty of hard problems concerned with understanding and engineering a large, decentralized, complex system such as the Internet. Working on the incremental deploy- 7 Issues with incremental deployment ment of new functionality in a changing global information intrastructure does not imply that the work must be mundane or lacking in fundamental intellectual challenges. There is a How should we let the realities of incremental deployment im- wealth of fundamental intellectual challenges involved in un- pact our research? derstanding and managing large complex systems such as the The deployment of multicast routing in the Mbone and of Internet. IPv6 [Hinden96] in the 6bone illustrate strategies of incremen- One approach to choosing a research topic is to consider tal deployment of new functionality in the infrastructure. Inte- potential problems that have not yet manifested in the Inter- grated services and RSVP are examples of projects facing the net. For example, in areas such as routing and traf®c dynamics challenges of incremental deployment. This non-ubiquity and there could be emergent behaviors of synchronization or self- incremental deployment of ªadvanced featuresº such as mul- organization that only manifest with a certain size of the Inter- ticast, active scheduling and queue management, IPv6, and so net or a certain level of multiplexing. As the Internet grows in on, poses a real chicken-and-egg problem, since mainstream scale, the development is not likely to be a simple extrapola- application developers generally steer clear of advanced fea- tion from the current dynamics. tures in the infrastructure until some critical mass has been Although there can be considerable dif®culty and frustration achieved. involved with working in a constantly-changing environment One problem with proposals that depend on the ubiquitous such as the Internet, there is also a great opportunity for cre- deployment of a new mechanism is that there is no viable path ativity and power that comes from working in an environment from getting from here to there; that is, no path for transition- of other newly-emerging applications and network services. ing from the current Internet to the world of ubiquitous deployment of the new proposed mechanisms. Fortunately, many new functionalities don© t need to be implemented everywhere 9 Acknowledgements to give value. One path of incremental deployment would be for such functionality to ®rst be deployed in intranets or other The note has resulted from conversations with a number of controlled environments, resulting in pools of feature-rich nets people, including Deborah Estrin and Greg Minshall (who has in the Internet. This is almost the opposite of the Mbone diffu- in fact contributed some of the text). This note has been heav- sion model, where the Mbone is layered on top of the Internet ily in¯uenced by the ideas and architectural approaches of by Mbone-capable routers connected by tunnels. Dave Clark and Van Jacobson. Thanks also to Jon Crowcroft, Of course, an acknowledgement of the reality of incremen- Craig Partridge, Vern Paxson, Scott Shenker, Lixia Zhang, and tal deployment in the intrastructure should not cut off research other members and participants of the End-to-end Research on proposals that would require ¯ag days or entirely new net- Group for feedback. The note began in response to a proposal works for deployment. A good discussion of concepts, simu- from K. K. Ramakrishnan. We emphasize that none of these lations, and experiments can be useful even if it does not lead contributors can be assumed to agree with all of the contents to eventual implementation. of the note. The Internet intrastructure also does not necessarily evolve This note is not a ®nished product, but is one stage of a by smooth increments, but also by ®ts and starts, or by punc- conversation that hopefully will result in further development tuated equilibrium. The necessity for having a viable path by other contributors.

4 References

[ADVICE] Leone, M., Web page on ºCollected Advice on Research and Writingº, http:// www.cs.cmu.edu/afs/cs.cmu.edu/user/mleone/web/how- to.html. [Clark88] Clark, D.D., The Design Philosophy of the DARPA Internet Protocols, SIGCOMM ©88, pp. 106-114. [Dawkins96] Richard Dawkins, Climbing Mount Improbable, W.W. Norton & Company, 1996. [Hinden96] Robert Hinden, IP Next Generation Overview, CACM, June 1996, V.39 N.6. [Kauffman95] Kauffman, S., At Home in the Universe: The Search for Laws of Self-Organization and Complexity. Oxford Univ Press, October 1995. [Leiner97] Barry M. Leiner, Vinton G. Cerf, David D. Clark, Robert E. Kahn, Leonard Kleinrock, Daniel C. Lynch, Jon Postel, Larry G. Roberts, Stephen Wolff, A Brief History of the Internet, http://www.isoc.org/internet- history/.

[Lampson84] Lampson, B., Hints for Computer System De- sign, IEEE Software, January 1984, pp. 11-28. [Shenker96] S. Shenker, D. Clark, D. Estrin, S. Herzog, Pric- ing in Computer Networks: Reshaping the Research Agenda, SIGCOMM CCR, V. 26 N. 2, April 1996.