The COOK Report on Internet Light, IP, and Gig E - Annual Report February 2001 They are getting some seed funding from cant role in these developments. We now could go into the dark fiber market and sell the Alberta government and that is to be used have many carriers building national infra- dark fiber and a fraction of what everybody to enable every community have low-cost structure that is independent of the old else was selling it for in order to gain mar- bandwidth such that any competitor, ISP, PSTN. ket share. So the question that we keep ask- business, or school can buy fiber strands. ing ourselves is whether the economic ben- COOK Report: Do you have any feeling at efits that we’re seeing now merely because COOK Report: So they are setting a kind of all that the cost effectiveness may move even we are leading the market place or are they green field example of running the fiber train beyond the range of 1000 to one? maybe there is a much more fundamental through for the first time and showing ev- driver? eryone how to jump on board and play the St Arnaud: It’s hard to speculate on that. game. What about dark fiber networks in the Nevertheless we are, right now, really down We are starting to understand some network provinces in general? to the true cost. We know how much it costs fundamentals that show that carriers cannot to deploy dark fiber as well as the incremen- gain huge economies of scale, and that, as a St Arnaud: A number of other provinces tal costs and the equipment. For the first time result, their costs would still remain higher. have are looking into building condominium in history we have a telecommunications dark fiber networks like Alberta. Our fed- system that is based on real input costs. The COOK Report: And the reason for this? eral government has just created a National biggest cost component is the installation and Broadband Task Force to look at how the deployment of the fiber. In my opinion, for “Scaling Issues on government can help accelerate broadband the moment, we’ve pretty well on a plateau. to every community, residence and business However, we may get some increasing the Internet Networks” and in Canada. On the research and education economy of scale as we put in more fiber side Newfoundland has a 1700 km 4 chan- strands. OBGP nel GbE DWDM network and we are just finalizing 250 km 2 GbE network to Prince COOK Report: On the other hand some is- St Arnaud: It is what is called the “n” Edward Island. The British Columbia uni- sues are beginning to emerge that might put squared phenomena. Namely their costs will versities will announce their optical network an end to the increases in cost efficiency. If go up by the square of the number of users plans very soon which will be much more some of the issues with increasing traffic and connected and the squaring of anything pro- along the CA*net 4 concept than CA*net 3. connections about which you write in your duces really dramatic results. Today’s net- New Brunswick and Ontario are getting new “Scaling Issues on Internet Networks” works can be visualized as snowflakes. If close. Nova Scotia will also announce soon. paper turn out to be verifiable and accurate, we’re talking about a telephone network for we may find that there is a point at which example, traditional carriers get significant new investment in costly equipment may economies of scale because “n” is stable and A Cost Effective National have to be made just to maintain the status growing very slowly. In the snowflake model Infrastructure quo. Correct? you get then economies of scale by aggre- gating traffic at network nodes. Today, the Independent of the PSTN St Arnaud: That’s right. The other issue is bigger the telephone network, the more sig- that because of what someone called the “n” nificant the economies of scale and the more COOK Report: What seems to be highly sig- squared phenomena is the inverse economies profitable such a network can be. nificant about the situation in Canada and of scale. In other words that operating a very indeed unique in the world, except perhaps big network in proportion to the number of With the Internet “n” is always growing be- for Sweden, is that you have the school and users may be actually more costly than op- cause it is related to the number of comput- municipal dark fiber build outs merging into erating a smaller network. ers and networked devices. provincial wide public sector networks and finally into the CANARIE National back- COOK Report: A big network in the sense COOK Report: But in talking in terms of bone. You a have just about completed a of a global Internet? Or a big network in the the Internet are you saying then that a way national public-sector telecommunications sense of a large autonomous system? to escape what may become the problems infrastructure having for the first time abso- of “n” squared is rather than running a single lutely nothing to do with the public switched St Arnaud: All of our work on dark fiber huge global network is instead to run an telephone network. While the dark fiber and and OBGP was initially intuitive in that we Internet composed of many topologically gigabit Ethernet that is run over that fiber is could do some things considerably less ex- limited smaller peripheral systems, like cur- the major component, in order to make all pensively than the carriers with their man- rent Internet autonomous systems? And to this work independently of the public aged network services. The question in our connect all of these by means of OBGP ac- switched network and of the carriers, it mind then became one of whether or not cording to the patterns that the users in each seems that you need the optical border gate these savings were just an artefact of still of them desire? way protocol (OBGP). living in a world of monopoly inefficiencies or some other fundamental factors. Did these St Arnaud: That’s correct. We are discov- We know that with OBGP your end users savings materialize only because we were ering that, in the Internet at large, size, does can manage their respective parts of the new leading the marketplace? We wanted to not bring those economies of scale. In fact public sector network. Consequently, for the know if we face a situation where in time, that there is a certain size beyond which the first time you may have been some means as carriers themselves start to move in this economies of scale may be actually inverted. of demonstrating the enormous cost effec- direction of dark fiber networks that with tiveness of this new technology which when their large economies of scale, all our cost COOK Report: Are you speculating that one you and I first talked about this about three savings would start to disappear? way to get out from under this conundrum years ago you pegged at 100 to one and may be to have a series of customer owned which now may be 1000 to one. Do you Could carriers with their very large finan- networks that form the edges of the Internet agree? cial muscle reach a point where they could and, honeycomb like, are all interconnected achieve greater efficiencies and negate the with each other using OBGP ? St Arnaud: Yes. But I must point out that usefulness of customer owned dark fiber and our carrier partners have played a signifi- wavelengths? Hypothetically a huge carrier St Arnaud: That’s right. What we think may 61 COOK Network Consultants, 431 Greenway Ave, Ewing, NJ 08618 USA scale far more efficiently is customer owned routers and sets up wave lengths between COOK Report: And some of the spectrum fiber with customer owned wavelengths and them. To do this in a satisfactorily cost-ef- of solutions at least are outlined in your pa- massive peering. Now by customer I mean fective way, we need a very simple switch per? either an ISP or a large organization. I don’t without a whole lot of legacy protocol ca- mean you or I. The reason why we still have pability built into it. For this one task we St. Arnaud: Correct . thousands of ISPs left in the world and why don’t need all the complexity and concomi- they may not be disappearing as rapidly as tant cost that most switches come with. COOK Report: Your awareness of these is- predicted is that at some point in their de- sues came primarily from where? The pa- velopment the economics of scaling actu- Scaling the Internet per by Andrew Oldyzko and issues raised ally begins to work in favour of the smaller by Mike O’Dell? ones. COOK Report: Let’s look at your new pa- St Arnaud: Yes. And what’s more all of this COOK Report: I understand that you are per on Internet scaling issues. If the hypoth- has happened within the last 60 days or so. working presently on two flavors of OBGP. esis on the problems in growth of connec- From our point of view the paper has been Would you please explain what they are. tions as well as bandwidth are verified, what driven by two things. First by issues raised is learned will have serious implications for with Mike O’Dell in the fact that Uunet’s St Arnaud: Currently we are going in two the topology of and way in which the entire network capacity is growing faster than its strategic directions with OBGP.