Global Traffic Exchange Among Internet Service Providers (Isps)

Global Traffic Exchange among Internet Service Providers (ISPs)

Berlin, June 7, 2001

J. Scott Marcus, Chief Technology Officer (CTO)

www.genuity.com Economics of Internet Interconnection • Background – Internet interconnection in North America – International Internet interconnection • Economic modeling of Internet backbone peering – The off-net pricing principle – Implications for international interconnection • Scaling challenges to the peering system • Cross-provider differentiated services, measurements and SLAs

www.genuity.com Internet Interconnection in North America • Historical roots • Peering and transit • Shared and direct peering • Shortest exit • Hierarchical structure

www.genuity.com www.genuity.com Privatization of the Internet (NSF 93-52) ISP ISP ISP Backbone ISP ISP ISP

Backbone ISP NAP

NAP New York NAP Chicago

Washington DC ISP San Francisco Backbone ISP NAP ISP Los Angeles

Atlanta

Dallas Backbone ISP

Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.

www.genuity.com Peering and Transit • Peering is usually a bilateral business and technical arrangement, where two providers agree to accept traffic from one another, and from one another’s customers (and thus from their customers’ customers). Peering does not include the obligation to carry traffic to third parties. • Transit is usually a bilateral business and technical arrangement, where one provider (the transit provider) agrees to carry traffic to third parties on behalf of another provider or an end user (the customer). In most cases, the transit provider carries traffic to and from its other customers, and to and from every destination on the Internet, as part of the transit arrangement. • Peering thus offers a provider access only to a single provider’s customers; transit, by contrast, usually provides access at a defined price to the entire Internet. • Historically, peering has often been done on a bill-and-keep basis, without cash payments, where both parties perceive roughly equal exchange of value; however, there is often an element of barter.

www.genuity.com Shared and Direct Peering • A few shared global traffic exchange points. • Smaller domestic shared traffic exchange points for regional concentration and exchange of traffic. • Direct traffic exchange carries most Internet backbone traffic. Even though shared traffic exchange points are losing market share, their traffic is likely to continue to grow in absolute terms. • Carrier hotels and fiber interconnects - an emerging trend that seeks to provide the best of both worlds. • Whether shared or direct, the prevailing pattern is shortest exit routing - the sending provider hands off traffic at the point most convenient to the sender.

www.genuity.com Shared and Direct Peering

Backbone ISP Router

Router Direct Interconnection

Router Backbone ISP Router Router

Direct Interconnection Router

Router Backbone ISP Router Shared Traffic Exchange Point

www.genuity.com Shortest Exit (“Hot Potato”) Routing

Backbone ISP

New York Shared Chicago Shared Washington DC Traffic ExchangeSan Francisco Traffic Exchange Point Point

Los Angeles

Atlanta

Dallas Web Site Backbone ISP

www.genuity.com Backbone and Secondary Peering

• Backbone peering interconnects providers that have no need for a transit relationship. It provides the only interconnection between those providers. • Secondary peering interconnects providers who would otherwise exchange traffic through some transit connection. • Secondary peering is an economic optimization, reducing traffic over the transit connection.

www.genuity.com Fundamental Economics of a Local or Regional ISP

Transit

Larger ISP or Connection Regional Backbone or Local ISP

Concentration to a larger ISP or backbone provider with global connectivity by means of a concentrated, high bandwidth Many remote locations connection connect to a regional or local ISP with individual, low bandwidth connections

www.genuity.com Secondary Peering

Transit Larger ISP or Connection Regional The secondary Backbone or Local ISP peering connection will tend to exist if the cost of the connection Transit to each ISP is less than the money each saves due Larger ISP or Connection Regional Backbone or Local ISP to reduced transit traffic.

www.genuity.com A Hierarchical View of the Internet

Backbone Backbone “Default Free” Zone Backbone Backbone

Backbone Backbone Backbone ISPs Backbone

ISPs

Customers Copyright (c) 1999 by Addison Wesley Longman, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher.

www.genuity.com A Hierarchical View of the Internet

Upstream IBP IBP Peering Connection

ISP ISP

Downstream ISP

cf. Lixin Gao

www.genuity.com Negotiations for North American Backbone Internet Interconnection • Typical US backbone interconnection guidelines – Bi-coastal US presence, with multiple potential points of interconnection – Significant transcontinental bandwidth – Consistent routes at all locations – Competent staff, professional 7 x 24 operation – Rough balance of ingress/egress traffic – Sufficient scale to justify transaction costs • Where criteria are not met, a backbone may: – decline to exchange traffic, OR – expect cash or non-cash compensation in return • Backbone providers negotiate traffic exchange terms and conditions on a case by case basis.

www.genuity.com Emerging Global Trends

• The traditional “hub and spokes” system • Traffic exchange trends in Europe • Global deployment • International diffusion of users and content

www.genuity.com Traditional “Hub and Spokes” System

Regional #2

Regional Regional #1 #3

NSP #1

ISP #1 ISP #3

NSP #2 NAP

NAP New York Asian NAP Chicago

Washington DC ISP #2 San Francisco ISP NSP #3 NAP ISP #4 Los Angeles

Atlanta

Dallas Non-U.S. vBNS ISP pays Non-U.S. European ISP pays ISPs

www.genuity.com Former “Hub and Spokes” in Europe

French ISP

U.S. Backbone

German ISP

www.genuity.com Internet Traffic Exchange in Europe Today

French ISP

U.S. Backbone Interconnection Point

German ISP

www.genuity.com Factors Driving European Evolution

• Decline in “street price” of circuits within Europe due to deregulation. • Declining cost of transoceanic capacity. • Increased number and density of customers and content (and caching). • Improved number and distribution of shared peering points. • Deregulation of European telecoms, and recognition of the need to minimize regulatory barriers to Internet growth. • New transit services terminated in Europe and elsewhere.

www.genuity.com Global Internet Backbone Deployment

Overseas Point of Presence (POP) European ISP pays

Regional #2

Regional Regional #1 #3

NSP #1

ISP #1 ISP #3

NSP #2 NAP

NAP New York Asian NAP Chicago

Washington DC ISP #2 San Francisco ISP NSP #3 NAP ISP #4 Los Angeles

Atlanta

Dallas Backbone vBNS pays Backbone pays

www.genuity.com International Diffusion

Number of Users Online Worldwide

250

200

150

100

50 Millions of Users 0 1995 1996 1997 1998 1999E

United States Rest of World

Source: IDC, Merrill-Lynch (Kende, FCC)

www.genuity.com Motivations for International Pressure for Cost Sharing (ICAIS) • Mistaken perception that U.S.-based backbones discriminate against overseas providers in our interconnection policies. • Dissatisfaction with allocation of transoceanic circuit costs, which often are fully carried by the non-U.S.-based provider.

www.genuity.com An Excerpt from Genuity’s Guidelines

1. Presence at three or more Shared Interconnection Points listed above (two of which must be MAE-East ATM and MAE-West ATM), for Domestic ISPs; presence at two or more Shared Interconnection Points listed above for International ISPs. 2. For domestic ISPs, United States coast-to-coast nationwide backbone of at least 155Mbps. 3. Consistent route announcements at all exchange locations. 4. Experienced, professional Network Operations Center staffed 24x7. 5. Loose Source Record Route (LSRR) capability at core border routers on network. 6. For domestic ISPs, roughly balanced traffic. 7. A minimum Internet traffic exchange of 1 Mbps with Autonomous System 1. 8. Willingness to enter into a formal Internet Interconnection Agreement. http://www.genuity.com/infrastructure/interconnection.htm

www.genuity.com Economic Modeling of Internet Backbone Peering • Analytical framework • The off-net pricing principle • Implications for international interconnection

www.genuity.com Analytical Framework

• Define:

co as cost of origination ct as cost of termination a as an access charge levied on the sender Network Network i j • Due to shortest exit, ct > co • Then cost for the originating

network is co + a cost for the terminating

network is ct - a

www.genuity.com Impact of Access Charges

• In a bill-and-keep system, access charges are zero. • With equal access charges and symmetric traffic, net access charges are still zero. • Providers will, however, view their marginal costs quite differently. (Laffont/Rey/Tirole)

www.genuity.com The Off-Net Pricing Principle

• Under a broad range of conditions, marginal price should equal marginal cost. • Providers will tend to charge the same for on-net traffic as for off-net traffic, i.e. there are no strong incentives for price discrimination.

www.genuity.com Implications for International Interconnection • Where a group of customers have distinguishable costs that are higher than those of other customers, the model predicts that marginal prices will be higher to exactly offset the higher costs. • In the absence of market power, there is in fact no way for the ISP to absorb the cost difference without reflecting it in the price. • This prediction is consistent with typical practices of US- based ISPs, both within the US and overseas.

www.genuity.com Scaling Issues

• AS Number growth • IPv4 address growth • Routing table growth

www.genuity.com Tony Bates’s Data

AS Growth

9000

8000

7000

6000

5000

4000 Active ASesActive 3000

2000

1000

0 Food

www.genuity.com Exponential Growth of Autonomous System (AS) numbers

80000

R2 = 0.9957

70000

60000

50000 ASNs to Oct2000 ASNs since Oct2000 Expon. (ASNs to Oct2000)

40000 AS Numbers (ASNs) Numbers AS

30000

20000

10000 Source: Scott Marcus, Genuity 10/01/1996 10/01/1997 10/01/1998 10/01/1999 10/01/2000 10/01/2001 10/01/2002 10/01/2003 10/01/2004 10/01/2005

0 www.genuity.com Exponential vs Quadratic (Bates Data) 25000

y = 6E-14e0.0011x R2 = 0.9957 20000

15000 ASNs to Oct2000 ASNs since Oct2000 Expon. (ASNs to Oct2000) Poly. (ASNs to Oct2000) 10000 Active ASes(Clean)

y = 0.002x2 - 138.37x + 2E+06 R2 = 0.9962

5000

0 10/01/1996 01/01/1997 04/01/1997 07/01/1997 10/01/1997 01/01/1998 04/01/1998 07/01/1998 10/01/1998 01/01/1999 04/01/1999 07/01/1999 10/01/1999 01/01/2000 04/01/2000 07/01/2000 10/01/2000 01/01/2001 04/01/2001 07/01/2001 10/01/2001 01/01/2002 04/01/2002 07/01/2002 10/01/2002 01/01/2003

www.genuity.com Chicken Little was Wrong! • This is far simpler to remedy than IPv4 address exhaustion, because – the solution need not impact end systems (hosts); – the solution need not impact DNS; and – the solution need not impact routers unless they speak BGP-4. • Any solution is complicated by the need for backward compatibility and phased migration. • Time until exhaustion is nonetheless sufficient to architect, design, implement and deploy solutions. • Cisco and Juniper are reportedly well into implementation.

www.genuity.com The RIRs Recognize the Need for Forecasting • Continuing need to further refine projections. • Need for forward-looking proactive forecasting on a regular basis not only for AS numbers, but also for route table entries and IPv4/IPv6 addresses. • Forecasting needs to incorporate allocation data from all three RIRs (APNIC, ARIN, RIPE NCC). • Forecasting needs to be institutionalized by the RIRs themselves, with data readily available to independent researchers.

www.genuity.com The Team

• Assembled by ARIN – Frank Solensky Gotham Networks – kc claffy CAIDA – Scott Marcus Genuity • Active contributions and support by APNIC and RIPE NCC

www.genuity.com IPv4 Address Allocations Cumulative IPv4 Addresses Allocated

2.5

RIPE ARIN

1.5 APNIC llions )

1. IPv4 Addresses (bi Addresses IPv4

. 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Source: Solensky/Marcus/Claffy

www.genuity.com Route Table Growth

Source: Geoff Huston

www.genuity.com Applications Have Distinct Needs

• E-mail is tolerant of delay, but every bit of data must ultimately be delivered correctly. • For real time WWW traffic, delay must be low, but moderate variability of delay is permissible. • Real time Voice over IP (VoIP) and video are tolerant of modest loss, but intolerant of large and variable delay.

www.genuity.com Cross-Provider Differentiated Services

• Internet services today are best-efforts. • Commoditized service, with marginal price equal to marginal cost. • Differentiated services are of limited value today. – Best-efforts are good efforts, so little incremental advantage for differentiated services. – Customers have no demonstrated propensity to pay a premium for better CoS for data services. – Better CoS for real time voice and video still interesting. – CoS today is limited to a single provider’s network.

www.genuity.com Cross-Provider Measurements and SLAs

• Potentially valuable for best-efforts traffic; essential for Differentiated Services. • Provider responsibility ends today at the peering interconnection point. • No consistent measurement framework. • No business arrangements to incent SLA adherence between peer networks. • Information sharing of proprietary data is a difficult but not intractable problem.

www.genuity.com Summary

• The distinction between peering and transit is vital to an understanding of the underlying business models that drive global Internet connectivity. • The business models that have evolved around peering, in North America and globally, are complex but rational. • Peering business relationships have adapted quickly in response to economic or regulatory stimuli, and will continue to evolve.

www.genuity.com References

Cremer, Rey and Tirole, “Connectivity in the Commercial Internet”, presented at “Competition and Innovation in the Personal Computer Industry”, San Diego, April, 1999. Marcus, Designing Wide Area Networks and Internetworks: A Practical Guide, Addison Wesley Longman, 1999. Chapter 14, which describes peering and transit, is available at http://www.genuity.com/about/executives/marcus_bio.htm Huston, “Interconnection, Peering and Settlements”, http://www.isoc.org/inet99/proceedings/1e/1e_1.htm Laffont and Tirole, Competition in Telecommunications, MIT Press, 2000. General analysis of the economics of telecommunications pricing. Kende, “The Digital Handshake: Connecting Internet Backbones,” FCC, 2000. Gao, “On Inferring Autonomous System Relationships in the Internet, IEEE GlobeCom 2000, San Francisco, November, 2000. Milgrom, Mitchell, and Srinagesh, “Competitive Effects of Internet Peering Policies”, presented at the American Economics Association, January, 2001. Laffont, Marcus, Rey and Tirole, “Internet Interconnection”, American Economics Review, May, 2001.