Future Internet
Revisiting the Internet architecture?
Research @ INET
Stefan Schmid 1 The “Internet” r Goal: m Better use of computing power m connect expensive machines at unis and research labs r Rumors: m Arpanet for robust communication
2 Internet design goals
(in decreasing order of importance) - Connect existing networks - Survivability - Support multiple types of services - Must accommodate a variety of networks - Allow distributed management - Allow host attachment with a low level of effort - Be cost effective - Allow resource accountability 3 The “Internet”: What it is
r Social phenomena m Cyperspace m Changing/redefining communication • Human to human • Human to computer r Business impact m Changing/redefining processes r A man-made artifact m Well defined (in principle)
What makes the Internet so sexy
- Applications can be deployed by anybody that is connected to the Internet (Fundamentally different to the Telephone world) - Multi-service network: Everything over the Internet
5 The “Future Internet”?
- Information and Media Society - Everyone generates content - Sensors and cameras everywhere - New distribution channels - Challenges - Verification - Fusion/filtering of information - Situation adaptation - Incentives, trust, privacy
6 Today’s Internet: Out of Shape! Data plane Control plane
Picture due to Rui Aguilar
Especially control plane not well structured (“fat waist?”) Redesign needed? 7 Today’s Internet: Architectural limits - Trust assumptions - Internet assumes cooperation - Competition - Original Internet assumed no commercial considerations - Edge diversity - Original Internet is host-centric - Ignores mobility, sensors, ... - Network services - Original Internet exposes limited information - Limits new services - Limits network management - Almost no changes in the network core - Designed to be a open, cooperating system - Focus on data plane
8 Why rethink the Internet architecture Reliability and availability E-Commerce increasingly depends on fragile Internet Debuggability Security Known vulnerabilities lurking in the Internet (prevent DoS?) Addressing security has a significant cost Scale & diversity Cyberspace (everything is networked) Support for new applications/services Mobility / Quality of service High speed connections to the home Economics Cost-effectively Business models
9 Rethinking the Internet architecture
Explore alternative architectures Approach Incremental • Apply point-solutions to the current architecture Clean slate design (CSD) • Start from scratch Advantage CSD No limitations: enables rethinking of the network and service architecture Architecture not intrinsic New types of experiments are possible
10 How to get there?
How to determine that one has a good new architecture? Paperware? No Built, evaluated, deployed? Yes Intellectual challenge: uncover otherwise ignored system aspects Approach: Research how to build/operate an experimental facility Research into new architectures Benefit: Go beyond point solutions
11 Clean slate design: Drivers - Technical - Virtualization techniques - Cloud computing / networking - Significant computational resources in the network - Fast packet forwarding hardware, e.g., OpenFlow
- Starting points - PlanetLab / OneLab - Geant2/Internet2 - Emulab - Vini - … 12 Future Internet: Sample initiatives and projects
EU IST FP7 Future CNGI Project (China) Internet projects http://www.future- internet.eu/activities/fp7- G-LAB funded by BMBF projects.html (Germany)
Future Internet network design – FIND http://find.isi.edu/ AKARI Project (Japan)
ANR (France) it839/u-it839 (Korea) Clean Slate Program (Stanford University) Groupe de Reflexion Internet du Futur (France)
Super Janet funded by EPSRC (UK)
Internet del Futuro (SP) NICTA (Australia)
13 OpenFlow: Flow-based Networking
An exciting new technology for separating hardware and software
• Incremental for data plane • CSD for control plane Quick 101
ControllerExecute packet_in OpenFlowevent handler program Default: forward Install rule; to controller forward packet
Host A Host B Packet Switch 1 Switch 2 Flow Table Rule 1 Match Actions Counters Rule 2 Dst: Host B Fwd: Switch 2 pkts / bytes Rule N
15 Why ?
Classical Switch: Cisco 3548XL
16 Why ?
Control software (Management, STP, Basic VTP, etc. )
Control Hardware: Embedded PC
Dataplane: Interfaces, TCAM, Switch fabric
Classical Switch: A closed box Cisco 3548XL 17 Why ?
r Operator says: Cisco's answer:
But I need Buy one of these!
extended VTP (VLAN trunk
protocol) / a 3rd spanport
etc. !
SPAN: switched port analyzer (port mirror) 18 Why ?
r Operator says: Cisco's answer:
I need something We don't better than STP for my have that! datacenter...
19 Why ?
r On a more serious note: m Closed HW/SW hurts innovation m Researchers can't try their ideas at scale m Data center ops can't differentiate by building customized network
20 OpenFlow – An enabler for open control of the network
Local header Network forwarding Control Generic decision per concept of packet: per flow A flow of switching today consists of OpenFlow Switch 100s of Network Forwarding Table packets Control Calculation Secure sw Channel Packet Forwarding Flow Traditional Engine hw OpenFlow Table Switch Switch r OpenFlow moves control path in each switch/router (middlebox) to an external controller, which makes policy decisions at the flow level (flow is defined by Layer 2, Layer 3 and/or Layer 4 header fields). r With OpenFlow, each switch speaks a separate control protocol with an external controller over TCP+SSL 21 Example service using OpenFlow Per-flow policy and QoS on demand
- Interface between the OpenFlow network and the client enables, e.g., on demand - Constraints on route - Temporary upgrade in service/QoS
OpenFlow devices
QoSQoS /privacyboost: Increasecontraints bandwidth: for this.Delay specific < 10ms flow. .Bandwidth > 1Mbps .Not through region X Controller
22 OpenFlow usage – Simple virtualization. Dedicated OpenFlow network.
Controller ’ Peter s code ’ OpenFlowPeter sSwitch Rule PC
Decision? OpenFlow Protocol
’ ’ OpenFlowPeter Switchs Rule OpenFlowPeter Switchs Rule
23 OpenFlow is already supported on routers/switches.
Juniper MX-series NEC IP8800 WiMax (NEC)
HP Procurve 5400 Cisco Catalyst 6k PC Engines
Quanta LB4G 24 Software-Defined Networking (SDN)
Third-party control program
Control Control Control Control
25 An OpenFlow based Router
Taking advantage of + OpenSource Routing Software + Inexpensive Switch Hardware
26 OpenFlow based router: FIBIUM. Evaluation of open-source routers. Levers and current situation Our Approach . ISPs spend significant annual . Understand if low cost switches with CAPEX for routers etc. open-source software can be suitable replacement for high-cost . Currently a quasi-monopoly vendor market for backbone routers. routers and routers in huge data . Build and evaluate prototype router centers. using low-cost components and open-source software. . Modularization/standardization of router components/open-source software may open the market . Similar to Linux . Industry standards for blade servers The OpenFlow might be a low cost, flexible alternative
27 OpenFlow based router: FIBIUM. Today’s inflexible routers.
Current IP routers Observations . Proprietary software . Carrier-grade open-source based routers, e.g., Vyatta, IPInfusion, Quagga . Limited features customization . High-performance and inexpensive layer- . No access to datapath 3 switches with OpenFlow support, e.g. HP, NEC, Cisco . Optimized but non- programmable hardware
28 OpenFlow based router: FIBIUM. Divide and conquer.
Principles Observations . Decouple routing and datapath . Current switches have sufficient datapath performance . Combine inexpensive OpenFlow-enabled switch with . Switch control logic is limited open-source routing software . Current commodity hardware has on commodity hardware better performance than embedded micro-processor on switch
29 Exploiting Zipf’s Law
30 Cache&Aggregate
Bottleneck:
: Aggregate smart
Exploit locality:
Details:
31 OpenFlow based router: FIBIUM. From concept to reality.
FIBIUM RouteVisor . Leverages OpenFlow interface of switch: . Ensures that switch and PC combination . RouteVisor programs the switch appears as a router to the outside world . Route cache management ensures . Interface between route control logic good fast path performance despite on PC and switch limited switch control logic . Collects traffic statistics from switch . Slow path handled by PC and updates data path on switch
32 OpenFlow based router: FIBIUM. Prototype.
Test lab Performance evaluation
. Test: 2 HP switches, commercial . Control plane: 100K BGP updates per routers (Cisco and Juniper) and minute Linux routers . Communication channel between PC and switch: . RouteVisor tasks: . Traffic statistics: 100K per second . Switch configuration . Switch data path updates: 1000 . Handle OSPF and BGP modifications per second messages
. Route Cache management
33 Lessons learned
- Open interface == new opportunities - Flexibility of software wins over closed systems - Example: FIBIUM - Control: Open source routing software - Data path: Commodity switches Case study: Locating performance problems aided by network virtualization
35 Network debugging: Motivation
How do you test an update to the configuration or software of your system?
testbed simulation ✓may be more accurate ๏ but user behavior? ✓scalable ✓ ๏ expensive to run cheap ๏ on large scale? ๏ accuracy? ๏ longtime?
Problem: bugs may happen rarely, non-deterministically, based on external events (Heisenbugs), … 36 Network diagnosis/debugging
- Problem: Implementation/configuration issue surface in large-scale, long-term deployments with real user traffic - Goal: - Do not change network under test - Avoid probe effect - Diagnosis methods: - Instrumentation - Regression tests
37 Instrumentation
Vhost Vhost Moni Moni
Vhost Substrate Moni VNET 1 Monitoring
- Pair production VNet with monitoring VNet - Copy all/selected packets to monitoring VNet - Processing is accounted to monitoring VNet 38 Regression testing – Shadow VNet
Ext 2 Ext 1 Output of Vnet 1.0 dist'ed to ext entities V1.0 Input dist'ed to Vnet 1.0 V1.0 and Vnet 1.1 V1.1 V1.1 Ctrl Ctrl
V1.0 V1.1 Ctrl Substrate VNet running V1.0 VNet running V1.1 Control Vnet Regression testing – Shadow VNet
Ext 2 Ext 1 Output of Vnet 1.0 dist'ed V1.0 to ext entities Input dist'ed to Vnet 1.0 V1.0 V1.1 and Vnet 1.1 V1.1 Ctrl Ctrl
V1.0 V1.1 Ctrl Substrate VNet running V1.0 VNet running V1.1 Control Vnet
- Run VNet1.0, VNet1.1 monitoring VNet - Distribute external input to both VNet1.0 and VNet1.1 - Ctrl compares output behavior of VNet1.0 and VNet1.1 for semantic equality - Only output of VNet1.0 is distributed to external entities 40 Lessons learned
- New network debugging features - Instrumentation - Regression tests - Goals - To not change network under test - Avoid probe effect - Solution: Network virtualization - Isolation - Accounting of resources Case Study:
OFRewind: Enabling Record & Replay Troubleshooting for Networks
42 The Vision Debugger for the network! Example problem:
After upgrade of router B, Router C loses connectivity to network 1. What happened?? 43
Goal: Replay
Idea: - collect sequences of routing messages at different switches, compare before and after upgrade (record + replay) - e.g., with Shadow VNet we can copy traffic, so also record and replay - noticed when reproducing error: routes to Network 1 are not announced by router B due to its handling of classful vs CIDR IP advertisements
44
Another motivating use case
switch CPU high => bad performance
CPU Utilization of an OpenFlow switch 45 No correlation!
Arrivals of PKT_IN msgs 46 No correlation!
Arrivals of FLOW_MOD msgs 47 No correlation!
48 Clueless...
Switch is a black box component Can't inspect internal state, source code No analytical explanation for the behavior Message arrivals do not correlate with symptoms Existing interfaces (CLI, SNMP) too coarse grained
49 A solution?
Replay debugging Log, then retrace your steps to the problem Not available in networks (yet) Existing systems target hosts (no blackboxes supported)
Full recording of all events infeasible?
50 A solution?
Record Replay Trouble- shoot
In production Reproduce at convenient location / pace 51 Existing approaches
Endhost Replay TCPDump / TCPReplay Debugging et. al. Full recording of all Fully deterministic replay, Capture/Replay events via binary instrumentationevents / feasible? ✘ Single vantage point, virtualization no network wide view ✘ No black boxes ✘ Scalability due to
✘ Scalability? dataplane rates
52 However...
Not all traffic is equal (ctrl plane: 1% traffic, 95-99% bugs!)* Behavior of many network devices:
Largely deterministic w.r.t. control plane network events
* Altekar / Stoica, 2010 53
Record Replay events + traffic Reinject events + traffic Selective: Record "best effort replay" important traffic (control) Replay partial recordings Skip/aggregate less Reproduce problem at a important traffic chosen time / location (data plane)
Go Network* Wide / Always On!
* controller domain 54 Replay tweaking
Localize problems through:
Device mapping Time dilation Trace bisection map recording to different iteratively replay subselection devices (change recording, Scale time localize events that e.g., IP address, before replay) investigate timing issues trigger failure versions investigate
regressions / vendor implementation issues 55 Goals
✓Record a controller domain ✓ Scalable, selective, consistent ✓ Even with black boxes ✓Coordinated Replay ✓ Replay tweaking ✓ Localize problems
56 Non-Goals
✘Root cause analysis ✘Automatic configuration of what to record ✘Fully deterministic replay
57 Introducing the tool
58 System design
2 components of 2 modules each:
59 OFRecord
OpenFlow controller
DataStores OFRecord DataStore
pm
p1 p2 sw1 sw3 sw2 c6 c1
c2 c4 c5 c3
60 OFRecord
OpenFlow controller
DataStores OFRecord DataStore
pm
p1 p2 sw1 sw3 syn sw2 c6 c1
c2 c4 c5 c3
61 OFRecord
OpenFlow controller
DataStores PKT_IN(p1) OFRecord DataStore
PKT_IN(p1) pm
p1 p2 sw1 sw3 sw2 c6 c1
c2 c4 c5 c3
62 OFRecord
FLOW_MOD(p1OpenFlow➞p2) controller
DataStores FLOW_MOD(p1➞OFRecordp2, pm) DataStore
pm
p1 p2 sw1 sw3 sw2 c6 c1
c2 c4 c5 c3
63 OFRecord
OpenFlow controller
DataStores OFRecord DataStore
pm syn p1 syn p2 sw1 sw3 sw2 c6 c1
c2 c4 c5 c3
64 OFReplay
DataStores FLOW_MOD(p1PKT_IN(p1)➞ p2) FLOW_MOD(pm➞OFReplayp2) DataStore
pm syn PKT_IN(pm) p1 p2 sw1 sw3 sw2
65 Typical Usage
Deploy OFRecord in production environment -> proxy to 'regular' controller Always-on OF messages, control plane, data plane summaries Alter selection rules as necessary Deploy OFReplay in lab environment Localize bugs / validate bug fixes
66 Example Usages
Debugging Black box components CPU inflation in an OpenFlow switch Debugging OpenFlow controllers NOX problem Debugging non-OF: Quagga RIP bug …
67 Lessons learned
New primitives:
Selective, consistent, Adaptive coordinated multigranularity & best-effort Network Recording Network Replay
Reproduce problems at convenient time and place Combined in OFRewind, an Open-Flow based tool for Network Record & Replay http://www.openflow.org/wk/index.php/OFRewind Enables practical record and replay of network domains
68 Software-Defined Networking (SDN)
Third-party control program
Control Control Control Control
69 SDN Promises
Advantages over status quo of management
Reduce complexity
New functionality through programmability
SDN is great, but …
70 … at the risk of bugs
71 Bugs in OpenFlow Apps Controller
OpenFlow Install ? program rule Install Delayed! rule
Unwanted distributed state! Host A (e.g., forwarding loop) Host B Packet Switch 1 Switch 2
72 Software Faults
• Will make communication unreliable
• Major hurdle for success of SDN
We need effective ways to validate SDN networks
73 Systematically Testing OpenFlow Apps
State-space exploration via Model Checking (MC) • Carefully-crafted streams of packets Target Unmodified • Many orderings of OpenFlow system packet arrivals program and events Environment model
Switch Switch 1 Complex 2 environment Host A Host B
74 Scalability Challenges Data-plane driven Complex network behavior Huge space of Huge space of possible possible packets event orderings
Equivalence Domain-specific classes of search packets strategies
Enumerating all inputs and event orderings is intractable
75 Input NICE Output No bugs Unmodified In OpenFlow Controller program Execution
Traces of State-space Network property search topology violations
Correctness propertiesNICE found 11 bugs in 3 real OpenFlow Apps (e.g., no loops)
76 Enabling Alternative Architectures
Network Virtualization Architecture: Proposal and Initial Prototype
77 The Vision: Sharing Resources Requests with Flexible Specification. Optimization and Migration?
„VPN++“ Datacenters
Goal: Fully specified CloudNet mapping constraints (e.g., end-points for a telco), but with QoS guarantees any (e.g., bandwidth) along links < 10ms „Guaranteed Berlin < 10ms > 100 resources, job „November > 100 MB/s MB/s deadlines met, 1Mbit/s 1Mbit/s 22, 1pm- no overhead!“ 2pm!“ any any < 10ms Tel Aviv > 100 MB/s Palo Alto 1Mbit/s
Spillover/Out-Sourcing Migration / Service Deployment
Elastic Goal: Move with the sun, with the commuters computing Berlin
< 50ms „50 TB storage, 10 Tflops computation!“ Berlin < 50ms (corporate access network)
Stefan Schmid Telekom Innovation Laboratories 79 Federated CloudNet Architecture. SIGCOMM VISA 2009 New Business Roles.
Roles in CloudNet Arch. As in Internet today: Service Provider (SP) Netflix, Google, (offers services over the top) World of Warcraft…
Virtual Network Operator (VNO) (operates CloudNet, Layer 3+, triggers migration
Virtual Network Provider (VNP) (resource broker, compiles resources) As in Internet today: Telekom, AT&T, … Physical Infrastructure Provider (PIP) + resource control interface (resource and bit pipe provider) (bootstrapping etc.)
Stefan Schmid Telekom Innovation Laboratories 80 Federated CloudNet Architecture. SIGCOMM VISA 2009 New Business Roles.
Roles in CloudNet Arch. Service Provider (SP) (offers services over the top)
Virtual Network Operator (VNO) (operates CloudNet, Layer 3+, triggers migration
Provide layer 2: assembles Virtual Network Provider (VNP) CloudNets, resource and management interfaces, Resource (resource broker, compiles resources) broker… (recursive) Physical Infrastructure Provider (PIP) (resource and bit pipe provider)
A virtualized infrastructure opens new roles for the allocation of resources and the operation of networks!
Stefan Schmid Telekom Innovation Laboratories 81 Federated CloudNet Architecture. SIGCOMM VISA 2009 New Business Roles.
Roles in CloudNet Arch. Service Provider (SP) (offers services over the top)
Build upon layer 2 (op on virt IDs): clean slate! (OSN, …) Virtual Network Operator (VNO) Routing, addressing, multi- (operates CloudNet, Layer 3+, triggers migration path/redundancy… (view inside CloudNet!)
Virtual Network Provider (VNP) (resource broker, compiles resources)
Physical Infrastructure Provider (PIP) (resource and bit pipe provider)
A virtualized infrastructure opens new roles for the allocation of resources and the operation of networks!
Stefan Schmid Telekom Innovation Laboratories 82 Federated CloudNet Architecture. SIGCOMM VISA 2009 New Business Roles.
Roles in CloudNet Arch.
Service Provider (SP)
(offers services over the top)
Contract Contract based: /specs /specs down the hierarchy Communicate requirements API
Virtual Network Operator (VNO) (operates CloudNet, Layer 3+, triggers migration API Virtual Network Provider (VNP)
(resource broker, compiles resources) s: e.g., provisioning e.g., s:
API
Physical Infrastructure Provider (PIP)
API (migration) interfaces (resource and bit pipe provider)
Stefan Schmid Telekom Innovation Laboratories 83 Research Challenge 1: Embedding. Connecting Providers (Geographic Footprint).
CloudNet 1: Computation CloudNet 2: Mobile service w/ QoS Specification: Specification: 1. > 1 GFLOPS per node 1. close to mobile clients 2. Monday 3pm-5pm 2. >100 kbit/s bandwidth for synchronization 3. multi provider ok
CloudNet requests
Provider 2 Provider 1
Physical infrastructure (e.g., accessed by mobile clients)
Stefan Schmid Telekom Innovation Laboratories 84 Research Challenge 2: Access Control.
Stefan Schmid Telekom Innovation Laboratories 85 Prototype at INET. YouTube Migration Demo
Stefan Schmid Telekom Innovation Laboratories 86
http://www.youtube.com/watc h?v=llJce0F1zHQ Lessons learned
- Isolate tasks => business opportunities - E.g.: Magnitude of the investment cost • AT&T plans to invest 17–18 Bn $ in 2009 compared to a revenue of 124 Bn $ in 2008 • Deutsche Telekom plans to invest 8.7 Bn Euro compared to revenues of 62 Bn Euro in 2008 1% is substantial! - Don’t forget control interfaces - Interprovider issues are tricky - Indirection and resource isolation are great tools
87 CSD: Reshaping the Internet - Impact on users: - Ease of access to relevant information - New control plane with new capabilities - Easy to introduce new applications with new features • Security, mobility, quality of service - Impact of new economic models: - New interfaces between providers (network/service) - New value-chain and new roles for providers - Open interfaces may enable new ecosystems of business alliances - Impact on society: - Information society - Impact on operators: - Easier network management 88 Evaluation
89 Exam
90 INET Lectures
91 Thanks for your interest in NPA!