TCP-XM: Unicast-enabled Reliable Multicast Karl Jeacle and Jon Crowcroft Computer Laboratory University of Cambridge fi[email protected] Abstract fully reliable data transfers, while lowering network utilisa- tion by taking full advantage of whatever multicast connec- In recent years, much work has been done on attempting tivity is available. to scale multicast data transmission to hundreds or thou- One way of implementing this combined unicast / mul- sands of receivers. There are, however, many situations ticast approach is to embed multicast capability within an where an application might involve transmission to just ten existing unicast transport protocol such as TCP. The con- or twenty sites. sequence of such an approach would be that programmers Using multicast for this type of application can provide would see a familiar, but extended, API that supports one- significant benefits including reduced load on the trans- to-many group communications. By using the group exten- mitter, an overall reduction in network traffic, and conse- sions, applications could take full advantage of any (partial) quently shorter data transfer times. multicast coverage, without having to explicitly program for In this project, we are investigating how partial or in- IP multicast. complete multicast can be exploited alongside reliable uni- cast to improve both speed and efficiency of data transfers 2 Protocol Design while maintaining reliability. The approach taken is to combine unicast with multicast Today, applications use TCP for reliable unicast trans- by modifying TCP to support multicast transfers, and run fers. It is a mature and well-understood protocol. By mod- this modified TCP engine over UDP as a userspace trans- ifying TCP to deliver data to more than one receiver at a port protocol. We describe the work to date on the design time, and use multicast when available, an application can and implementation, and provide experimental results from transparently send data reliably to multiple recipients. Us- our tests across both local and wide area networks. ing existing TCP mechanisms, the modified protocol en- sures that data is delivered reliably. Multicast transmission is attempted for performance reasons, but fallback to uni- 1 Introduction cast preserves backwards compatibility and reliability guar- antees, and capitalises on more than a decade of experience Today, small scale group communication will typically that TCP implementations enjoy. take place using unicast transmission. Only when the net- Network protocols are typically implemented in the ker- work is capable, and the application demands it, will mul- nels of hosts and network devices. Any proposals that re- ticast be used. As a result, large clouds of native IP multi- quire modifications to these protocols imply changes to ker- cast enabled networks are unnecessarily burdened with du- nel code. This immediately restricts deployment opportuni- plicate unicast data. ties. By limiting changes to code that runs in user-space on Depending on the state of unicast and multicast connec- end stations, new protocols can be developed and tested on tivity in the network, the right combination of unicast and live networks. multicast transmission can provide a large increase in effi- TCP is a reliable end-to-end unicast transfer protocol im- ciency at the cost of a small decrease in speed. plemented in host kernels. It is possible to modify TCP We believe that for large sender-initiated data transfers behaviour between end stations without any changes to in- to a small group of receivers, not only does such a com- termediate devices, however this requires kernel changes. If bined unicast and multicast transmission sweet spot exist, TCP is moved into user-space, changes can be made with- but that it can be quantified, and automatically found dur- out modifying the kernel, but again, some form of privileged ing the transmission process. Applications can then initiate access is needed by the user-space TCP implementation to directly send and receive packets on a host’s network inter- 2. The user can specify a group address for multicast, faces. While not as significant a barrier to widespread de- otherwise a random group address will be allocated ployment as kernel changes, this privileged access require- automatically. ment severely limits the ease with which new code can be widely tested. 3. Multiple TCP Protocol Control Blocks (PCBs) are One solution to this problem is to implement a modified created – one for each destination. multicast TCP over UDP. User-space applications can freely 4. Independent 3-way handshakes take place. send and receive UDP packets, so a small shim layer can be introduced to encapsulate the TCP engine’s packets into 5. The group address is sent as option in the SYN UDP. While there are performance implications by running packet. Multicast depends on the presence of the in userspace, the instant deployment potential of a userspace group option in the SYNACK. A PCB variable then library, coupled with the scalability of multicast, mean that dictates the transmission mode for the PCB. any such limitations are more than acceptable. The key advantage of this approach is that any applica- 6. User data writes are replicated and enqueued on all tion can make use of this new protocol by simply linking PCB send queues. against the supplied library. No changes are required in the 7. Data packets are initially unicast and multicast simul- network (other than enabling IP multicast). taneously. Multicast packets have protocol type TCP, Because the protocol is not tightly coupled to the appli- but the destination is a multicast group address. cation, should it become widely adopted, a native imple- mentation can be built in the kernel to boost performance. 8. Native IP network multicast transmission capability is used 2.1 Modifying TCP 9. Unicasting ceases on successful multicasting. In this section we will describe our modifications to TCP 10. Multicasting continues for the duration of the session. to support multicast. We call our new protocol TCP-XM. The key feature of TCP-XM is its focus on combining 11. All retransmissions are unicast. both unicast and multicast simultaneously. Multicast trans- mission will always be attempted, but equally, fallback to 12. Automatic fallback to unicast transmission takes unicast transmission will always be available. place after n failed multicasts packets. The TCP-XM approach is a pragmatic one. It states 13. Unicast & multicast sequence numbers stay synchro- some ground rules and initial assumptions about the tar- nised – min(cwnd) & min(snd wnd) used. get audience (small scale group communications). It lends itself to a prototype implementation over UDP, with no 14. Connections are closed as per TCP. changes required in the network. Unlike most other pro- posed reliable multicast protocols, this allows the protocol From the receiver’s perspective: to be implemented and tested on a large scale. This capa- bility will serve as a useful feedback loop into the design of • No API change is necessary. the protocol. • In order to describe TCP-XM functionality, we will walk Normal TCP listen takes place. through a typical TCP-XM session. Let us first state some • IGMP group join on incoming TCP-XM connect. rules or assumptions about how the protocol should operate: • Accept data on both unicast and multicast addresses. • TCP-XM is primarily aimed at push applications • It is sender initiated TCP PCBs are stored on a linked list. This has not been changed, however some extra variables have been added to • The sender has advance knowledge of destination the PCB structure. These include: unicast addresses struct tcp_pcb { Given these assumptions, the “call process” for a TCP- ... XM sender is as follows: struct ip_addr group_ip; 1. The application connects to multiple unicast ad- enum tx_mode txmode; dresses. It does not connect to a multicast group ad- u8t nrtxm; dress (a Class D IP address in the range 224.0.0.0 to struct tcp_pcb *nextm; 239.255.255.255). } The group address and transmission mode are dependent 2.2 Multicast Detection on the TCP group option discussed below. The nrtxm variable keeps track of how many times a When TCP-XM transmits data via unicast or multicast, unicast retransmission has been necessary for segments that an acknowledgement packet will be returned on success. have previously been multicast. Too many of these retrans- But a standard ACK does not indicate whether a segment missions will result in the transmission mode of the PCB has been received via unicast or multicast. falling back to unicast. In order for a TCP-XM transmitter to make an informed The nextm pointer references the next PCB in a chain decision about when to switch from unicast to multicast of PCBs associated with a single TCP-XM connection. transmission, it needs to know what the state of the net- When a TCP-XM connection is made, a SYN packet is work is, or to be more precise, what the state of multicast sent that includes a TCP option specifying a multicast group reception is like at the receiver. address. If not specified by the user, the group address is au- A feedback mechanism that allows the receiver to in- tomatically chosen by TCP-XM. The presence of this option form the sender of multicast reception is desirable. TCP- means that the originating host has TCP-XM capability. XM achieves this through the introduction of a TCP header If the receiver is running TCP-XM, it can accept the option that indicates the percentage of the last n segments group address offered or, in exceptional circumstances, re- that have been received via multicast. The value of n de- turn the SYNACK with an alternative proposed group ad- faults to 128. dress. On connection establishment, an IGMP join request The receiver inspects each segment that arrives and is issued.