Using Dynamic Configuration to Manage a Scalable

Computer Communications 24 (2001) 105±123 www.elsevier.com/locate/comcom

Using dynamic con®guration to manage a scalable multimedia distribution systemq

F. Kon*, R.H. Campbell, K. Nahrstedt

Department of Computer Science, University of Illinois at Urbana-Champaign, 1304 West Spring®eld Avenue, Urbana, IL 61801-2987, USA1 Received 15 March 2000; accepted 4 September 2000

Abstract Multimedia applications and interfaces will change radically the way computer systems will look like in the coming years. Radio and TV broadcasting will assume a digital format and their distribution networks will be integrated to the Internet. Existing hardware and software infrastructures, however, are unable to provide all the scalability, ¯exibility, and quality of service (QoS) that these applications require. We present a framework for building scalable and ¯exible multimedia distribution systems that greatly improves the possibilities for the provision of quality of service in large-scale networks. We show how to use architectural-awareness, mobile agents, and a CORBA-based framework to support dynamic (re)con®guration, ef®cient code distribution, and fault-tolerance. This approach can be applied not only for multimedia distribution, but also for any QoS-sensitive distributed application. q 2001 Elsevier Science B.V. All rights reserved.

Keywords: Multimedia distribution; Dynamic con®guration; Middleware; CORBA; QoS-aware resource management

1. Introduction on architectures for distributing digital multimedia content through the Internet and intranet infrastructures. In particu- Multimedia interfaces will play a fundamental role in lar, researchers working on the technologies related to the human±computer interaction in next generation computer MBone [1,2] and to adaptive audiovisual streaming proto- systems. Applications will gradually abandon dull inter- cols [3,4] provided signi®cant contributions. We noticed, faces based on text and static graphics and move towards however, that existing MBone and unicast solutions do a more interesting look based on audio, video, and anima- not provide the degree of control, ¯exibility, and the possi- tions. Within one decade, there will probably be no distinc- bilities for QoS management that the next generation appli- tion among the telephone, radio, TV, and data networks; and cations require. In this article, we present an architecture for in future decades, no distinction between computer and scalable multimedia distribution that meets those require- dedicated radio and TV receivers. Everything will be inte- ments. The architecture combines modern technologies grated on an extension of today's Internet. from different Computer Science ®elds and extends them The Internet as we know today, however, is not suited for when necessary. the distribution of high-quality multimedia to a large We applied the ideas presented in this article to build an number of clients. A number of changes in the hardware object-oriented multimedia distribution system in C11 [5] and software that currently support the Internet will have and demonstrated that it is possible to use the existing Inter- to occur. We need systems to better control large numbers of net to distribute low and medium bandwidth multimedia to multimedia ¯ows, with support for Quality of Service (QoS) thousands of simultaneous users. Our early experiments, provision. though, pointed out dif®culties in managing such a large- In the past ten years, several research groups have worked scale system and keeping it available with an acceptable QoS. It showed the necessity for a better support for q This research is supported by the National Science Foundation, grants dynamic (re)con®guration, code distribution, and provision CCR 96-23867. 98-70736, 99-70139, and 99-72884EQ. Fabio Kon is of fault-tolerance. supported in part by a grant from CAPES, Brazil, proc. 1405/95-2. This article describes our most recent achievement in * Corresponding author. this area, i.e. an integrated software architecture addressing E-mail addresses: [email protected] (F. Kon), [email protected] (R.H. Campbell), [email protected] (K. Nahrstedt). the problems we encountered in the past. Our solution is 1 http://choices.cs.uiuc.edu/2K. based on an extensible Re¯ector system, mobile agents,

... Intermediate Public Reflector Reflector Capture Master Station Reflector

Public Intermediate Reflector Reflector

Public Reflector

Intermediate . Reflector . . Capture Master Station Public Reflector Intermediate Reflector Reflector

Fig. 1. A Re¯ector network distributing two video streams. and a CORBA framework for dynamic con®guration and the system to the characteristics of the environment in which recon®guration. it executes. This property is known as dynamic con®gura- Section 2 presents an overview of the basic concepts tion. Also of extreme importance is the ability of a system to involved in scalable multimedia distribution. Section 3 change, on-the-¯y, its internal components and con®gura- describes our early design, discussing its limitations. tion parameters to adapt to changes in the environment. This Section 4 describes our new, enhanced architecture with is called dynamic recon®guration2. support for dynamic con®guration and explains the syner- (4) Quality of service (QoS). Multimedia applications gistic relationships between dynamic con®guration and have stringent requirements with respect to computational QoS. Section 5 presents a concrete implementation of our resources such as memory, CPU, and network [6]. If these architecture and Section 6 discusses performance evalua- requirements are not met, the quality of the multimedia tion. Finally, Section 7 discusses related work and Section service is degraded. This quality can be measured according 8 presents our conclusions and future work. to different metrics. A videoconference service, for example, can be evaluated according to the size and the number of colors of the video frames, the audio sampling rate, the 2. Basic concepts video frame rate and jitter, the synchronization of audio and video, the delay from the sender to the receiver, the number Before going into a more detailed description of our of frames that are lost or defective, and so on. architecture, we ®rst present a brief overview of the most (5) CORBA. The OMG Common Object Request Broker important concepts related to our approach to scalable Architecture is an architecture for distributed object multimedia distribution. communication that is both language-independent and (1) Re¯ector. It is the key element of our distribution platform-independent. It is based on a standard interface system. It acts as a relay, receiving input data packets de®nition language (IDL) and includes standard de®nitions from a list of trusted sources and forwarding these packets for interoperable distributed communication [7]. It also to other Re¯ectors or to programs executed by end-users. de®nes standard interfaces for services such as naming, The distribution system is composed of a network of Re¯ec- trading, real-time, persistence, and transactions [8]. tors that collaborate with each other to distribute the multimedia data over local-area, metropolitan-area, and wide- area networks. 3. Scalable distribution (2) Re¯ector administrator. A privileged user that is responsible for managing a Re¯ector network. Different Using traditional, centralized video-on-demand servers, it portions of the network may be managed by different is possible to stream video to hundreds of simultaneous administrators, making large systems more manageable clients [3]. Exploiting IP-Multicast [9], this number and helping to deal with different security domains. increases to several thousands or, maybe, a few millions. (3) Dynamic (re)con®guration. As computer environments become more dynamic, a major requirement for 2 For simplicity, the term dynamic con®guration may be used to denote next generation computer systems is the ability to customize both dynamic con®guration and dynamic recon®guration. F. Kon et al. / Computer Communications 24 (2001) 105±123 107

IPMulticast/ Reflector local Ethernet F

TCP/ TCP / longdistance fiber link interstate fiber WTCP Master Reflector Reflector IPMulticast/ A MBone

Reflector TCP/ B phone dialup

UDP/ TCP/ ISDN Capture Master Reflector transatlantic satellite link Station Reflector C

TCP/ TCP / transatlantic satellite link longdistance fiber link

Reflector Reflector VDP/ D E IPMulticast/ UDP/ phone dialup MBone LAN

Fig. 2. A heterogeneous Re¯ector network.

However, when using IP-Multicast to send video to a large news, or current stock values. Clients and administration number of clients, one has little control over the transmis- programs can connect to a Re¯ector in order to get informa- sion. It becomes dif®cult to provide support for a number of tion about available channels, Re¯ector load, bandwidth desired features such as QoS, security, accounting, and utilization, historical statistics, etc. reliability. To address this problem, we developed a scalable The Re¯ector network topology is determined by each multimedia distribution framework whose architecture is Re¯ector's con®guration. This information speci®es input described in this section. and output connections, access privileges, maximum allowed number of users, etc. The information is stored in 3.1. The Re¯ector a database controlled by the Re¯ector administrator. Fig. 1 depicts a generic Re¯ector network that distributes two As noted above, the Re¯ector is the key element of video streams in different channels. In this ®gure, two the distribution framework. It is a user-level program capture stations send their video streams to ªmasterº Re¯ec- that, at the application level, performs activities similar tors; the streams may traverse several ªintermediateº to the ones performed by network routers at the hard- Re¯ectors until they reach ªpublicº Re¯ectors to which ware level (which is where IP-Multicast is implemen- end-user clients can connect and receive the video streams. ted). Since it is implemented in software, not hard- All the Re¯ectors in the ®gure are initiated with exactly the wired into the router, the Re¯ector is more ¯exible, same code. As explained in Section 4.1, the system then easy to deploy and evolve, and can be dynamically customizes each Re¯ector according to their individual customized to users, applications, and environments. requirements. Re¯ector data packets are encoded with RTP, a user- To provide fault-tolerance Re¯ectors can accept redun- level protocol for real-time applications [10] de®ned by dant inputs for the same channel from several sources. It the Internet Engineering Task Force (IETF). RTP pack- assumes that one of them is the primary source and ets can be transmitted over different kinds of low-level ignores the data coming from other redundant sources. transport protocols such as TCP, UDP, and IP-Multicast. If an error in the connection with the primary source is The Re¯ector combines data packets into logical groups detected or if it remains silent for a pre-de®ned period of called channels. It examines a 2-byte channel_ID ®eld in the time, the Re¯ector automatically switches to the next packet header and forwards the packet to the clients and source in its redundancy list. Re¯ectors that subscribed to that particular channel. Each Software routing at the Re¯ectors may introduce extra channel can contain multi-track data for different applica- latency and jitter. Typically, latency will only be a problem tions such as video and audio conferencing, live high- for conferencing applications and only if a long chain of bandwidth MPEG broadcasting with TV quality, stored Re¯ectors is used. Jitter can be minimized by reserving low-bandwidth H.263 video and GSM audio, HTML CPU and memory as described in Section 4.1. 108 F. Kon et al. / Computer Communications 24 (2001) 105±123 3.2. Data distribution protocols mixing, and downsampling. Finally, one can create new Connection types by composing existing ones. For example, In order to support different types of inter-Re¯ector data one can create a CryptoMulticast connection type Ð that distribution protocols transparently, the Re¯ector frame- encrypts the data and sends it out using Multicast Ð by work encapsulates the concept of a network connection composing a Crypto connection with a Multicast connec- into an abstract C11 class named Connection that de®nes tion. the basic interface for all types of network connections used by the Re¯ector. This abstract class implements some of its 3.3. Experience and lessons learned own methods, but the majority of the connection-related code is implemented by its subclasses: TCPConnection, This technology was utilized in the live broadcast of UDPConnection, MulticastConnection, etc. Re¯ector NASA's JPL Path®nder mission [13]. During this broadcast control information and meta-data is sent through reliable Ð which lasted for several months Ð more than one TCP connections. Multimedia data may be sent through million live video sessions were delivered to dozens of unreliable connections in order to achieve a higher through- different countries across the globe by a network of more put. Also, different channels may use different connection than 30 Re¯ectors spread across ®ve continents. The Re¯ec- types; for example, in a single Re¯ector a video channel tors ran in ®ve different operating systems (Solaris, Linux, may use UDP while a text channel with news articles uses Irix, FreeBSD, and Windows) and transmitted their streams a TCP connection. over different kinds of network links. End-users could watch Fig. 2 depicts a concrete example of a highly heteroge- the video stream simply by pointing their web browsers to neous Re¯ector network. In this example, the network selected locations, causing their browsers to download a distributes two audiovisual streams. The ®rst comes from Java applet containing the video client. The applet a mobile camera, mounted on a helicopter, that sends its connected to the Re¯ector automatically, received the stream to a ªmasterº Re¯ector over a wireless link. This video stream, decoded it, and displayed it to the user in kind of link presents a high rate of packet loss not related real-time. to congestion, which makes protocols like TCP perform During this broadcast, we experienced three major poorly. Thus, it is desirable to use a protocol optimized problems: for this kind of situation like, for example, WTCP [11]. The second stream is sent to its ªmasterº Re¯ector through 1. As the code had not been tested on such a large scale and a dedicated ISDN line. To optimize the bandwidth, one can on so many different platforms, we found many program- use UDP as the communication protocol since its overhead ming errors both in the Re¯ector code and in the client is lower than that of TCP and the link offers a low loss rate. applet. Surprisingly, ®xing the error was, sometimes, Re¯ector C sends its streams to Re¯ector D over the public easier than updating the code in the dozens of machines Internet through a transatlantic satellite link. Even though that formed the distributed system. The same problem this is a high-bandwidth link, its loss rate may be high, thus occurred when a new version of the Re¯ector, with it is more appropriate to use TCP. Re¯ectors A and D intro- added functionality, was released. System administrators duce the video streams into two distant points of the global had to manually connect to dozens of machines, upload MBone. the new code, shutdown the old version, and start the new Selecting the appropriate protocol for each situation, the one. Re¯ector administrators improve the quality of service 2. Often, we had to recon®gure the re¯ector network by offered to the end-users, optimizing the utilization of the dynamically changing the distribution topology, or by available network bandwidth and minimizing packet losses. setting new values to the re¯ector con®guration para- Most of the Re¯ector's code deals with objects of type meters (e.g. maximum number of users, number of multi- Connection and is not aware of the Connection's underlying media channels). The con®guration information for the implementation. The actual connection type is speci®ed re¯ectors was stored in a centralized location. After when the connection is created and does not need to be updating this centralized database, we had to connect to known after that. This approach allows programmers to each of the re¯ectors and instruct them to download their plug in customized Connection subclasses by providing updated con®guration. This process was tiresome and their own implementation of the Open, Close, Bind, error-prone. Connect, Send, and Receive methods. In this manner, it is 3. The only mechanism the Re¯ector provided to support possible to incorporate, into the Re¯ector, Connection fault-tolerance was the redundant inputs described in subclasses that implement different transport protocols Section 3.1. But this mechanism leads to a large waste (such as the VDP QoS-aware adaptive protocol for the Inter- of bandwidth. The redundant streams are always trans- net [3] and the xbind QoS-aware reservation protocol for mitted even though they are seldom used. ATM networks [12]). Developers also use this mechanism to implement Connection subclasses that perform various With this experience, we learned that a user-level Re¯ector operations on the data such as encryption, transcoding, system is, indeed, a powerful tool for managing large-scale F. Kon et al. / Computer Communications 24 (2001) 105±123 109 multimedia distribution. It gives Re¯ector administrators a use of the best policies for each situation. For example, a tight control over the distribution, allowing for a better mobile computer displaying a video clip to its user could use control of QoS. It achieves that through the de®nition of a protocol optimized for wireless connections when the the distribution topology, the selection of appropriate computer is using a wireless link, but dynamically recon®- communication protocols, and the possibilities for limiting gure itself to use a TCP connection when the computer is the number of clients according to the available resources. hooked to a wired Ethernet connection. However, if the We also learned, however, that it was important to recon®guration process, itself, affects the quality of service provide better mechanisms for distributed code updates, negatively, it may not be worthwhile to do any recon®gura- dynamic recon®guration, and fault-tolerance. In the next tion at all. Coming back to the example, if the dynamic section we describe a novel architecture for multimedia recon®guration to the TCP connection is so expensive that distribution that addresses the problems encountered in the the video is interrupted for several seconds, it is better to previous approaches, leading to a more ef®cient, manage- keep using the wireless link even when the wired link able, and reliable system. becomes available. We, now, present an enhanced architecture that greatly improves the possibilities for QoS management with new 4. Architecture mechanisms for automatic con®guration (Section 4.1), scalable code distribution and dynamic Re¯ector recon®gura- With the exponential growth in the number of workstation (Section 4.2), and dynamic recon®guration of the tions, laptop computers, and PDAs connected to the Inter- network topology to provide fault-tolerance (Section 4.3). net, and with the extra levels of mobility and dynamism that Furthermore, the dynamic con®guration is carried out with these devices bring, it becomes increasingly important for no negative impact on the QoS of the multimedia streams. multimedia systems to be able to cope with dynamic changes in the environment. They must react to variations 4.1. Automatic con®guration in resource availability, dynamically adapting their algorithms, updating parts of the system, and replacing software To solve the problem of maintaining the Re¯ector components when needed. instances up-to-date as the code of the Re¯ector program The Re¯ector is no exception to this trend as it needs to evolves and to customize each Re¯ector according to its cope with different kinds of devices that may connect to it. role, we adopted the automatic con®guration approach A client running on a PDA, for example, may need to dyna- [16,17]. The Re¯ector is divided into a minimal core and mically add a proxy into the Re¯ector so that it can trans- components that are linked to the core at runtime to extend code the multimedia stream into a format suitable for the its functionality. These components may include the imple- PDA (as done in the PalmPlayer video system [14]). These mentation of the different data distribution protocols and new requirements, together with the lessons learned in the mechanisms for encryption, access control, accounting, past experiences (see Section 3.3), point out the necessity of down-sizing, and transcoding. supporting dynamic con®guration of the Re¯ector system. The core is the only code that is initially shipped to the Although dynamic con®guration is a ®eld that has been nodes that are going to execute the Re¯ector. This is enough studied intensively in the past decade [15], the dynamic to initiate the bootstrapping process. The con®guration con®guration of QoS-sensitive systems pose completely information for all the Re¯ectors in a certain administrative new challenges that have not yet been thoroughly explored. domain are stored in a con®guration service. Fig. 3 depicts a In this case, the system must not only be able to recon®gure schematic view of the automatic con®guration process, correctly, but also be able to carry out this recon®guration which is driven by a module of the Re¯ector core called with minimal impact on QoS. dynamic con®gurator. All the code for con®guration is The synergistic relationships between dynamic con®gura- encapsulated in the dynamic con®gurator. tion and QoS are clear. Dynamic con®guration allows the When a Re¯ector starts, it ®rst contacts the con®guration

outputs 1 inputs Dynamic Reflector Configuration 4 Configurator Service core 3 5 2

QoSAware Resource Management

Fig. 3. Automatic con®guration architecture. 110 F. Kon et al. / Computer Communications 24 (2001) 105±123 4.2. Scalable code distribution and dynamic recon®guration

The automatic con®guration process described above consists of a pull-based approach for code updates and con®guration. In other words, the Re¯ectors take the initia- tive to pull the code and con®guration information from a central location. In large-scale systems, this may not be enough. We also need a mechanism to allow system administrators to push code and con®guration information into the Fig. 4. Low-level QoS speci®cation using the Simple Prerequisite Descrip- tion Format. Re¯ector network ef®ciently. Our architecture achieves that by using the concept of mobile agents [21]. service to retrieve its con®guration (step 1 in Fig. 3). The Using a special con®guration agent builder tool, Re¯ec- con®guration service provides three kinds of con®guration tor administrators are able to create a mobile agent and information to the Re¯ector: inject it into the Re¯ector network. The agent traverses the Re¯ector network and, in each node, is able to perform 1. A list of the components that must be dynamically linked three kinds of actions: recon®gure the Re¯ector changing its to the core to customize this instance of the Re¯ector. working parameters (e.g. maximum number of clients, 2. A speci®cation of the QoS parameters required by this security keys, access control lists), install the code for Re¯ector. new components (e.g. new data distribution protocols, 3. The input and output connections to be created at startup. security mechanisms), and collect information about the Re¯ector dynamic state (e.g. bandwidth utilization, current In step 2, the Re¯ector not only receives its con®guration number of clients, total number of clients since startup). information, but also the actual code for the components to After visiting the nodes speci®ed by the administrator, the be linked to the core. In step 3, it passes the QoS speci®ca- agent returns and displays the results of its actions and the tion to an underlying QoS-aware resource management collected information. service that is responsible for managing local machine Although the introduction of mobile agents promotes resources such as CPU and memory, and, when possible, signi®cant bene®ts, it also brings additional security distributed resources such as the network. The QoS speci®- concerns. This is especially worrisome for QoS-sensitive cation may be as simple as an SPDF speci®cation [17] like applications since malicious or defective agents may utilize the one in Fig. 4, or as elaborated as a QML speci®cation excessive resources and degrade the Re¯ector QoS. Luckily, [18]. the problem of mobile agents security has been studied The QoS-aware resource management service can be extensively in recent years and good results are starting to provided by systems like SMART [19] or the Dynamic appear. In the Re¯ector case, it is important to (1) restrict the Soft Real-Time Scheduler (DSRT) [20]. DSRT, for exam- sources of mobile code to trusted, authenticated entities ple, can use the QoS speci®cation to perform QoS-aware [24], (2) limit the mobile code's resource consumption admission control, negotiation, reservation, and scheduling. [22], and (3) limit its capabilities by con®ning it to secured Since the architecture relies on user-level processing of data environments like sand boxes for Java byte code or the packets, the use of a soft real-time scheduler is important to Janus sand box for native code [23]. guarantee that the Re¯ectors do not impose additional jitter to the multimedia streams. 4.3. Fault-tolerance The con®guration is completed in step 4, when the dynamic con®gurator instructs the Re¯ector to open the Reliability is an important aspect of QoS. It is dif®cult to required input and output connections. Afterwards, when maintain a system working properly in the presence of the Re¯ector is running, the resource management service network failures, node failures, software failures, and partial may detect exceptional changes in the resource availability system shutdowns for maintenance. In addition, keeping the or violations of the QoS requirements that may require desired level of QoS irrespective of all these disruptive modi®cations in the Re¯ector con®guration. In such cases, events is a major challenge for modern QoS-sensitive it issues a call-back to the dynamic con®gurator with infor- systems. The architecture addresses this problem by using mation about the exceptional conditions (arrow labeled 5 in architecturally aware dynamic recon®guration based on a Fig. 3). framework for representing dependencies in distributed This automatic con®guration process greatly simpli®es systems [16,17]. system management since the Re¯ectors always download In multimedia distribution, the goal with respect to fault- the most up-to-date version of each component from the tolerance is to maximize availability without relying on con®guration service. In that way, it eliminates the need redundant transmissions, which lead to waste of bandwidth. to upload components to the entire network each time a To achieve that, the architecture supports the dynamic component is updated. recon®guration of the Re¯ector inter-connections when F. Kon et al. / Computer Communications 24 (2001) 105±123 111

Name Configuration Component Service Service Repository

2 6473 1

Client Source 910Reflector Reflector Reflector 8 5

DSRT Component Cache

Fig. 5. Bootstrapping a Re¯ector. failures occur. The distributed system has knowledge of its by communicating with its immediate neighbors, without own structure and is able to establish alternative routes to being a burden to the centralized con®guration server. On deliver the streams to its users. Furthermore, whenever the other hand, the con®guration server should maintain possible, the system performs these recon®gurations with- global knowledge about the system topology. This centra- out affecting the QoS perceived by its users. lized, global knowledge should be used not only as backup, in case the Re¯ector's localized information is not enough to keep the system functioning, but also to perform optimi- 4.3.1. Fault-recovery models zations such as dynamic changes in the network topology to improve the quality of service and promote load balancing. In order to build an alternate distribution topology, the Our architecture adopts the hybrid model described system must store enough information so that alternate above. It distributes the knowledge throughout the Re¯ector routes can be found when failures occur. The question is: network and makes each Re¯ector aware of its dependence where to maintain this information? We considered, initi- relationships with other Re¯ectors. Thus, the Re¯ectors are ally, a solution in which all the information about fault able to make recon®guration decisions by themselves, with- recovery would be placed in a centralized database acces- out relying on a centralized entity. In addition to this, the sible by every Re¯ector. When a Re¯ector R1 detects that global system topology is maintained in the con®guration one of its inputs failed or that it is silent for too long, it service so that a Re¯ector administrator or an ªintelligentº would contact a con®guration server with access to the software module can perform global optimizations in the centralized database and request the address of a re¯ector distribution network. R2 from which it could receive the missing streams. The Each Re¯ector contains an object of the type Component- con®guration server would return this information and Con®gurator [16] which stores a list of entities on which the contact R2, recon®guring it to send data to R1. The advantage Re¯ector depends (its source Re¯ectors) as well as a list of of this approach is that very little information is stored on entities that depends upon the Re¯ector (client Re¯ectors or the Re¯ectors, all the fault-recovery information and end-user clients). In addition, a ComponentCon®gurator policies are centralized in a single location, facilitating the may also contain a list of alternatives, which could serve manipulation of this information by a single entity: the as alternate inputs for the multimedia streams in case the con®guration server. input for a given channel becomes silent. The second solution is to store fault-recovery information in the Re¯ectors. That is, each Re¯ector would store a list of alternate Re¯ectors that could be used as backups. The 5. Implementation advantage of this approach is that it does not lead to a single point of failure and does not impose an extra load on a We implemented a prototype of the architecture possibly already overloaded con®guration server. This solu- described in the previous section as a CORBA-based frame- tion may be more dif®cult to implement but it tends to be work, which brings two immediate bene®ts. First, we are more scalable. able to utilize a number of standard CORBA services imple- We believe that the optimal solution to this problem is mented by the CORBA community such as the Naming one that encompasses both models. On the one hand, each Service and interact with existing CORBA tools. Second, Re¯ector should have some knowledge about its local we have implemented our mechanisms for automatic surroundings and should be able to perform recon®gurations con®guration and dynamic recon®guration as CORBA 112 F. Kon et al. / Computer Communications 24 (2001) 105±123

Fig. 6. (a) Screen shot of the Name Service GUI. (b) Agents GUI. services exporting well-de®ned IDL interfaces. This makes and 2 in Fig. 5). From the con®guration service, it retrieves them easily accessible to other systems and reusable in other its speci®c con®guration information, which may contain contexts. three kinds of speci®cations: (1) the components that must be dynamically loaded to build this instance of the Re¯ec- 5.1. Implementing automatic con®guration tor; (2) the amount of physical memory and the share of the CPU that should be reserved for this instance of the Re¯ec- The ®rst major change we had to do in the Re¯ector tor; and (3) the input and output connections to be created at implementation to accommodate the new design was the startup. adoption of a component-oriented model. We reorganized The Re¯ector uses the information of the ®rst kind (which the implementation of the Re¯ector program, breaking it is obtained in step 3) to contact our CORBA Component into dynamically loadable components. Surprisingly, this Repository and fetch the code implementing the desired proved to be not so dif®cult, thanks to the original object- components3 (step 4). It then dynamically links these oriented design of the Re¯ector that was based on loosely components into the runtime system (step 5). If con®gura- coupled objects interacting via well-de®ned interfaces. This tion speci®cations of the second kind are present, they are component-based model facilitates the customization of the fetched in steps 6 and 7. Next, the Re¯ector contacts the under- Re¯ector program at startup, allowing the system to select lying middleware to request the reservation of the required the components that are best suited for a speci®c environ- resources, which is achieved with the help of the Dynamic ment at a certain time. It also facilitates the dynamic recon- Soft Real-Time Scheduler (DSRT) [20]. ®guration of running Re¯ectors to adapt to changes in the After loading all the required components, the Re¯ector environment and to install new versions of components on- registers itself with the Name Service, so that it can be easily the-¯y. located by other system entities, and opens all the input and Fig. 5 presents a schematic overview of the Re¯ector bootstrapping process in which the Re¯ector con®gures itself. At startup time, each Re¯ector contacts the CORBA 3 To minimize startup time and network load, the components fetched Name Service to locate the con®guration service (steps 1 from the Component Repository can be cached locally. F. Kon et al. / Computer Communications 24 (2001) 105±123 113 output connections using the speci®ed protocols (steps 9 and windows (see Fig. 6(b)) allows the administrator to draw the 10). vertices and edges of a directed graph that speci®es the Administrators of the Re¯ector system can look at the collection of Re¯ectors that will receive the new code and available broadcast and videoconference sessions by using the path through which the code will be transmitted. Each a graphical user interface that interacts with the CORBA node in Fig. 6(b) refers to a Re¯ector running on a different Name Service. Fig. 6(a) is a screen shot that exempli®es location. The directed edges specify the paths the agents the use of this interface. It shows three independent Re¯ec- traverse. Using the GUI, the Re¯ector administrator can tor networks: the ®rst called VirtualMeetingRoom reserved send state inspection or recon®guration commands to for audio and videoconferencing, possibly divided accord- subsets of the Re¯ector network. The operation results are ing to interest groups; the second called News that could combined and returned to the administrator who can verify contain several news channels; and the third called Olym- the results of all operations in the Re¯ectors and take care of picGames that could contain audio and video broadcast individual failures, if any. channels related to Olympic events. The administrator can also see that there are three avail- 5.3. Supporting fault-tolerance able component repositories for three different kinds of architectures and can use the GUI to upload, download, The architecture supports fault-tolerance by using and remove Re¯ector components from the repositories. ComponentCon®gurator objects to represent the dependen- Finally, the administrator can see that the OlympicGames cies between Re¯ectors. When failures occur, the system network is composed of the ®ve re¯ectors shown in the uses the dependence information to locate alternate routes upper right-hand window. The CORBA IOR (Interoperable and keep the system functioning. The ComponentCon®- Object Reference) of the Re¯ector at delirius.cs.uiuc.edu is gurator implementation stores the dependencies as a list shown in the bottom right-hand window. of CORBA IORs, which allows for prompt communication no matter where the objects are located. A subclass of ComponentCon®gurator, called Re¯ector- 5.2. Mobile recon®guration agents Con®gurator, contains the policies for reshaping the To support ef®cient code distribution and the dynamic network topology in case of failures and encapsulates all recon®guration of large-scale Re¯ector networks, we use a the code to deal with these recon®gurations. This approach generic middleware infrastructure for mobile agents [24]. proved to be very effective in keeping a clear separation of This infrastructure is based on dynamic TAO [25], an concerns in the Re¯ector code. The classes that deal with the enhanced CORBA Object Request Broker (ORB) that Re¯ector's normal operation are totally unaware of the exports a recon®guration interface that can be used to (1) Re¯ectorCon®gurator and of any code that deals with upload executable code to remote ORBs, (2) dynamically recon®guration. This clean separation also makes it easy load components into the system runtime, (3) change the to plug different kinds of Re¯ectorCon®gurator to support con®guration parameters of application components, and different recon®guration policies. (4) modify the internal architecture of component-based applications. 5.3.1. Triggering recon®gurations The infrastructure includes the support for recon®guration agents which work as ªsmart capsules'' that traverse a Four kinds of events can trigger dynamic recon®guration: network of distributed ORBs speci®ed by the application administrator. These agents contain a directed graph repre- 1. A Re¯ector shutdown message sent by the Re¯ector senting the order in which the ORBs are to be visited and a administrator or a kill command executed by the local script (or Java program) specifying recon®guration or system administrator. inspection commands that are processed by each of the 2. Software errors that lead to a segmentation fault or bus ORBs in their path. The results are collected and returned error. to the application administrator by following the reverse 3. A recon®guration order sent by the Re¯ector administra- direction in the directed graph. tor. This infrastructure was all we needed to solve the two 4. Sudden machine crashes or network disconnections. initial problems (see Section 3.3) in the previous Re¯ector version. The recon®guration interface supports the opera- In the ®rst two cases, the Re¯ector captures those events tions upload_implementation and con®gure_implementa- using signal handlers installed with the UNIX signal function, which can be used for code distribution and for tion or the Windows SetConsoleCtrlHandler function. In sending con®guration messages to application components, the UNIX implementation, for example, the administrator respectively. can kill a Re¯ector by pressing Ctrl-c on the terminal Using a graphical user interface, the Re¯ector adminis- executing the Re¯ector, by sending a shutdown message trator can build a recon®guration agent that carries new to the re¯ector using telnet, or by using the kill command. implementations of Re¯ector components. One of the GUI The Re¯ector captures the events generated by Ctrl-c, kill, 114 F. Kon et al. / Computer Communications 24 (2001) 105±123

Reflector Server Reflector Client A D Reflector C Reflector Reflector Server B E Client

(a)

Reflector Reflector Server A D Client

Server Reflector Reflector Client B E

(b)

Fig. 7. (a) Network with 5 Re¯ectors. (b) After Re¯ector C is killed. segmentation faults, and bus errors by implementing signal Re¯ectors in the list of clients of the ®nishing Re¯ector to handlers for the SIGINT, SIGTERM, SIGSEGV, and its output list. SIGBUS signals, respectively. Fig. 7(a) shows a sample Re¯ector network where Re¯ec- In the third case, the Re¯ector contacts the con®guration tor C has two inputs and two outputs. When C is killed and service to retrieve its new con®guration information and the recon®guration process described above completes, the reprocesses it, recon®guring its input and output connec- new con®guration becomes the one in Fig. 7(b). In order to tions. Finally, the fourth case is the only one in which it is be able to carry out the recon®guration without any glitches not possible to keep the client multimedia stream uninter- in the multimedia streams and without affecting the system rupted without relying on redundant streams to the same quality of service, we had to adopt a multi-threaded solution Re¯ector. The solution in this case is to detect when the which we describe in Section 6.3. input for a given channel has failed or has been silent for too long and then locate an alternative input either by using the local list of alternatives or by contacting the con®gura- 5.3.3. The recovery process tion server. As described below, our current implementation focuses on supporting dynamic recon®guration in the When a Re¯ector starts its execution for the ®rst time or presence of the ®rst two kinds of events. when it is restarted after being shutdown for some reason, it executes an initialization process. In this process, in addition to performing the actions described in Section 5.1, it 5.3.2. The recon®guration process performs the following three actions:

When an administrator (or other system entity) requests 1. Registers the Re¯ector with the Name Service. to kill a Re¯ector, the system executes a special event hand- 2. Using CORBA, sends a STARTED event to the Re¯ec- ler called abandonRe¯ectorNetwork. This handler takes the torCon®gurators of all the clients (outputs) of this following three actions. Re¯ector; the event carries a list of the sources of the new Re¯ector. 1. Unregisters the Re¯ector from the Name Service. 3. Sends a STARTED event to the Re¯ectorCon®gurators 2. Using CORBA, sends a FINISHED event to the Re¯ec- of all the sources (inputs) of this Re¯ector, carrying a list torCon®gurators of all the sources (inputs) of this Re¯ec- of its clients. tor; the event carries a list of the clients of the ®nishing Re¯ector. Upon receiving a STARTED event from a new source 3. Sends a FINISHED event to the Re¯ectorCon®gurators Re¯ector, the client Re¯ector opens a new input connection of all the clients (outputs) of this Re¯ector, carrying a list to the new Re¯ector. If it is also receiving input from one of of its sources. the sources of the new Re¯ector, it closes that input connection as soon as the data from the new source is available. An When a Re¯ectorCon®gurator receives a FINISHED event analogous process happens upon receiving a STARTED from a source Re¯ector, it adds all the Re¯ectors in the list event from a new client Re¯ector. These mechanisms of sources of the ®nishing Re¯ector to its list of inputs. allow the distribution system to recover its original topology Conversely, when a Re¯ectorCon®gurator receives a after a faulty re¯ector restarts. Therefore, if the system FINISHED event from a client Re¯ector, it adds all the con®guration is the one in Fig. 7(b) and the Re¯ector C F. Kon et al. / Computer Communications 24 (2001) 105±123 115 ran our TVStation application in three SPARCstations 20 running Solaris 5.5 and using the SunVideo interface for video capturing. Using an H.263 software encoder, the TVStation was able to generate an average of three frames per second while still allowing the machines to be used as personal workstations. For more than one month without interruption, the machines sent live video from our of®ces to a small Re¯ector network. Users could receive the video by pointing their Web browsers to our personal home pages on the web. An applet implementing an H.263 decoder was automatically loaded by the web browser and was able to receive the video even over standard telephone lines. Since H.263 decoding is much less computationally intensive than encoding, a simple 133 MHz PC was able to decode and display the three video sessions simultaneously, totaling more than nine frames per second. Although rather small in the number of Re¯ectors and bandwidth, this initial experiment showed that it was possible to keep a Re¯ector network running for several weeks without any interruption. Our technology was later chosen to broadcast, live over the Internet, the NASA JPL Mars Path®nder mission [13,5] 24 h a day from July to October 1997. The capture station at NASA's Jet Propulsion Laboratory was composed of a Fig. 8. The TV client Applet. 133 MHz Pentium-based PC equipped with an Osprey- 1000 card for capture and encoding H.263 video. The data recovers, then the con®guration automatically switches rate was set to 24 kbps so the stream could be received by back to the one in Fig. 7(a). 28.8 kbps modems. It contained half-rate GSM audio and Note that we do not have an automatic mechanism for 3±5 frames per second of H.263 video, depending on the restarting faulty Re¯ectors. This requires the administrator's encoded images. Fig. 8 shows the TV client applet running intervention, which seems to be natural since a Re¯ector goes within Netscape. out of service either because of an administrator's command or In order to carry out a broadcast of such an enormous because of a failure in the system. In both cases, the adminis- magnitude and avoid the complete collapse of our networks, trator's attention is advisable. Alternatively, if desired, the we requested collaboration from other institutions. In a few Re¯ector can be added to the list of daemons that are executed days, we were able to establish a global network utilizing when a machine boots, eliminating the need for manual inter- resources from several universities, research laboratories, vention when a machine crashes and restarts. space agencies, Internet service providers, computer manu- facturers, and mass media corporations. 6. Performance evaluation Our group organized a distribution tree composed of more than 30 Re¯ectors spread across ®ve continents. Fig. In this section, we describe the experiments we performed 1 depicts a network similar to the one we utilized. ªMaster'' with the Re¯ector system in the course of the last four years re¯ectors received data from the capture station and and draw conclusions about its performance. The experiments forwarded them to ªintermediate'' Re¯ectors across the are divided into the following groups. (1) Evaluating the globe. These Re¯ectors forwarded the data to ªpublic'' system's reliability and usability in large-scale, wide-area Re¯ectors which were associated with Web pages and broadcasts. (2) Measuring the capacity of a single Re¯ector accepted connections from end-user applets. This scheme with respect to the maximum number of clients and maximum provided live audio and video to dozens of countries around bandwidth it can support. (3) Evaluating the impact of recon- the globe. Each of the Re¯ector sites was capable of serving ®guration of the network topology on the QoS perceived by the from 30 to 300 simultaneous clients depending upon the end-users. (4) Measuring the performance improvements bandwidth of its connection to the Internet. This network obtained by using mobile agents for code distribution and could potentially serve more than 3000 simultaneous clients recon®guration of Re¯ector networks. 24 h a day. During the initial four days of the broadcast, the Re¯ector 6.1. Large-scale, wide-area broadcast network was able to deliver half a million multimedia streams. Within two weeks, usage climbed to more than In March 1997, we performed the ®rst experiments with one million video and audio sessions. After the ®rst few the initial implementation of the Re¯ector. At that time, we weeks, interest in the Path®nder mission decreased, but 116 F. Kon et al. / Computer Communications 24 (2001) 105±123

Reflector Reflector Reflector TestServer TestClient A B C

Fig. 9. Recon®guration experiment testbed.

the broadcast of the NASA Select TV over the Internet number of Re¯ectors from a single point. We developed continued for several months. the mechanisms for automatic recon®guration of the During the course of the broadcast, some recon®guration Re¯ector network in case of failures to address this exact of the network topology was required in order to accommo- problem. Even with this automated facility, it is advisable to date changes in the availability of Re¯ector hosts. We had to divide the Re¯ector network in collaborating groups of redirect streams to different primary Re¯ectors and often 100±200 Re¯ectors, each group being managed by a differ- add or remove public Re¯ectors. All these modi®cations ent administrator. were performed remotely without stopping the transmission. 6.3. Recon®guration impact on the QoS 6.2. Single re¯ector capacity More recently, we conducted experiments to evaluate the The only bottleneck in the NASA experiment was the band- impact of the automatic recon®guration of the Re¯ector width of Internet connections on the sites hosting Re¯ectors. In network on the quality of the multimedia streams received order to guarantee a good quality of service to our clients, we by the end-users. We carried out this set of experiments on were forced to deny approximately one million connection seven Sun Ultra machines (two Ultra-60 and ®ve Ultra-5) requests. During a different broadcast (featuring the Indy500 connected by a 100 Mbps Fast Ethernet. As depicted in Fig. race), a single Re¯ector running on a Sun Ultra 2 was capable 9, three of these machines executed our Re¯ector program. of serving nearly 800 simultaneous clients. This showed that a The fourth machine executed TestServer, a program we much larger number of users could be serviced if enough created to synthesize an RTP stream with bandwidth and bandwidth were available. packet rate speci®ed in its command line. A ®fth machine Recent experiments in our laboratory show that a Re¯ec- executed TestClient, a program that receives the packets tor running on a Sun Ultra-60 with Solaris 7, two 360 MHz from a Re¯ector and logs the packet arrival times into a processors and 1280 MBytes of RAM requires less than ®le. The sixth machine executed the CORBA Name Service 30% of one of the processors' time to stream a 26 kbps and the last one, the con®guration service and component video to 1020 simultaneous clients using TCP over a repository. 100 Mbps Fast Ethernet. The bottleneck in this experiment As explained in Section 5.3, when the Re¯ector B goes is the maximum number of sockets that a Solaris process is down, the system recon®gures itself automatically and allowed to open in our test machine: 1024. In machines that adopts the new topology shown in Fig. 10. When the Re¯ec- execute Solaris 7 in 64-bit mode, this limitation no longer tor B recovers, the system recon®gures itself back to the exists. The same machine transmitting a 1 Mbps stream to topology shown in Fig. 9. We evaluated the impact of 85 clients, uses less than 10% of one of its processors' time. these recon®gurations on the QoS perceived by the end- The bottleneck in this case is the network which is not able users by using the information in the log ®le to compute to sustain a TCP payload data rate of more than 85 Mbps. the packet inter-arrival time at the TestClient. The following Assuming that enough machines with high-bandwidth ®gures plot packet inter-arrival time (in seconds) over time connections were available, we could use the software tech- (in seconds). Each experiment lasted for 16±18 s. nology described in this article to build, for example, a Fig. 11 shows the packet inter-arrival times at our testbed network of 300 Re¯ectors serving 1000 clients each, total- client when no recon®gurations take place and with the ing 300,000 simultaneous unicast clients. Since the Re¯ec- TestServer sending an RTP/UDP stream of 1.2 Mbps at 30 tor also supports IP-multicast, the actual number of clients packets per second. The TestServer program uses an adap- could be much larger. tive algorithm to keep its average output bandwidth as close With large networks composed of more than 100 Re¯ec- as possible to the output bandwidth speci®ed at its command tors, the problem becomes the manageability of such a large line. Once every K seconds, the program checks its output

Reflector Reflector Reflector TestServer TestClient A B C

Fig. 10. Recon®guration when Re¯ector B goes down. F. Kon et al. / Computer Communications 24 (2001) 105±123 117

Fig. 11. No recon®gurations. Bandwidth 1.2 Mbps, packet rate 30 pps. bandwidth and modi®es its packet inter-transmission time to the Re¯ector's sources and destinations to announce the end adapt to the changes in its measured output bandwidth. The of the service, the old thread continues to perform the packet size is ®xed throughout each experiment and it is Re¯ector's normal packet-forwarding operations. The computed by dividing the bandwidth by the packet rate. Re¯ector only shuts itself down after it receives a con®rma- In our ®rst experiments, we set K 1 and the desired tion from its neighbors that its service is no longer needed. If packet rate to 30 packets per second. The TestServer, this con®rmation does not arrive, the Re¯ector administrator then, tries to send one packet every 33.3 ms. But, since still has the chance to kill the Re¯ector anyway at his or her the Solaris clock tick is set to 10 ms, it does not let a own discretion. Alternatively, the Re¯ectorCon®gurator can program sleep for exactly 33.3 ms. Hence, our adaptive be programmed to timeout after a given period so that one program ends up sending one packet every 30 or 40 ms does not need to rely on administrator intervention. which explains most of the variations throughout the experi- One can observe another problem in the graph of Fig. 12: ment in Fig. 11. The small and the localized variations (such as the Re¯ector recovers, there is a sudden change in the as the one at 8.5 s) are due to changes in the network and packet inter-arrival time, which approaches zero for a machine loads caused by other applications. couple of packets in a row. This happens because the end- Fig. 12 shows the impact of recon®gurations in our ®rst user receives some repeated packets that are sent both by the implementation of the fault-tolerance mechanism. The ªbackup'' connection being deactivated and the new, continuous arrows point to the instants in which the Re¯ec- ªrecovered'' connection. This problem of duplicated pack- tor B is killed and the Re¯ector network topology switches ets is solved very easily by implementing a new subclass of from the con®guration in Fig. 9 to the one in Fig. 10. The Connection that uses the RTP sequence number to drop dashed arrows point to the instants in which the Re¯ector B repeated packets. recovers and the topology switches back to the one in Fig. 9. Fig. 13 shows the result of running the same experiment One can see that when we killed the Re¯ector, there was a after implementing these two improvements. One can see big peak in the inter-arrival time at the client. This happened that the recon®gurations take place without affecting the because, in that implementation, as soon as the Re¯ector QoS perceived by the end-user (there is no correlation received the termination signal, it stopped forwarding pack- between the arrows in Fig. 13 and the variations in packet ets and sent events to its neighbors announcing that it was inter-arrival times). Furthermore, since the old Re¯ector going to shutdown. This caused a delay until the other keeps a thread performing the Re¯ector's normal operations Re¯ectors were able to recon®gure their inputs and outputs. until the recon®guration is completed, the quality of service In a wide-area network, the delays would be even larger and is not degraded even in wide-area networks where the the QoS perceived by the end-user would be degraded. latency to contact the neighboring Re¯ectors may be large. We solved this problem by creating a new thread to The lack of any correlation between the Re¯ector manage the recon®guration. While the new thread contacts recon®gurations and the jitter (i.e. the variance in the packet 118 F. Kon et al. / Computer Communications 24 (2001) 105±123

Fig. 12. Re¯ector recon®gurations. Bandwidth 1.2 Mbps, packet rate 30 pps.

Fig. 13. Re¯ector recon®gurations. Bandwidth 1.2 Mbps, packet rate 30 pps. F. Kon et al. / Computer Communications 24 (2001) 105±123 119

Fig. 14. High network and CPU load. Bandwidth 1.2 Mbps, packet rate 30 pps. inter-arrival time) is even more clear in Fig. 14 that shows a used the Re¯ector network to distribute an RTP stream similar experiment carried out in a period of high network generated by the vat audioconference tool for the MBone. and machine loads. In this case, we set K to 10 so that the The acoustic perception of the users listening to audiocon- TestServer would keep its transmission rate constant for ference con®rms what one sees in the graph: the recon®- longer periods. gurations do not degrade the quality of service of the Finally, Fig. 15 shows one more experiment in which we audioconference.

Fig. 15. Re¯ector recon®gurations. vat bandwidth 45 kbps, packet rate 25 pps. 120 F. Kon et al. / Computer Communications 24 (2001) 105±123

40 distribution tree 35 point to point

10 RoundTrip Time (s) 5

0 0.5 1 2 4 8 16 32 64 128 256 Component Size (KBytes) (a) 1.7 distribution tree 1.6 point to point 1.5 1.4 1.3 1.2 1.1 1 0.9 0.8 RoundTrip Time (s) 0.7 0.6 12345678 (b) Number of Commands in Agent Fig. 16. (a) Uploading component to 9 nodes. (b) Agent visiting 9 nodes.

6.4. Code distribution and dynamic recon®guration with network. To avoid drastic oscillations in the available Inter- mobile agents net bandwidth and latency, and to minimize undesired inter- ference, we carried out the experiments during the night4. In order to evaluate the response time and relative perfor- We measured the average bandwidth between our labora- mance gains made possible by our mechanisms for Re¯ector tory in Illinois and the remote ones by transferring a code distribution and recon®guration based on mobile 100 Kbytes ®le via FTP ®ve times and measured the latency agents, we established an intercontinental testbed with the by using the ping command. The average bandwidth and collaboration of the Departments of Computer Science at round-trip latency between our lab at cs.uiuc.edu and the Rey Juan Carlos University in Spain and at the Campi- the nodes at escet.urjc.es were 76 KBps and 170 ms, nas State University in Brazil. The testbed consisted of the respectively. Between our lab and ic.unicamp.br, following three groups of machines. (1) Two Sun Ultra-60 32 KBps and 270 ms, respectively. and one Sun Ultra-5 machines running Solaris 7 in the To measure how the bene®ts of using a distribution tree cs.uiuc.edu domain, (2) three 333 MHz PCs running similar to the one in Fig. 6(b) varies with the size of the Linux RedHat 6.1 at escet.urjc.es, and (3) three agent, we sent a series of agents carrying the code for 300 MHz PCs running Linux RedHat 6.1 at ic.uni- components to be installed in the remote nodes. As shown camp.br. The machines inside each group were connected in Fig. 16(a), as the size of the component being uploaded by 100 Mbps Fast Ethernet networks while the groups were connected among themselves through the public Internet. 4 When we ran the experiments during times of high network traf®c and We executed nine instances of our middleware, one in congestion, the performance numbers were even more in favor of our each node, and injected different kinds of agents in this mobile agents approach. F. Kon et al. / Computer Communications 24 (2001) 105±123 121 increases, the relative gain of using a distribution tree evolution of the communication mechanisms with respect instead of point-to-point connections to each node increases to QoS, security, and reliability. signi®cantly. Fig. 16(b) shows the elapsed time for execut- Amir, McCanne, and Katz developed an active service ing agents carrying from one to eight recon®guration framework [26] that provides support for the dynamic commands. Each point in the ®gures represents the arith- instantiation of server agents, providing the desired service, metic mean of 10 runs of each experiment. The vertical bars in a collection of distributed nodes running their host represent the standard deviation. manager (HM) daemon. They used this framework to imple- These results demonstrate clearly that our approach for ment a Media Gateway (MeGa) service [2]. As our Re¯ec- code distribution and recon®guration based on mobile tor, the MeGa service can be used to transcode multimedia agents can provide extreme performance improvements streams. Indeed, their research has focused on algorithms and better predictability in the management of wide-area for ef®cient transcoding and down-sizing and, more distributed QoS-sensitive systems. We expect it to be of recently, on adaptive bandwidth allocation algorithms and great help in the management of our multimedia distribution on the dynamic instantiation of media gateways between system. MBone sessions to deal with heterogeneous networks. Our research on the Re¯ector architecture shares some concerns 6.5. Summary with their active service framework (which has some simi- larities with our automatic con®guration service) and with The experiments described in this section let us draw the their MeGa service (which could be implemented in our following conclusions about the Re¯ector system. architecture with customized Connection classes loaded into the Re¯ector). But, differently from them, we focus 1. It provides a good level of scalability for multimedia on providing scalable multimedia distribution using multi- distribution over wide-area networks and is able to ple communication protocols, recon®guration of the keep the service running for several months without network topology to deal with failures, and scalable code interruption. distribution and dynamic recon®guration. 2. It supports, at least, 1000 unicast clients per Re¯ector In the last few years, the ®rst commercial Re¯ector-like depending upon the available network bandwidth. In systems for multimedia distribution started to appear (see, our Ethernet testbed, each Re¯ector is able to handle for example, VTel's Turbo Cast Re¯ector [27] and White streams with a TCP payload data rate up to 85 Mbps. Pine's Multipoint Control Unit for Meeting Point [28]). But, 3. Its mechanism for dynamic recon®guration of the distri- unfortunately, literature on this topic is still scarce. Baldi, bution network to deal with Re¯ector failures is ef®cient Picco, and Risso designed a videoconference system based and does not impact the quality of service as perceived by on active networks [29]. Their proposed architecture allows its end-users. clients to customize their videoconference server (or Re¯ec- 4. Its mechanisms for scalable code distribution and tor) by uploading mobile Java code. The main difference is dynamic recon®guration based on mobile agents provide that, in their approach, the Re¯ector is designed to run in signi®cant performance improvements when compared active network routers while ours is designed as a user-level to traditional client/server mechanisms. application to be executed on a networked workstation. We chose to implement our system in C11 because our experience shows that Java is not yet ready to provide the high-performance and predictability that most QoS- 7. Related work sensitive applications require. Active networks or programmable networks may potentially provide a better throughput Our research bene®ts from and builds on previous work and less jitter, since routing decisions are performed at on IP-Multicast [9] and the Internet MBone [1]. The MBone specialized routers rather than on commodity workstations. is a virtual network that is layered on top of the Internet to However, active and programmable networks are not avail- support routing of IP-Multicast packets. It is composed of able except for in a few research laboratories. It is not yet islands that can directly support IP-Multicast linked by clear if these technologies will become widespread in the virtual point-to-point links called tunnels. The system next years. described in this article can be seen either as extending the MBone capabilities or (since we also support IP- Multicast), as a set of tools for managing MBone broad- 8. Conclusions and future work casts. The MBone relies on a multicast distribution engine that is hard-wired into the networking infrastructure, Multimedia applications dealing with large-scale, real- making it dif®cult to deploy new technologies for distribu- time streaming will become commonplace in future compu- tion. Our approach not only uses user-level entities that can ter environments. The existing hardware and software infra- be replaced easily, but it also supports dynamic recon®gura- structure, however, does not provide the high degree of tion of running systems, allowing for easy incremental quality of service, ¯exibility, con®gurability, and scalability 122 F. Kon et al. / Computer Communications 24 (2001) 105±123 required by these applications. Using our ®ve-year experi- video, in: WWW Consortium Workshop on Real Time Multimedia ence with Internet multimedia streaming technology, we and the Web, Sophia Antipolis, 1996. have designed and implemented an architecture for scalable [5] F. Kon, R.H. Campbell, S. Tan, M. Valdez, Z. Chen, J. Wong, A component-based architecture for scalable distributed multimedia, multimedia distribution that meets these requirements. In in: Proceedings of the 14th International Conference on Advanced this article, we presented a detailed description of our Science and Technology (ICAST'98), Lucent Technologies, system and discussed how the lessons learned in early Naperville, 1998, pp. 121±135. experiments motivated us to implement the enhancements [6] R. Steinmetz, K. Nahrstedt, Multimedia: Computing, Communica- that led to the current architecture. tions, and Applications, Prentice-Hall, Englewood Cliffs, NJ, 1995. [7] OMG, CORBA v2.2 Speci®cation, Object Management Group, As a future improvement to the Re¯ector system, it would Framingham, MA, OMG Document 98-07-01, February 1998. be important to develop a service providing ªintelligentº [8] OMG, CORBAservices: Common Object Services Speci®cation, management of the data ¯ows among Re¯ectors and Object Management Group, Framingham, MA, OMG Document between Re¯ectors and end-user clients. In the current 98-12-09, 1998. implementation, a bandwidth control subsystem is used to [9] S. Deering, Host Extensions for IP Multicasting, RFC 1112, August 1989. measure the bandwidth utilization for each input and output [10] H. Schulzrinne, S. Casner, R. Frederick, V. Jacobson, RTP: a trans- connection. It could be easily extended to limit the number port protocol for real-time applications, revision of RFC 1889, of users who are allowed to access a particular Re¯ector at a January 2000. given time. With further research, we could also use the [11] P. Sinha, N. Venkitaraman, R. Sivakumar, V. Bharghavan, WTCP: a bandwidth control subsystem to (1) redirect multimedia reliable transport protocol for wireless wide-area networks, in: Proceedings of ACM Mobicom, Seattle, WA, 1999. streams to different paths in the network to optimize band- [12] M.C. Chan, A.A. Lazar, Designing a CORBA-based high perfor- width utilization and (2) to select which Re¯ector should be mance open programmable signalling system for ATM switching used by each client based on their location and on Re¯ector platforms, IEEE Journal on Selected Areas in Communications 17 (9). load. [13] M.P. Golombek, et al., Overview of the Mars Path®nder Mission and Another interesting future work would be the develop- assessment of landing site predictions, Science 278 (5344) (1997) 1734±1742. ment of a subclass of Connection, called RSVPConnection, [14] C.K. Hess, D. Raila, R.H. Campbell, Design and performance of that would reserve network bandwidth using the standard MPEG streaming to palmtop computers, in: Multimedia Computing RSVP protocol [30]. This seems to be a relatively easy task and Networking 2000 (MMCN00), San Jose, CA, 2000. if we reuse an existing middleware supporting RSVP such [15] J. Purtilo, R. Cole, R. Schlichting (Eds.), Fourth International Confer- as the one developed by the MONET research group at the ence on Con®gurable Distributed Systems, IEEE, 1998. [16] F. Kon, R.H. Campbell, Supporting automatic con®guration of University of Illinois. component-based distributed systems, in: Proceedings of the Fifth We believe that the architecture presented in this article USENIX Conference on Object-Oriented Technologies and Systems, has a great potential to in¯uence the next generation of San Diego, CA, 1999, pp. 175±187. scalable multimedia distribution systems. But, the methods [17] F. Kon, Automatic con®guration of component-based distributed we describe in this article apply not only to multimedia systems, PhD thesis, Department of Computer Science, University of Illinois at Urbana-Champaign, May 2000. systems. They give signi®cant insights on how to manage [18] S. Frùlund, J. Koistinen, Quality of service aware distributed object any QoS-sensitive distributed application with respect to systems, in: Proceedings of the Fifth USENIX Conference on Object- scalability, dynamic con®guration, and fault-tolerance. Oriented Technology and Systems (COOTS'99), San Diego, 1999, pp. 69±83. [19] J. Nieh, M.S. Lam, The design, implementation and evaluation of Acknowledgements SMART: a scheduler for multimedia applications, in: Proceedings of the 16th Symposium on Operating Systems Principles, ACM, Saint Malo, France, 1997. The authors gratefully acknowledge the work performed [20] K. Nahrstedt, H.hua Chu, S. Narayan, QoS-aware resource manage- by Miguel Valdez, Jim Wong, and See-Mong Tan in the ment for distributed multimedia applications, Journal of High-Speed early versions of the Re¯ector software. Chuck Thompson Networking, Special Issue on Multimedia Networking 7 (1998) 227± and Andrew MacGregor provided valuable system admin- 255. istration support. [21] D. Kotz, R.S. Gray, Mobile agents and the future of the Internet, ACM Operating Systems Review 33 (3) (1999) 7±13. [22] J. Bredin, D. Kotz, D. Rus, Market-based resource control for mobile agents, in: Proceedings of the Second International Conference on References Autonomous Agents, 1998, pp. 197±204. [23] I. Goldberg, D. Wagner, R. Thomas, E.A. Brewer, A secure environ- [1] H. Eriksson, MBone: the multicast backbone, Communications of the ment for untrusted helper applications: con®ning the Wiley Hacker, ACM 37 (8) (1994) 54±60. in: Proceedings of the USENIX Security Symposium, 1996. [2] E. Amir, S. McCanne, H. Zhang, An application-level video gateway, [24] F. Kon, B. Gill, M. Anand, R.H. Campbell, M.D. Mickunas, Secure in: Proceedings of ACM Multimedia, San Francisco, CA, 1995. dynamic recon®guration of scalable CORBA systems with mobile [3] Z. Chen, S.-M. Tan, R.H. Campbell, Y. Li, Real-time video and audio agents, in: Proceedings IEEE Symposium on Agent Systems and in the world wide web, in: Fourth International WWW Conference, Applications/Mobile Agents, Zurich, 2000. Boston, 1995. [25] F. Kon, M. RomaÂn, P. Liu, J. Mao, T. Yamane, L.C. MagalhaÄes, R.H. [4] S. Tan, R. Campbell, Z. Chen, W. Liao, D.K. Raila, F. Kon, M. Campbell, Monitoring, security, and dynamic con®guration with the Valdez, Adaptation and synchronization in low-bandwidth Internet dynamicTAO re¯ective ORB, in: Proceedings of the IFIP/ACM F. Kon et al. / Computer Communications 24 (2001) 105±123 123

International Conference on Distributed Systems Platforms and Open white paper available at http://www.wpine.com/meetingpoint, Distributed Processing (Middleware `2000), no. 1795 in LNCS. New 1999. York, 2000, pp. 121±143. [29] M. Baldi, G.P. Picco, F. Risso, Designing a videoconference system [26] E. Amir, S. McCanne, R. Katz, An active service framework and its for active networks, in: Proceedings of the Second International application to real-time multimedia transcoding, in: Proceedings of Workshop on Mobile Agents (MA `98), 1998, pp. 273±284. the ACM SIGCOMM Conference, Vancouver, BC, 1998. [30] R. Braden, et al. Resource ReSerVation Protocol (RSVP) Ð Version [27] VTel, TurboCast Architecture, http://www.turbocast.vtel.com, 2000. 1 Functional Speci®cation, IETF Proposed Standard, RFC 2205, [28] White Pine, Deploying H.323 Conferencing on Your IP Network, September 1997.