The following paper was originally published in the Proceedings of the USENIX Symposium on Technologies and Systems Monterey, California, December 1997

Salamander: A Push-based Distribution Substrate for Internet Applications

G. Robert Malan, Farnam Jahanian, and Sushila Subramanian University of Michigan

For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. : [email protected] 4. WWW URL:http://www.usenix.org/

Salamander: A Push-based Distribution Substrate



for Internet Applications

G. Rob ert Malan, Farnam Jahanian Sushila Subramanian

Department of EECS School of Information

University of Michigan University of Michigan

1301 Beal Ave. 550 East University Ave

Ann Arbor, Michigan 48109-2122 Ann Arbor, Michigan 48109-1092

frmalan,[email protected] [email protected]

Abstract 1 Intro duction

The availability of ubiquitous network connec-

The Salamander distribution system is a wide-

tions in conjunction with signi cant advances in

area network data dissemination substrate that has

hardware and software technologies have led to the

b een used daily for overayear by several group-

emergence of a new class of distributed applications.

ware and web casting Internet applications. Sp eci -

The Salamander data distribution substrate pro-

cally, Salamander is designed to supp ort push-based

vides the systems supp ort needed for two of these

applications and provides a variety of delivery se-

applications: groupware and web casting. Partic-

mantics. These semantics range from basic data de-

ipants in these wide-area distributed applications

livery, used by the Internet Performance Measure-

vary in their hardware resources, software supp ort

ment and Analysis IPMA pro ject, to collab ora-

and quality of connectivity[11]. In an environment

tive group used by the Upp er At-

such as the Internet they are connected by network

mospheric Research Collab oratory UARC pro ject.

links with highly variable bandwidth, latency, and

The Salamander substrate is designed to accom-

loss characteristics. In fact, the explosive growth of

mo date the large variation in Internet connectivity

the Internet and the proliferation of intelligent de-

and resources through the use of application-

vices is widening an already large gap b etween these

sp eci c plug-in mo dules. These mo dules provide a

memb ers. These conditions make it dicult to pro-

means for placing application co de throughout the

vide a level of service that is appropriate for every

distribution network, thereby allowing the applica-

memb er of a collab oratory or web casting receiver.

tion to resp ond to network and pro cessor resource

Web casting applications use push technology to

constraints near their b ottlenecks. The delivery

send data from network servers to client machines.

substrate can b e tailored by an application for use

In group collab oratories, p eople from a global p o ol

with a heterogeneous set of clients. For example the

of participants come together to p erform work or

IPMA and UARC pro jects send and receive data

take part in meetings without regard to geographic

from: Java applets and applications; Perl, C and

lo cation. A distributed collab oratory provides: 1

C++ applications; and Unix and Windows 95/NT

human-to-human and shared soft-

clients. This pap er illustrates the architecture and

ware to ols and workspaces; 2 synchronized group

design of the Salamander system driven by the needs

access to a network of data and information sources;

of its set of current applications. The main archi-

and 3 remote access and control of instruments for

tectural features describ ed include: the data distri-

data acquisition. A collab oratory software environ-

bution mechanism, p ersistent data queries, negoti-

ment includes to ols such as whiteb oards, electronic

ated push-technology, resource announcement and

noteb o oks, chat b oxes, multi-party data acquisition

discovery, and supp ort for Application-level Qual-

and visualization software, synchronized informa-

ity of Service p olicies.

tion browsers, and video conferencing to facilitate

e ectiveinteraction b etween disp ersed participants.



This work is supp orted by the National Science Foun-

Akey challenge for the designers of wide-area col-

dation under the co op erative agreement IRI-92-16848 and a

lab oratories is the creation of scalable distribution generous gift from the Intel Corp oration.

and dissemination mechanisms for the shared data. transparently connect di erent p ortions of a wide-

area application. This connectivityisachieved

The Salamander substrate provides supp ort for

through the use of a channel subscription service.

b oth web casting and groupware applications by pro-

Second, Salamander allows for the addition of appli-

viding virtual distribution channels in an attribute-

cation plug-in mo dules along the distribution path

based data space. In a Salamander system, a tree of

to allowforavariety of data mo di cations and deliv-

distribution no des servers can b e dynamically con-

ery decisions. The remainder of the pap er will fur-

structed to provide p oints of service into the data

ther describ e the Salamander substrate. Section 2

space. Clients can connect to this tree to b oth pub-

provides background material on Salamander's cur-

lish and subscrib e to data channels. The data is

rent applications. Section 3 enumerates the main ar-

pushed from suppliers to the clients through the dis-

chitectural features. Section 4 provides an overview

tribution tree. Opaque data ob jects are constructed

of the Salamander application programming inter-

by clients that are describ ed using text-based at-

face API as well as describ es its administration

tribute lists. Clients provide p ersistent queries to

and security features. Section 5 gives b oth a quali-

the Salamander substrate using attribute expres-

tative and quantitative p erformance evaluation. Fi-

sions that represent the data ows they wish to re-

nally, in Section 6 we conclude and describ e our cur-

ceive, thereby subscribing to a virtual data chan-

rent and future researchinterests.

nel. Salamander connections are rst-class ob jects

and are addressable if desired. This addressability

allows for feedback from subscrib ers to data pub-

lishers. Additionally, Salamander allows for plug-in

2 Application Domain

mo dules at any p oint in the distribution tree for ap-

plication co de to a ect the data distribution. These

The Salamander substrate currently supp orts two

plug-in mo dules provide the mechanism to supp ort

applications with diverse needs: the UARC and

application-level Quality of Service p olicies.

IPMA pro jects. The Upp er Atmospheric Research

Although the terminology is relatively new, Push

Collab oratory UARC[3, 17] is a distributed sci-

technologies have b een around for manyyears. The

enti c collab oratory over the Internet. The UARC

seminal push technology is electronic mail . Email

pro ject is a multi-institution research e ort, whose

has b een pushed from publisher to subscrib ers for

fo cus is the creation of an exp erimental testb ed

decades through mailing lists. Moreover,

for wide-area scienti c collab oratory work. The

news[9] has b een used to push data ob jects arti-

UARC system provides a collab oratory environment

cles based on text attributes group names. Ex-

in which a geographically disp ersed communityof

tending netnews, the SIFT to ol[18] has b een used

space scientists p erform real-time exp eriments at

to redistribute netnews articles based on text-based

a remote facilities, in lo cations such as Greenland,

user pro les. Atalower level the combination

Puerto Rico, and Alaska. Essentially, the UARC

of nativeIPmulticast supp ort and sp eci c Mb one

pro ject enables this group to conduct team science

[5] routers provides a mechanism for the push of

without ever leaving their home institutions. These

IP datagrams throughout p ortions of the Internet.

scientists p erform exp eriments on remote instru-

Multicast datagrams are pushed based on a single

ments, evaluate their work, and discuss the exp eri-

attribute, namely the IP multicast group address.

mental results in real-time over the Internet. This

Recently, many commercial push-based compa-

community of space scientists has extensively used

nies have started: BackWeb, IFusion, InCommon,

the UARC system for over three years; during the

Intermind, Marimba, NETdelivery,PointCast, and

winter months, a UARC campaign { the scientists

Wayfarer. These commercial ventures promise to

use the term campaign to denote one of their exp eri-

manage the complexity of the web by providing data

ments { takes place almost every day. This commu-

to users by means of subscription; similar to what

nity has grown to include regular users from such

current mailing lists provide, only more intrusively.

geographically diverse sites as: SRI International

In fact, many of these pro ducts are really pol l and

in Menlo Park, California; the Southwest Research

pul l instead of push. The clients in these systems

Institute; the Danish Meteorological Institute; the

p erio dically p oll the servers for new data, and then

Universities of Alaska, Maryland, and Michigan;

fetchitifitisavailable, reducing scalability.

and the Lo ckheed Palo Alto Research Lab oratory.

The UARC system provides a variety of services The Salamander substrate di ers from past tech-

to its users including shared synchronized displays nologies in several ways. First, it is not user-centric,

for instrument data, multipartychat b oxes, a shared but application-centric. Salamander is an under-

annotation database, and a distributed text editor. lying substrate that applications can plug into to

However, the primary mechanism for collab oration which provides a graphical lo ok at real-time condi-

is the real-time distribution of atmospheric data to tions across an instrumented network.

the exp eriment's participants. This data is collected

at remote sites such as Kangerlussuaq, Greenland,

and is distributed over the Internet to the scien-

ti c collab oratory using the Salamander substrate

describ ed in this pap er. Figure 1 shows several dif-

ferent data feeds displayed during a real-time cam-

paign.

Figure 2: Example screen grab from ASExplorer

session.

3 Architecture

The Salamander substrate's architecture can b e

Figure 1: Example screen grab from April 1997

describ ed in terms of its coarse-grained pro cesses,

UARC campaign.

channel subscription interfaces, and distribution se-

mantics. The key contributions of the architecture

The second pro ject that uses Salamander is the

are its:

Internet Performance Measurement and Analysis

IPMA pro ject[8 ], a joint e ort of the University

 Channel Subscription Service: The Sala-

of Michigan and Merit Network. The IPMA pro ject

mander substrate uses a publish/subscrib e ser-

collects a variety of network and interdomain p erfor-

vice that combines the strength of database re-

mance and routing statistics at Internet Exchange

trieval with a dynamic distribution mechanism.

Points IXPs, internal ISP backb ones, and campus

This service is used to provide a continuous ow

LAN/WAN b orders. A key ob jective of the IPMA

of data from publishers to subscrib ers.

pro ject has b een to develop and deploy to ols for

real-time measurement, analysis, dissemination and

 Application-level Quality of Service:

visualization of p erformance statistics. Two ma jor

Application-level quality of service p olicies are

to ols from the IPMA pro jects are ASExplorer and

supp orted that provide the ability to adapt

NetNow. They b oth use Salamander to web cast the

the delivery channels in resp onse to changes in

real-time data from the IPMA Web servers to their

client and network resources and subscriptions.

connected Java applets. ASExplorer is intended to

These p olicies are supp orted by the use of ap-

explore real-time autonomous system AS routing

plication sp eci c plug-in mo dules that can b e

top ology and instability in the Internet. It supp orts

used for data ow manipulation.

measurement and analysis of interdomain routing

statistics, including: route ap, growth of routing  Lightweight Data Persistence: Salamander

tables, network top ology,invalid routing announce- employs the use of a caching and archival mech-

ments, characterization of network growth and sta- anism to provide the basis for high-level mes-

bility. An example of the ASExplorer client is shown sage orderings. A two-tiered cache keeps cur-

in Figure 2. NetNow is to ol for measuring network rent data in memory while migrating older data

loss and latency. The NetNow daemon collects a to p ermanent storage. This storage takes the

variety of loss and latency statistics b etween net- form of a lightweight temp oral database tai-

work p eers. The NetNow clientisaJava applet lored to Salamander's needs.

A Salamander-based system is comp osed from used by Salamander to \lo okup" destinations for the

two basic units: servers that act as distribution ob ject. Subscrib ers place p ersistent queries to the

p oints and are usually collo cated with Web servers; Salamander substrate using lists of attribute expres-

and clients that act as b oth data publishers and sub- sions that can b e used to match b oth current and

scrib ers. These units can b e connected together in future ob jects published to the Salamander space.

arbitrary top ologies to b est supp ort a given applica- Alternatively, this pro cedure can b e thoughtofas

tion see Figure 3 for an example. The Salamander accessing a distributed database where the queries

is designed from a utilitarian p ersp ective, in are p ersistent. These p ersistent queries are matched

that it can stand alone, or like a software backplane by b oth the ob jects archived in Salamander's p er-

can b e multiplied to increase scalability. The cur- sistent rep ositories as well as future up dates to the

rentversion of the server is a POSIX thread imple- database space. These future up dates and additions

mentation on Solaris. Salamander clients can b oth are dynamically matched with outstanding queries

publish and subscrib e to virtual data channels. In that are then pushed to the queries' corresp onding

b oth the IPMA and UARC pro jects, the main data clients. The query asp ect of Salamander's attribute-

suppliers are written in either Perl or C; whereas based subscription service di ers those in traditional

the mainstay of the subscrib ers are Java applets. nameservice and database systems, in that instead

Applet development has progressed for over a year of the queries acting once on a static snapshot of

and a half on the UARC pro ject, and Web browsers the dataspace, they are dynamic entities that act

are the de facto subscrib er platform. In the absence on b oth the current state of the system and fu-

of multicast supp ort in Java 1.0.2, and the lackof ture up dates. Publishers may come and go with-

universal Mb one[5] connectivity, the decision was out a ecting the connection b etween the Salaman-

made to create our own distribution top ology using der database and the subscrib ers.

Salamander servers in place of existing Mb one in-

Salamander allows for feedback from subscrib ers

frastructure. With the adventofJava 1.1 we are

to their publishers in the form of negotiated push.

b eginning to implement the Salamander interface

This is accomplished though the combination of

using nativemulticast supp ort.

unique endp oint addresses, endp oint namespace

registration, and the ability to send unicast mes-

Saarullik (Greenland)

ween Salamander clients. Each Salaman-  sages b et



der connection is given a unique address that it can

Aricebo (Puerto Rico) SRI (California)

insert into a global namespace. Connections man- 

Denmark tries in this global namespace using the

 age their en

supply command. The supply command is given an

attribute list, similar to the one used in the query

command, that is paired with its identi er in the

ts can then nd entries in  Maryland namespace. Other clien

 y matching the attributes with a

 UARC Data Supplier NSF (Wash DC) the namespace b

supply query.Having obtained the identi er of an

UARC Server Michigan UDEMO (Florida)

endp oint, the connection can then send it a unicast UISR (Alaska)

UARC Client message. In practice, these sets of commands are

used to denote the availability of data or memb er-

ship in a group. In negotiated push, this mechanism

Figure 3: Example UARC campaign top ology used

can b e used to allow subscrib ers to ask publishers

during the April 1997 campaign

to b egin data distribution, or to mo dify a supplier's

data ows at the source. The UARC application

uses this mechanism to turn di erent data ows on

3.1 Channel Subscription Service

and o . These data ows are computationally ex-

p ensive, and should only b e generated when there

The Salamander substrate provides an abstrac-

is a demand for the data.

tion for the distribution of data from publishers

A noti cation service is also provided within the to subscrib ers through its channel subscription in-

Salamander namespace to allow for propagation of terface with b oth anonymous and negotiated push

various system events to endp oints. For example, techniques. The basic idea in anonymous push is

a connection can register a close event with their that publishers package opaque data ob jects, or Ap-

server that will cause the generation of a close no- plication Data Units as termed in [2], with text-

ti cation message to a sp eci ed endp oint. In this based attribute lists. These attributes can then b e

way, clients can maintain memb ership information so that it b est matches the client connection's ser-

within their groups. vice level. These quality of service parameters are

taken directly from the user in the UARC applica-

tion. When users discover that they are oversub-

3.2 Application-Level Quality of Ser-

scrib ed, they can use the interface to sp ecify prior-

vice

ities among the subscrib ed ows, and assign drop

p olicies for the individual ows. A skipover p olicy

The Salamander architecture provides

can b e used to sp ecify ne-grained drop orders. Cur-

application-level Qualityof Service p olicies to

rent skip over p olicies consist of b oth a FIFO thresh-

deliver data to clients as b est ts their connectivity

old, and a drop distance parameter. The threshold

and pro cessing resources[10 ]. These p olicies can

is used to allow for transient congestion; once this

rely on either b est e ort service or utilize network-

threshold of data has accumulated in the proxy, the

level QoS guarantees[19] if available. Application

drop distance parameter is used to determine which

sp eci c p olicies are used to allo cate the available

data are discarded, and which are delivered.

bandwidth b etween a client's subscrib ed ows, pro-

viding a client with an e ective throughput based

In addition to discrete delivery p olicies, the Sala-

on semantic thresholds that only the application

mander substrate provides for on-demand data

and user can sp ecify. These application-level QoS

degradation and conversion of data ob jects. In gen-

p olicies are achieved through the use of plug-in

eral, these mechanisms are used to convert one ob-

p olicy mo dules at p oints in the distribution tree.

ject into another. In order to supp ort real-time

Figure 4 shows an example top ology with several

collab oration b etween heterogeneous clients, some

typ es of mo dules.

mechanism for graceful data degradation must b e

made available to provide useful data to the slower

Data Suppliers: Client (Participant):

participants. At the application level, we under- tics of the shared Applet 1 Applet 2 stand something ab out the seman

Supply 1

data. We can exploit this knowledge, and pro- Server:  vide a graceful degradation of the data based up on

 tics. The Salamander substrate uses on-    these seman           demand lossless and lossy compression on seman-       

  

yp ed data, tailoring contents to the sp eci c   tically t  



ts of the clients. Lossless compression

Supply 2 requiremen

techniques are used for those data that cannot b e

 

hasraw text, binary executables, and

Discrete Delivery Policy Module degraded, suc  

 

terpreted program text. Lossy compression tech-  in    Degredation/Conversion Module 

 niques are applied to data that can su er some loss

y without losing their semantic information,

Intergration/Synchronization Module of delit

examples of which are: still and moving images; au-

dio; and higher level text like p ostscript or hyp ertext

Figure 4: Example collab oratory top ology with

markup languages. We give the application layer

plug-in mo dules at suppliers, servers and clients.

control over this qualityby providing an interface

The architecture allows for the placement of data

that adjusts the delity of the real-time data ow

pro cessing mo dules at any p oint in the distributed

on a p er-client basis.

datapath. This sp eci c example shows the path

data takes from two suppliers to two client applets.

Wewere surprised by our exp eriences with the

UARC pro ject, in discovering that the system b ot-

tleneckwas not network bandwidth, but was instead To illustrate the use of these mo dules, we use

the pro cessing p ower on the clients. The UARC sys- UARC as an example. Sp eci cally, the UARC ap-

tem uses sophisticated Java applets to pro cess raw plication uses discrete delivery, data degradation,

scienti c data as they arrive. During the April 1997 and data conversion plug-in mo dules. By multi-

campaign, the Salamander substrate could deliver plexing the subscrib ed ows, discrete delivery mo d-

the UARC data to a large numb er of clients with- ules can b e used to prioritize, interleave, and dis-

out diculty; however, the clients' Javainterpreters card discrete data ob jects. Wehave constructed a

couldn't keep up with the incoming data rate. Our exible interface that allows the client to b oth: de-

solution was to put plug-in mo dules in the distribu- termine its current p erformance level with resp ect

tion tree that could convert the data in-transit to a to the supply, and to gracefully degrade its qual-

more manageable format. ity of service from an application-level standp oint

Previous work has addressed the variabilityin technologies in the future.

client resources. Client resources are addressed in

In addition to p ersistent state, Salamander main-

the Odyssey[13] system by sending di erentversions

tains a memory cache used to bu er ob jects in-

of the same ob ject from the server dep ending on

transit through the system. Together, the memory

the client's resources. In [6], Fox et.al. provide a

bu er and database act as a two-level cache that

general proxy architecture for dynamic distillation

provides b oth high-level delivery semantics on the

of data at the server. The use of hierarchically en-

virtual ows and the ability to replay sections of

co ded data distributed over several multicast groups

ows from an archive. Salamander's virtual chan-

is discussed in [12] for the delivery of di erent qual-

nels in the ob ject space denote implicit groups. The

ities of audio and video data. Balachandran et.al

delivery semantics within these groups varies de-

target mobile computing in [1] and argue for adding

p ending on an application's needs. Groups can stay

active lters at base stations. Active network pro-

anonymous where the suppliers havenoknowledge

p onents [15] argue that Internet routers should b e

of the receivers, or they can b ecome more explicit

enabled to run arbitrary application-level co de to

where the suppliers keep a tally of their receivers. A

assist in proto col pro cessing. The contribution of

p ersistent disk-based cache of data ob jects, in con-

our work is the use of client feedback to allow for

junction with the use of application level framing[2],

prioritization among ows; the construction of ap-

can b e used to provide high-level delivery semantics.

plication and user interfaces for ow mo di cation;

This includes: FIFO, causal, ordered atomic, etc[7].

and the ability to place mo dules at any p ointinthe

distribution tree.

4 Salamander Interfaces

3.3 LightweightTemp oral Database

There are twointerfaces to the Salamander sub-

Salamander provides data p ersistence by incor- strate: the application programmer interface API

p orating a custom lightweight temp oral database. and the administration interface. The API has sev-

This database supp orts Salamander's needs by: eral layers and implementations, dep ending on the

storing a virtual channel's data as a sequence of

develop er's needs. At the lowest level, the substrate

write-once up dates that are primarily based on

provides a simple interface in b oth C and Java that

time; and satisfying requests for data based on tem- gives four primitive op erations: send, receive, con-

p oral ranges within the up date stream. A temp oral nect, and disconnect. These op erations are used

database [14] generally views a single data record in conjunction with a property list manipulation li-

as an ordered sequence of temp orally b ounded up-

brary to send and receive Salamander data ob jects.

dates. In the Salamander database these records

Much of this API can b e illustrated by the exam-

corresp ond to virtual channels. An administrator ple application co de in Figure 5. This example is a

can determine which sets of virtual channels will b e small subroutine that subscrib es to a simple virtual

archived by the system. The Salamander system ex- channel consisting of a single attribute.

p orts only a simple query interface to this database

The connect op eration is demon-

based on ranges of time and attribute lists. By fore-

strated on line 8 of the listing, where a

going the complexity of most commercial and re-

connectToSalamanderServer call is made. While

search temp oral databases, we could build a small

hostname is straightforward, the sskey parameter

and ecient custom database that met our simpler

requires some explanation. The sskey is a simple

needs.

form of access control to the Salamander substrate.

In the current implementation this key is very

Salamander's synergy b etween real-time data dis-

small, but one can imagine using a more robust key

semination and traditional temp oral and relational

in conjunction with a secure so cket implementation

databases is one of its signi cant contributions. It is

to achieve a greater level of con dence. Another

taken for granted that queries on a relational or tem-

security measure, implicit to the client, is an IP

p oral database act as a static atomic action against

access list that restricts the access of data channels

a snapshot of a system's dynamic data elements.

and administrative commands. This IP ACL is

Our mo del alters this, by providing supp ort for p er-

maintained on a p er server basis.

sistent queries that act over b oth a snapshot of the

data elements present in the database and anymod- After connecting to the substrate, the example

i cations real-time up dates to the database ele- then subscrib es to the virtual channel by creating a

ments that may o ccur in the future. We plan to fur- query and submitted it to the server. This query is

ther address the impact of this mo del on database constructed in lines 14 through 19. The prop erty list

1 void

subscribeToChannelchar * hostname, unsigned long sskey, char * queryName {

plist_t plist;

5 void * dataptr;

unsigned long data_length;

if connectToSalamanderServerhostname, sskey != SALAMANDER_OK {

fprintfstderr, "Connection Error.";

10 return;

}

/* Create the query. */

plist = createPropertyList;

15 updatePropertyplist, COMMAND_PROPERTY, QUERY_COMMAND;

updatePropertyplist, NAME_PROPERTY, queryName;

updatePropertyplist, COOKIE_PROPERTY, "queryCookie";

sprintftmpbuf, "RANGE d d", begin, end;

updatePropertyplist, TIMESTAMP_PROPERTY, tmpbuf;

20

/* Send it to the Server. */

if salamanderSendServerDataplist, NULL, 0 != SALAMANDER_OK {

fprintfstderr, "Error making query.";

return;

25 }

destroyPropertyListplist;

/* Read the responses as they come. */

for ;; {

30 if salamanderReadServerData&plist, &dataptr, &data_length != SALAMANDER_OK

break;

handleResponseplist, dataptr, data_length;

}

35

disconnectFromSalamanderServer;

}

Figure 5: Simple C example that connects to a Salamander server and makes a single p ersistent query.

1 ; 2

plist t is the data structure that contains a data riety of browsers . Empirically, the Salamander

ob ject's header and attribute information. Certain substrate is used by b oth the UARC and IPMA

attributes are considered wel l-known by the system. pro jects as their base middleware for daily op era-

An example, is the COMMAND PROPERTY shown in line tion. Salamander has b een used for overayear by

15. The command prop erty tells the substrate what the UARC scientists and has made a signi cant im-

to do with the data ob ject up on receipt. In the ex- pact in the space science research community. The

ample, the command is a query, whichisintercepted IPMA pro ject has b een using Salamander around

by the substrate and is pro cessed at the server. the clo ck for over nine months for a variety of tasks.

The NAME prop erty is the only mandatory query at- These tasks include: acting as the data collection

tribute in the current implementation. While any mechanism for the NetNow prob es at the Internet

numb er of attributes can b e used to describ e a vir- Exchange Points, and as the web casting delivery

tual channel, one of them must b e the NAME prop- mechanism from a central server tree to a collec-

erty. The COOKIE prop erty on line 17 is used to tion of Java applets available from its website[8].

match queries with a unique identi er that is re- The UARC pro ject has gone through several week-

turned by the Salamander substrate. This query long campaigns using Salamander as the data dis-

identi er is added to any ob ject that is returned tribution substrate that connects the remote instru-

by a subsequent receive op eration, and is used to ment sites to the scientists' Java applets. During

match resp onses with virtual channels. Finally,on these campaigns over sixty scientists have had mul-

lines 18 and 19, the wel l-known TIMESTAMP prop- tiple connections receiving many di erenttyp es of

erty is de ned that designates the range of data for data. The empirical results from these two pro jects

which the query corresp onds. The send op eration demonstrate that the Salamander system is b oth

is carried out on line 22, followed by the destruction extremely robust and scalable.

of the prop erty list. han-

After the routine subscrib es to the virtual c Server

nel, it go es into a lo op that reads data ob jects from

the substrate on lines 29 through 34. When the con-

nection is severed, or another error o ccurs, the co de

onnects and exits on line 36.

disc R R Clients

In addition to the base API, Salamander also ex-

p orts an administrativeinterface. The substrate

can b e administered b oth remotely from a higher

level API, and directly on the servers as con gura-

tion les. When used remotely, sp ecial commands

are sent to the server using the base API. Example

commands include: list current connections, list ac-

Figure 6: Exp erimental apparatus for Salamander

tive virtual channels, prune a connection, resize a

server p erformance tests.

channel's memory bu er, establish a server p eering

connection, mo dify debugging level, etc. A server's

A series of exp eriments were p erformed on a sin-

con guration le is used primarily to establish static

gle Salamander server to help quantify its p erfor-

server p eering relationships, to de ne default bu er

mance. For all of the following exp eriments, the

sizes, and sp ecify delivery semantics for sp eci c vir-

same setup was used. This corresp onds to the net-

tual channels.

work con guration shown in Figure 6. In this gure,

the single Salamander server, shown in the upp er

left corner, is a SUN Ultrasparc-1 with a 140 MHz

pro cessor and 192 Mbytes of memory. A 10 Mbps

5 Performance Evaluation

Ethernet segment connects the server to a router

1

Although counter-intuitive, cross browser compatibility

is not a given. There are signi cantvariations in JavaVM

The p erformance of the Salamander substrate can

implementationsbetween browsers, and it was common dur-

ing UARC applet development for co de that would work un-

be characterized by b oth empirical and exp erimen-

der Netscap e to break under IE or HotJava and

tal results. Salamander is currently implemented

vice versa.

2

asamultithreaded server on Solaris, and supp orts

Coincidentally, HotJava continues to b e the UARC de-

a set of complex Java applets that run on a va- velop ers' browser of choice for applet execution.

that is connected by a switching fabric to a second p er second for a p erio d of ve minutes. The hori-

router. This second router has twointerfaces that zontal axis represents the numb er of concurrent re-

connect to university computing lab oratories. Both ceivers during the exp eriment; whereas the vertical

of these lab oratories consist of Ultrasparc-1 work- axis shows the mean delivery latency for the ob jects

stations connected by 10 Mbps Ethernet LANs. on a logarithmic scale. These results show that for

a small data payload a signi cantnumb er of clients

The p erformance exp eriments highlight the scal-

can b e supp orted. During the execution of these

ability of the server in several dimensions: the num-

tests the available bandwidth b etween the server

b er of simultaneous connections, and a throughput

and the lab oratories was approximately 400 Kbytes

metric of ob jects p er second. Both of these exp er-

p er second. This bandwidth was measured b oth

iments characterize the substrate's p erformance in

informally using FTP latency and rigorously with

terms of a data ob ject's end-to-end latency from a

the treno to ol[16]. This explains the steep rise in

data supplier to a receiving client. To measure this

the 10 Kbyte payloads b etween 40 and 50 receivers,

latency, a timestamp is written into the ob ject's at-

that corresp onds to 400 Kbytes and 500 Kbytes p er

tribute list as it ows down the distribution tree.

second resp ectively.At levels of throughput ab ove

These timestamps are written to a log le up on re-

a link's capacity, Salamander's memory bu er lls

ceipt at the subscrib ers. By analyzing the log les

and all latencies reach a steady state due to the -

oine, the exp eriment's latency statistics can b e ex-

nite bu er space and drop tail delivery semantics.

tracted. Since the analysis of data in these exp er-

A virtual channel's bu er space can b e sp eci cally

iments relies on this distributed timestamp infor-

tailored to b ound this maximal latency of received

mation, a metho d for synchronizing the clo cks on

ob jects.

an exp eriment's hosts was applied. To comp ensate

for the di erence in the clo cks, a probabilistic clo ck

synchronization technique, similar to the proto cols

develop ed by Cristian [4]was used.

0.4

100.00

0.3

10.00 0 byte payload 0.2 1 Kbyte payload 10 Kbyte payload

1.00 0.1 Object distribution latency (sec) Mean latency (sec)

0.10 0.0 0 1000 2000 3000 4000 5000 Objects Per Second (100 byte objects)

0.01 Figure 8: Measured latency of ob jects through the

0 40 80 120 160 200 240 280 aried.

Number of receivers. system when ob jects p er second is v

Figure 7: Results from a single supplier Salamander

The second set of exp eriments shows the scalabil-

scalability exp eriment.

ity of a single Salamander server in terms of maxi-

mum ob jects p er second ops that can b e pro cessed.

Figure 7 shows the results of exp eriments that This is similar to a datagram router's packets p er

evaluate the scalability of a single Salamander server second pps metric. The results of this set of exp er-

in terms of b oth numb er of concurrent receivers and iments is shown is Figure 8. The data p oints in this

size of the data payload. Three payload sizes were graph represent the mean and standard deviation of

used: 0, 1 Kbyte, and 10 Kbytes. However while the the latency of ob jects from supplier to client under

payload varied, each of the ob ject's headers were increasing server load. These p oints were generated

lled with approximately 100 bytes of testing and by running exp eriments that sent an ob ject with a

channel information. For all of these results, a sin- header of 100 bytes and a payload of length zero

gle data supplier was used that sent an ob ject once from a set of senders to a set of receivers. Each

sender sent a single ob ject once p er second. For interfaces for channel mo di cation; and the ability

example, in order to generate the 2400 ops data to place channel conversion mo dules at any p ointin

p oint a series of exp eriements were run using sets the distribution tree.

of 40 suppliers and 60 receivers. Unfortunately, the

Salamander's incorp oration of a lightweight tem-

bandwidth b etween the server and the lab oratories

p oral database provides the basis for a p owerful

never exceeded 500 Kbytes p er second for sustained

synergy b etween real-time data dissemination and

p erio ds while these exp eriments were undertaken.

traditional temp oral and relational databases. Our

However, during the 4900 ops exp eriments the rel-

mo del provides supp ort for p ersistent queries that

atively slow 140 MHz pro cessor was approximately

act over b oth a snapshot of the data elements

25 idle, leaving ro om for further scaling. The vari-

present in the database and any mo di cations real-

ation b etween the data p oints results from using an

time up dates to the database elements that may

active testb ed.

o ccur in the future.

Together, these two sets of exp eriments show that

Salamander's research contribution is comple-

a single server substrate scales with the available

mented by the utility and robustness of the cur-

network bandwidth. Further improvements in scala-

rent implementation. Several Internet applications

bility can b e made by utilizing a heirarchy of servers

were describ ed that motivated Salamander's push-

in the substrate to ooad bandwidth and pro cess-

based approach to data distribution. These ap-

ing overhead. The empirical results concur, showing

plications, namely the UARC and IPMA pro ject

Salamander to b e b oth scalable and robust.

applets, use Salamander around the clo ck to pro-

vide application connectivity throughout the Inter-

net. Through day-to-day use, these applications

have shown Salamander's empirical p erformance to

6 Conclusion

b e go o d. Moreover, the quantitative p erformance

exp eriments show that a single server scales well

This pap er presented b oth a functional descrip-

for a signi cantnumb er of connections; the server's

tion of the Salamander distribution substrate's ar-

bandwidth was the rst-order b ottleneck in these

chitecture and interfaces; and a quantitative anal-

exp eriments.

ysis of its p erformance characteristics. The con-

Our currentwork is fo cused on enhancing the

tributions of the architecture are its combination

scalability of the system. Sp eci cally,we are investi-

of channel subscription service with a lightweight

gating the applicability of scalable routing technolo-

temp oral database, and the de nition and use of an

gies to Salamander's attribute-based data dissemi-

application-level QoS framework.

nation. Additionally,we are addressing several sys-

The Salamander substrate's channel subscription

tem administration asp ects of the multiserver sub-

service provides for b oth an anonymous and a ne-

strate, including pairing it with a key distribution

gotiated push of data from a set of suppliers to a

mechanism. In concert with these activities, we plan

set of receivers. The supp ort for anonymous push is

to further address the impact of Salamander's p er-

straightforward, the suppliers know nothing ab out

sistent query mo del on database technologies. Fi-

its set of receivers. In contrast, negotiated push sup-

nally,we are in the pro cess of p orting the server to

p ort enables subscrib ers and publishers to negotiate

b oth Java and Win32 based platforms, in order to

the content of their data channels. To provide this

ob jectively compare the base system in a varietyof

functionality, Salamander includes: a registration

settings.

and matching service, similar to a name service; and

a noti cation service that propagates various system

events, including client termination, to Salamander

endp oints.

7 Availability

Application-level Quality of Service is de ned and

supp orted in the Salamander substrate as a wayof

Additional information on the Salamander system

tailoring the available resources to b est t the user

can b e found at the following URL:

and application. This is done by utilizing seman-

tic knowledge that only the application has ab out

www.eecs.umich.edu/~rmalan/salamander/

its data and providing mechanisms for the graceful

Information that can b e found there includes:

degradation of its virtual data channels. The sp e-

ci c contribution of our work is the use of client

feedback to allow for prioritization among virtual  Binaries and source co de for the Solaris version

channels; the construction of application and user of the Salamander server,

 Generic C and Javainterface libraries for b oth [7] Vassos Hadzilacos and Sam Toueg. Distributed

data suppliers and clients, Systems,chapter 5, pages 293{318. Addison-

Wesley, second edition, 1993.

 Status of b oth a Java and Win32 p ort of the

Salamander server. [8] Internet Performance Measurement and

Analysis IPMA pro ject homepage.

 Information ab out ongoing research e orts

http://nic.merit.edu/ipma/.

based on Salamander.

[9] B. Kantor and P. Lapsley. Network news trans-

fer proto col nntp. RFC 977, February 1986.

Acknowledgments

[10] G. Rob ert Malan and Farnam Jahanian. An

application-level quality of service architecture

Sp ecial thanks go to: Craig Lab ovitz the inde-

for internet collab oratories. In Proceedings of

fatigable Salamander evangelist; Jim Wan for his

the IEEE RTSS Workshop on ResourceAlloca-

development and debugging aid; Mike Bailey for his

tion Problems in Multimedia Systems, Decem-

constructive criticism; and Michael Willis for his ini-

b er 1996.

tial e orts. Thanks also go to the entire UARC de-

velopment team: Peter Kno op, Terry Weymouth,

[11] G. Rob ert Malan, Farnam Jahanian, and Pe-

and Mike Burek. The p erformance exp eriments

ter Kno op. Comparison of two middleware

would have b een unb earable without Don Lib es's

data dissemination services in a wide-area dis-

expect. Again wewould liketoacknowledge the

tributed system. In Proceedings of the 17th

National Science Foundation and Intel for their gen-

IEEE International Conference on Distributed

erous funding that made this work p ossible.

Computing Systems,May 1997.

[12] S. McCanne, V. Jacobson, and M. Vetterli.

Receiver-driven layered multicast. In Proceed-

References

ings of ACM SIGCOMM '96, August 1996.

[1] Anand Balachandran, Andrew T. Campb ell,

[13] Brian D. Noble, Morgan Price, and M. Satya-

and Michael E. Kounavis. Active lters: Deliv-

narayanan. A programming interface for

ering scaled media to mobile devices. In Pro-

application-aware adaptation in mobile com-

ceedings of the Seventh International Workshop

puting. Computing Systems, 84, 1995.

on Network and Operating System Support for

[14] Ab dullah Uz Tansel, James Cli ord, Shashi

Digital Audio and Video,May 1997.

Gadia, Sushil Ja jo dia, Arie Segev, and

[2] David D. Clark and David L. Tennenhouse.

Richard Sno dgrass. Temporal Databases, The-

Architectural considerations for a new genera-

ory, Design, and Implementation. Ben-

tion of proto cols. In Proceedings of ACM SIG-

jamin/Cummings Publisher Company, 1993.

COMM '90, Septemb er 1990.

[15] D. L. Tennenhouse and D. J. Wetherall. To-

[3] C. R. Clauer et al. A new pro ject to sup-

wards an active network architecture. Com-

p ort scienti c collab oration electronically. EOS

puter Computer Review, 262, April 1996.

Transactions on American Geophysical Union,

[16] Treno Web Page.

75, June 1994.

info.. http://www.psc.edu/networking/treno

[4] Flaviu Cristian. Probabilistic clo ck synchro-

[17] Upp er Atmospheric Research Collab-

nization. Distributed Computing, 3:146{158,

oratory UARC pro ject homepage.

1989.

http://www.si.umich.edu/UARC/.

[5] H. Eriksson. Mb one: The multicast backb one.

Communications of the ACM, 378, 1994. [18] Tak W. Yan and Hector Garcia-Molina. SIFT -

a to ol for wide-area information dissemination.

[6] Armando Fox, Steven D. Gribble, Eric A.

In Proceedings of USENIX 1995 Technical Con-

Brewer, and Elan Amir. Adapting to network

ference, January 1995.

and clientvariability via on-demand dynamic

distillation. In Proceedings of the Seventh In- [19] L. Zhang, S. Deering, D. Estrin, S. Shenker,

ternational ConferenceonArchitectural Sup- and D. Zappala. RSVP: A new resource reser-

port for Programming Languates and Operating vation proto col. IEEE Network, September

Systems, Octob er 1996. 1993.