ONLINE MULTICAST GROUPING FOR DYNAMIC DATA DISTRIBUTION MANAGEMENT
Katherine L. Morse Science Applications International Corporation 10260 Campus Point Drive, MS B-1-E San Diego, CA 92121 858-826-5442, 858-826-5112 [email protected]
Michael Zyda MOVES Academic Group Naval Postgraduate School Monterey, CA 93943-5118 [email protected]
KEYWORDS: HLA, Data Distribution of the resulting groupings against an offline Management, multicast, RTI grouping algorithm, and projects the performance impact on a production RTI. ABSTRACT 1 INTRODUCTION The High Level Architecture’s Data (Morse 2000b) describes two algorithms for Distribution Management (DDM) (Morse 1997, performing offline grouping for dynamic DDM DoD 1998) services are the most recent in a and analyzes their performance in terms of their succession of systems designed to reduce the ability to deliver required updates in a timely amount of data received by individual manner. In this paper we describe an online simulations in large-scale distributed implementation of one of those algorithms and simulations. A common optimization in these evaluate both the goodness of the groupings it interest management systems is the use of produces and the potential impact of multicast groups for sending data to a selected incorporating the algorithm in a production subset of all potential receivers. The use of RTI. In section 2, we analyze the potential multicast has met with considerable success in performance improvements for using multicast this application. However, its use to date has grouping. Section 3 describes the offline and relied on a priori knowledge of communication online versions of the grouping algorithm. patterns between simulations and static Section 4 compares the goodness of the results assignment of multicast groups to these generated by both implementations and patterns. As larger, more complex, and less analyzes the potential impact of implementing predictable simulations are built, the need has online grouping in a production RTI. Section arisen for more efficient use of multicast groups 4.2 outlines future work1. as they are a restricted resource. This paper describes an implementation of online multicast 1 grouping, compares the message delivery time Initial work on this project was funded under DARPA ASTT contract MDA9972-97-C-0023. 2 MULTICAST GROUPING 2.1 Connection Graphs The most promising optimization identified to By virtue of regions, we know the destination date is the use of multicast groups for routing of attribute updates before they are sent. Figure data to a controlled subset of all member 2-1 illustrates the problem with a connection simulations in a simulation (Abrams, Watsen, pattern graph. Federate f1 sends the attribute and Zyda 1998; Calvin et al. 1995; Macedonia update(s) represented by connection path c1 to et al. 1995; Mastaglio and Callahan 1995; Rak federates f2 and f3. Federate f3 sends the and Van Hook 1996). The ultimate measure of attribute update(s) represented by connection effectiveness of any interest management path c2 to federates f2 and f4. Federate f1 also system is the latency between sending a piece sends c3 to f4. The connection set, C, of data and an interested receiver getting it. represented by this graph is {
Time to propagate message through the time a message is sent for a connection. network (tp(fi,fj)) - measured between ts tds(c) k • ts federate fi and fj, the publisher and Equation 2-2. tds for c not in a Group subscriber of the connection, respectively. However, adding connections to an existing The algorithms begin by assuming point-to- multicast group may cause extraneous point communication for all messages and falls communication at some receivers, increasing tq back to this position when the network and for some valid communications. For example, scenario make multicast grouping impossible, putting all the connections in Figure 2-1 would i.e. when ts + tp + tr + tq > tmax for some result in the following extraneous connections. communications: c3 to f2 and f3 The decision to add a connection, c, to a c1 to f4 multicast group is based on the expected impact c2 to f1 This simple observation illuminates a much on tmax of all connections and federates already in the group. If connection c is sent to k larger point. We are addressing potential receivers point-to-point, t(c), the time from the performance improvements. There are some beginning of c’s sending to the end of the last circumstances under which we cannot improve receiver’s receipt is bounded by: over the performance of point-to-point. 3 GROUPING ALGORITHMS k · ts + max(tp (fi,fj)) + max(tq (fj)) + tr Equation 2-1. Time Bound for k Point-to- 3.1 The Input-Restricted Largest Outgoing point Communications Connection (IRLOC) Algorithm This bound is based on the worst case The input-restricted largest outgoing connection assumption that the kth communication has the (IRLOC) (Morse 2000b) algorithm seeks to minimize both t and t to produce better longest t and the longest t . ds q p q average results than its predecessor, the largest outgoing connection (LOC) algorithm. This algorithm recognizes three facts about adding a mobile agents system (Bic 1996). The baseline connection to an existing group: prototype implements a minimal subset of RTI Any receivers of the connection who are not necessary to test DDM. The online IRLOC already in the group will receive additional algorithm integrates the baseline prototype with connections equal to the sum of the the basic structure of the IRLOC algorithm. The connections already in the group, the group distributed algorithm operates with degraded weight. information for several reasons. First, the Any group members who are not receivers information about connections and incoming of the connection will receive the additional weights is distributed among the federates, and weight of the connection. collecting it would be prohibitive in a very large
Assuming that ts = tr, improvements in scale distributed simulation. If the simulation average message delivery time created by were small enough to be able to collect all this sending a connection via multicast are data at a central point and still make timely directly offset by the “negative weight” grouping decisions, it wouldn’t need multicast created by the first two facts. grouping. Second, this same information is changing in real time. Regions and region The IRLOC performs the following steps: intersections are changing while the grouping 1. Calculate the positive cumulative effect of decisions are being made. Even if the algorithm each connection, (k - 1) • w. had access to global information when it started 2. Add the receivers of the connection with the grouping, there is a non-zero probability that largest cumulative effect; in the event of a the information would be out of date by the tie, add the lowest numbered such time the grouping completed. Finally, the connection. MESSENGERS system doesn’t provide a 3. Add the next largest connection such that a) straightforward, timely mechanism for passing the input weight of the current group dynamic data structures to Messengers. Some simplifying assumptions have been made which members does not exceed t by the max account for this. In a production system this addition of the connection weight, b) the final constraint could be relaxed. input weight of the connection’s receivers not already in the group does not exceed The online grouping algorithm is triggered by tmax by the addition of the group weight, c) the discovery of a connection or connections. A the positive cumulative effect is greater than grouping Messenger is injected which begins the negative weight. Note that the positive searching for a potential group. The grouping cumulative effect is only a function of the Messenger searches three places in the connection, while the negative weight is a following order: function of the connection and the current 1. on the init node on the local machine; state of the group. 2. at the multicast server where it may find an 4. Repeat step 3 with the remaining unused group; connections. Halt when all connections are 3. at most one hop from the multicast server at assigned or all multicast groups are used. another machine.
3.2 The Online Grouping Algorithm If the grouping Messenger finds an unused The online, distributed IRLOC algorithm is group at the multicast server, it marks the group built on top of a baseline prototype (Morse as taken to the requesting federate’s machine 2000b) implemented in the MESSENGERS and “carries” the group home. This is how groups migrate away from the multicast server. 4 RESULTS If the group is unused, there’s no need to check There are two measures of merit for a DDM for overflow and the group can be used implementation, latency and overhead. As immediately. Before a partially used group can described in Section 2, we wish to minimize the be taken from another federate, it must be average message delivery time while not checked for overflow. The grouping incurring too much overhead cost. Messengers carries the connection weight and connection receivers with it. The current group 4.1 Grouping Results weight and members is always stored with the The first experiments performed were to group at its current init node. All of this determine the goodness of the groupings information is consistent with the IRLOC generated by the online grouping algorithm algorithm and is always up to date. However, relative to the offline version. These the IRLOC algorithm also makes use of the experiments were conducted with random current incoming connection weights of both connection sets. For each desired random the current group members and the connection’s connection set we generated 10 sets of the given receivers. Here is where slightly degraded size, using different random seeds, and information is used. Instead of having the averaged the 10 results. This is to minimize the current incoming weights of all the impact on the results of randomly pathological connection’s receivers, the grouping Messenger connection sets. Table 4-1 lists the parameters carries the last known, largest incoming weight of the random connection sets generated. of all the receivers. The incoming weights of receivers are piggybacked on subscription regions, so they may be out of date due to Table 4-1. Random Connection Set Tests subsequent subscriptions. Instead of the current f 10 incoming weights of all the group’s members, 3 tr = ts 10 milliseconds the group is stored with the last known, largest tp 10 milliseconds between incoming weight of any of all the group’s all federates members. tmax 1000 milliseconds (n, m) (5, 2), (5, 3), (6, 2) If the grouping succeeds, the group’s weight and member list is updated. The grouping is Since the online grouping algorithm can only be reported to the requesting federate which run in real time with the MESSENGERS system changes its connectivity and adjusts its outgoing underneath, this comparison test required connection weight down. It also injects “join” manually generating regions and DDM API Messengers for all the connection receivers who calls whose resulting region intersections were not previously members of the group. In produce the connection sets listed in Table 4-1. this system, this Messenger only informs the The groupings produced in this way were receiver to adjust its incoming weight upward manually edited into connection set files and to account for other traffic from the group and run through the offline simulator. In the online to add to a reference count for this group. If configuration there is no way to create the multicast hardware were available, this entire connection set statically. As soon as a Messenger would also be the trigger for the connection or connections are detected at any receiver to issue the appropriate system calls to join the multicast group. 3 Hoare and Fujimoto (Hoare and Fujimoto 1998) measured these values for RTI 1.3. federate, the RTI component at that federate grouping algorithm implement the barest triggers grouping. This required writing minimum of HLA functionality necessary to auxiliary Messengers which locate existing test the hypothesis with almost no error multicast groups and add new receivers to the checking. RTI 1.3 is a robust, fully-compliant group when they subscribe for a connection HLA 1.3 implementation with all the specified which has already been assigned to the group in service groups and error checking, and the question. overhead implied by that. Table 4-2. Runtime Experiments Even with degraded information about the input weights of the federates, the online grouping # (federates, r i i algorithm compared quite favorably to the regions) /min. /min. 1 (2,50), (5, 200), 0 r/25, r/10 0 offline IRLOC algorithm. In over half the cases, (10,1000) seventeen cases, the online algorithm generated 2 (2,50), (5, 200), r r/25, r/10 0 the same solution as IRLOC. In six of the cases, (10,1000) it generated a better average message delivery 3 (2,50), (5, 200), r r/50, r/25 i time. In one case, the degraded information (10,1000) about input weights caused the online algorithm to generate too conservative a solution. In two Experiment 1 tests the impact of intersection cases, the order in which connections were calculations on initialization time. It establishes discovered affected the order in which they a basis for projection of performance of the were added to groups, i.e. early addition of a online grouping algorithm in an actual connection prevented later addition of a implementation. The baseline prototype and different connection which would have RTI 1.3 are used because the online grouping produced a better result. In one case, the fact algorithm is built on top of the baseline that the grouping Messenger was restricted to prototype, while the baseline prototype has an only looking one hop from the multicast server architecture for DDM which closely models prevented it from putting a potential connection the architecture of RTI 1.3. into an existing group. In three cases, the grouping was affected by both of the previous The average per federate initialization times for two factors, ordering and restricted hops. each of the federates in the RTI 1.3 tests are listed in Table 4-3. Separate tests verified that 4.2 Runtime Performance Results the growth in initialization times is due The experiments described in Table 4-2 are primarily to a larger number of federates, not to designed to test the potential impact of a larger number of regions. The change in integrating online grouping with a production initialization time for grouping was calculated RTI using relative measures between the online as the difference in initialization time between grouping algorithm, the baseline prototype, and the online grouping algorithm and the baseline an actual RTI implementation with DDM, RTI prototype. Given that average per federate 1.3 v4. The experiments use the benchmark initialization time increase is more than three algorithm described in (Morse 1999b). When orders of magnitude smaller than the interpreting the results, it’s critical to remember initialization time without grouping, using that the baseline prototype and the online grouping has no appreciable impact on initialization.
Table 4-3. Initialization Times f r r/ min i i/ min RTI 1.3 (sec.) Change for Grouping 2 50 0 2 0 13.190171 .012 2 50 0 5 0 13.103029 .008881 5 200 0 8 0 24.895595 .015031 5 200 0 20 0 24.134336 .010996 10 1000 0 40 0 132.050392 .009019 10 1000 0 100 0 129.133358 .005554 Averages 56.251147 .010247
Experiment 2 tests the impact of region changes CPU usage more severely than experiment 2 without any intersection changes. As discussed since connectivity changes must be made. in Section 3.2, this should impact CPU usage, Predictably, the RTI produced more bad loops but not severely since the system should for experiment 3 than for experiment 2, 5.7% recognize that connectivity hasn’t changed. vs. 2.6%. However, the grouping algorithm Since regions are uniformly assigned to only resulted in an average of 384 msec more federates, r per federate = r/f which ranges time. The fact that this is lower than the time from 25 to to 100. Here the methodology is to for experiment 2 are initially surprising, but the determine if the RTI can do its job without numbers are so small compared to the robbing the federates of the CPU cycles they measurable resolution that even small need to do their job. Although the Sun Sparc 5s perturbations in the CPU load or network load used in the experiment are slightly on these non-dedicated machines can result in underpowered compared to platforms typically proportionally large differences. used for HLA-based simulations, the RTI performed fairly well. Each experiment is run For the sake of completeness, the additional for 7 minutes with 10 loops per second for a time for all the experiments with the online total of 4200 loops. During each loop, the grouping algorithm and the baseline algorithm federate code performs all the calculations it were recorded and averaged. The average requires and the remainder of the time in the additional time was 2596 msec or .62 msec per loop allocated for the RTI to perform its loop. functions. A “bad” loop is one in which the RTI fails to complete all its processing in the All of this is overshadowed by the time it takes remaining loop time allocated to it. Across all to reconfigure multicast groups in routers. six tests in experiment 2, the RTI only suffered According to (IETF 1997) and (Cisco 1999), an average of 2.6% bad loops. Running the joining a multicast group across a LAN can take benchmark algorithm with the online grouping no time at all, while leaving a multicast group algorithm and the baseline grouping algorithm across a WAN may take on the order of 260 only resulted in an average of 2173 msec more seconds. In summary, it is not the time it takes time taken by the RTI across a 7 minute period; to calculate the multicast groups which is the approximately .5 msec per .1 sec loop. That’s impediment to dynamic multicast grouping as less time than it takes to receive a single has been asserted in the past, but the time it extraneous message that would have been takes to change the groups in the routers. delivered without multicast!
Experiment 3 tests the impact of region changes with intersection changes. This should impact 5 FUTURE WORK required receivers of each connection. The recalculation would be very expensive. Like the 5.1 Maintaining Incoming Weight grouping algorithms, ungrouping could almost Information certainly benefit from heuristic approaches. The offline grouping algorithms always have perfect global knowledge of the incoming 5.3 The Real World weights of all federates. And the algorithms Finally, the true test of this research would be to update this information as they build groups. implement the distributed IRLOC algorithm in The distributed online grouping algorithm has a a production RTI and test it in a very large much less consistent view of this information. scale, dynamic virtual environment. A receiver could form many new connections and be added to other groups in between the 6 CONCLUSIONS time when it sends its incoming weight to a As the size of distributed simulations grow, sender with its subscription region and the time unwanted data received by member simulations when the sender begins to form a group. The will continue to grow as a limiting factor. accuracy of the offline grouping algorithm Multicast has been identified as a highly could be improved by periodically updating effective and efficient tool for controlling the incoming weight information to receivers. One delivery of unwanted data, but multicast groups possibility is to keep track of the incoming are a limited resource. Static assignment of weight sent with each subscription region as multicast groups to particular geographic well as the lowest reported such weight. When regions and data types have yielded positive the difference between the lowest reported results, but may not be extensible to very large weight and the current incoming weight reaches simulations or simulations which exhibit a large a threshold, an auxiliary Messenger with a new degree of chaotic clustering. We have taken incoming weight would be dispatched to update major steps toward dynamic assignment of all senders holding the subscription region. The multicast groups in the context of the HLA’s threshold would have to be determined DDM services. We have demonstrated that it is experimentally, trading off the cost of the feasible to implement dynamic multicast additional overhead of tracking reported grouping in a production RTI, but use of incoming weights against the cost of the dynamic grouping rests on the ability to quickly extraneous messages resulting from inaccurate reconfigure multicast hardware. incoming weight information. 7 REFERENCES 5.2 Ungrouping Abrams, H.; K. Watsen; and M. Zyda. 1998. The focus of this research has been on “Three-Tiered Interest Management for Large-Scale developing groups. Connections are dropped Virtual Environments.” In Proceedings of 1998 from groups and groups are dissolved when all ACM Symposium on Virtual Reality Software and the receivers have dropped out. However, the Technology (VRST'98, (Taipei, Taiwan). departure of only one or two receivers may Bic, L.; M. Fukuda; and M. Dillencourt. 1996. adversely affect the effectiveness of the Distributed Computing using Autonomous Objects. grouping. In the extreme case the cost function IEEE Computer, 29(8), August 1996. could be recalculated every time a receiver Calvin, J.; D.P. Cebula; C.J. Chiang; S.J. Rak; and drops out. Doing so would require keeping D.J. Van Hook. 1995. “Data Subscription in Support complete information about the connections in of Multicast Group Allocation.” In 13th Workshop the set, including the identities of all the on Standards for the Interoperability of Distributed Morse, K.L; L. Bic; and M. Dillencourt. 2000. Simulations (Orlando, FL, September) 367-369. “Interest Management in Large Scale Virtual Environments.” MIT Presence, March 2000. Cisco IOS 12.0 Solutions for Network Protocols Morse, K.L. 2000b. “An Adaptive, Distributed Volume 1: IP. Cisco Press. 1999. Algorithm for Interest Management.” Ph.D. Department of Defense High Level Architecture Dissertation, University of California, Irvine, May Interface Specification, Version 1.3, DMSO, April 2000. 1998, available at http://hla.dmso.mil. Morse, K.L.; M. Zyda. 2000b. “Multicast Grouping Hoare, P.; and R. Fujimoto. 1998. “HLA RTI for Dynamic Data Distribution Management.” In Performance in High Speed LAN Environments.” Proceedings of the 2000 Distributed Simulation - In Proceedings of the 1998 Fall Simulation Real Time Workshop. (San Francisco, CA, August). Interoperability Workshop. (Orlando, FL, Papadimitriou, C.H. 1994. Computational September). 501-510. Complexity. Addison-Wesley, New York. Pg. 190. IETF Network Working Group. Internet Group Rak, S.J.; and D.J. Van Hook. 1996. “Evaluation of Management Protocol, Version 2, RFC 2236. Grid-Based Relevance Filtering for Multicast Group Available at http://rfc.fh- Assignment.” In 14th Workshop on Standards for koeln.de/rfc/html/rfc2236.html, November 1997. the Interoperability of Distributed Simulations (Orlando, FL, September) 739-747. Macedonia, M.; M. Zyda; D. Pratt; and P. Barham. 1995. “Exploiting Reality with Multicast Groups: a Network Architecture for Large Scale Virtual AUTHOR BIOGRAPHIES Environments.” In Virtual Reality Annual International Symposium ’95. 2-10. KATHERINE L. MORSE is a Senior Computer Scientist with SAIC. She received Mastaglio, T.W.; and R. Callahan. 1995. “A Large- Scale Complex Virtual Environment for Team her B.S. in mathematics (1982), B.A. in Russian Training.” IEEE Computer 28, no. 7 (July): 49-56. (1983), M.S. in computer science (1986) from the University of Arizona, and M.S. (1995) and Morse, K.L.; and J.S. Steinman. 1997. “Data Ph.D. (2000) in information & computer Distribution Management in the HLA: science from the University of California, Multidimensional Regions and Physically Correct Irvine. Dr. Morse has worked in industry for Filtering.” In Proceedings of the 1997 Spring over 20 years in the areas of simulation, Simulation Interoperability Workshop (Orlando, FL, computer security, compilers, operating March). 343-352. systems, neural networks, speech recognition, Morse, K.L.; L. Bic; M. Dillencourt; and K. Tsai. image processing, and engineering process 1999. “Multicast Grouping for Dynamic Data development. Her Ph.D. dissertation is on Distribution Management.” In Proceedings of the dynamic multicast grouping for Data 1999 Society for Computer Simulation Conference. Distribution Management, a field in which she (Chicago, IL, July). is widely recognized as a foremost expert. Morse, K.L; L. Bic; and M. Dillencourt. 1999b. “Characterizing Scenarios for DDM Performance MICHAEL ZYDA is a Professor in the and Benchmarking RTIs.” In Proceedings of the Department of Computer Science at the Naval 1999 Spring Simulation Interoperability Workshop, Postgraduate School, Monterey, California. March 1999. Professor Zyda is also the Chair of the NPS Modeling, Virtual Environments and Simulation Academic Group. Since 1986, he has been the Director of the NPSNET Research Group. Professor Zyda's research interests include computer graphics, large-scale, networked 3D virtual environments, computer- generated characters, video production, entertainment/defense collaboration, and modeling and simulation. He is known for his work on software architectures for networked virtual environments. Professor Zyda was a member of the National Research Council's Committee on "Virtual Reality Research and Development". Professor Zyda was the chair of the National Research Council's Computer Science and Telecommunications Board Committee on "Modeling and Simulation: Linking Entertainment & Defense". From that report, for the Deputy Assistant Secretary of the Army for Research and Technology, Professor Zyda drafted the operating plan and research agenda for the USC Institute for Creative Technologies (ICT).
Professor Zyda is a member of the National Research Council Committee on Advanced Engineering Environments. Professor Zyda is also a Senior Editor for Virtual Environments for the MIT Press quarterly PRESENCE, the journal of teleoperation and virtual environments. He is a member of the Editorial Advisory Board of the journal Computers & Graphics. Professor Zyda is a member of the Technical Advisory Board of the Fraunhofer Center for Research in Computer Graphics, Providence, Rhode Island.