Reza Jalili A Visit to the Frauenkirche and Peter D. Kirchner IBM Research Division T. J. Watson Research Center Yorktown Heights, NY 10598 Abstract Jorge Montoya and The Frauenkirche was destroyed when Dresden was bombed by the Allied forces Feb- Stephen Duncan ruary 13-14, 1945. The church is now being reconstructed in an effort led by the Research Triangle Institute Foundation for the Reconstruction of the Frauenkirche. The VRDECK software package Research Triangle Park, NC 27709 developed at the IBM T. J. Watson Research Center was used to view and walk through a model created from the original church plans. A Polhemus tracker and a cus- Luc Genevriez tom-built joystick using the Logitech 3D mouse were used for six-degree-of-freedom I I Rue Ledru-Rollin input to the application. The interactive fly-through of the church is in an immersive 92500 Rueil-Malmaison, France environment. One can navigate around the model wearing a head-mounted display, sitting in front of a standard monitor, looking at a stereo image produced on a stereo James S. Lipscomb monitor, or standing before a projection screen displaying a stereo image of the scene. Robert H. Wolfe The system was developed for and exhibited at the IBM booth in the CeBIT fair in Han- and nover, in March 1994 with funding from IBM Germany. Christopher F. Codella IBM Research Division T. J. Watson Research Center I Introduction Yorktown Heights, NY 10598 The Frauenkirche was destroyed when Dresden was bombed by the Al- lied forces February 13-14, 1945. The church is now being reconstructed in an effort led by the Foundation for the Reconstruction of the Frauenkirche (Whit- ney, 1994). The original plans were used to create a computer model of the church to aid the reconstruction effort. From that model, a more elaborate and aesthetically accurate model was developed. Those models were then made available for the project that is described in this paper. The goal of the public exhibit was to produce a realistic experience of being inside the church as it would have looked before the war. The Frauenkirche exhibit made use of the best technology available to us to produce such an im- mersive walkthrough experience. In Section 2 we briefly describe the software architecture used to build the distributed application.

2 VRDECK Software

2.1 General Overview

The Frauenkirche virtual reality exhibit was developed using two worksta- tions, each handling part of the necessary tasks. The VRDECK software was used to distribute, define, invoke and monitor, and control the tasks. This section reviews the design and operation of the VRDECK, Virtual Re- et Presence, Vol. 5. No. I. Winter 1996. 87-94 ality Distributed Environment and Construction Kit (Codella al., 1993) de- © 1996 by the Massachusetts Institute of Technology veloped at the IBM T. J. Watson Research Center. Ease of use was emphasized

Jalili et al. 87

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 88 PRESENCE: VOLUME 5, NUMBER I

both for the application builder and the toolkit extender. mie ExampleRuleName when(DataTypeA InEvent) Under an extensible architecture, the system supports produceCDataTypeB OutEvent) the distribution of and the processes building of mul- { tiuser shared environments. Specialized I/O devices are // some C++ code which can read data from InEvent supported with prebuilt exécutables for interfacing with // some C++ code which can write data to OutEvent gloves, three-dimensional (3D) position sensors, sound if (condition) produce(OutEvent); }; generators, speech recognizers, and 3D graphics dis- plays. Virtual world environments are created using a • ExampleRuleName is the name of this particular rule. mixed object-oriented and event-based paradigm for • DataTypeA and DataTypeB are built-in or user defined types (classes or structs). behavior. defining system • InEvent is the identifier of the input or triggering event whose data type is DataTypeA Basic units, called modules, represent entities in the • OutEvent is the name of a possible output event whose data type is DataTypeB. world such as objects, operations, functions, or users. Modules are always separate exécutables, and can be dis- Figure I. An example showing the rule syntax. tributed across a network. Modules communicate with each other by producing and consuming events, and are defined at a high-level using rules written in C+ + that determine how events are handled. The toolkit includes vices and simulators. Extensive run-time support trans- a development environment consisting of C+ + class parently handles event queuing, matching of events to libraries for module construction, communication, and rules that process the events, intermodule event and data device support. The intermodule and intramodule event transport, networking support, I/O device support, and communications are handled transparently by the interfaces to the X Window System®. This run-time VRDECK code. support, contained within C++ class libraries, is de- The run-time environment includes a GUI control signed to relieve the world designer from low-level sys- panel for dynamically constructing an application from a tems issues. It also provides the toolkit extender with a collection of modules, allocating processes among hosts, rich set of facilities upon which to enhance the base sys- saving configurations, and monitoring application op- tem. eration. A set of modules for various devices and a li- A module's rules describe (1) events of interest, (2) brary of classes for common operations facilitate applica- events it may generate, and (3) code to implement the tion construction. behavior of the module (see Fig. 1). When a module receives an event, it determines which of its rules have an interest and the event to those rule for 2.2 Modules, Rules, Events passes objects processing. When passed an event, a rule determines Modules can be thought of as the building blocks whether it is triggered by that event. If so, the rule ex- of a virtual world. They usually represent discrete objects ecutes its body ofC+ + (or C) code, which may itself or collections of objects, specific services (like I/O device cause the production of new events. An event's data type servers), or functions that can be encapsulated (like information is checked when it is received by a module simulators). Intra- and intermodule communication oc- to ensure that the type of the event matches the type of curs via events. Events are defined as discrete occur- the data expected by the rules. This helps prevent errone- rences broadcast to other modules. In addition to an ous interpretation of the data. identifier or name, an event carries a strongly typed data Events may be produced outside of rulesets by object (possibly void). The external view of a module is explicitly calling the ModuleClass member-function defined by the events it produces and consumes. ProduceEventQ with the event name, data type, and event A module, at its highest level, is a collection of rules to data and size. Rule bodies can, therefore, call subrou- handle the processing of events generated by the mod- tines that produce events. Moreover, input device drivers ule, other modules, or external sources such as I/O de- use this ProduceEvent method to package data into events.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 Jalili et al. 89

Events received and produced may be either built-in Network Connection types (e.g., int, float, char), or user-defined types (i.e., classes or structs). In the simplest cases, built-in types can be used and their value assigned or read just as with normal variables ofthat type. If no data are to be associ- ated with an event, the type may be specified as "void." Events of type "void" are often simply used as flags to rules. trigger Workstation 1 Workstation 2 In a rule definition, the when() clause (see 1) lists Fig. RS-232 a group of input events that must be received before the Serial Line code in the rule body will be executed. At least one input event must be specified. If only one event is specified, as (Tracker Hardware Stereo Monitor in the rule time an event with Figure 1, triggers every Head-mounted Joystick Hardwan the specified identifier is received. If more than one event Display is a conditional function must be specified, provided, Large-screen indicating the condition under which the rule will be Stereo Projector Head Four-button The built-in conditional functions are triggered. and, Position Joystick then, and followed by. They may not be mixed within Sensor the same when() clause. Figure 2. System diagram of the Frauenkirche walkthrough application. Software components are represented as elipses. Hardware components are represented by blocks. 3 Building the Walkthrough Application 3.1 Application Overview Application Tracker VRDECK was used to put together a set of soft- ware components that as a set form the Frauenkirche walkthrough application. The complete system is made up of some specialized hardware and software. Figure 2 shows a block diagram of the system components. Fig- ure 3 shows module connections. These connections happen to use sockets and communicate via TCP/IP. The process of building the Frauenkirche application can be outlined as shown below:

• Translate the model (geometry, textures, materials, and lighting) from the TDI format to one usable by the Performer library. • Preview the model.1 Figure 3. The application is connected to the tracker module and also • model as until is Simplify necessary performance to the joystick module. maximized. • Write a renderer to load and display the model in • Write a VRDECK ruleset to handle the tracker stereo. data. • Write a VRDECK ruleset to use the renderer code. • Purchase and integrate various items (HMD, Fas- 1. The Perfly sample program that comes with the Performer pack- trak, etc.). age was used. • Test and establish performance.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 90 PRESENCE: VOLUME 5, NUMBER I

3.2 Hardware Polhemus tracker. The joystick module is a VRDECK module providing button-state events. The hardware system was comprised of the follow- ing: 3.4 Model Data • an IBM RISC System 6000 model 560, • a Silicon Graphics Onyx® Reality Engine, A model of the Frauenkirche was available in CA- • a Polhemus 3SPACE (R) Fastrak (R), TIA format. The original model was transferred to TDI • software and remade. About a home-built joystick based on the Logitech 3D (Thomson Digital Image) mouse, 10% of the original CATIA model was kept. All the de- tails were redrawn. Sources used to create the model of • an Ikegami Stereo projector, and the church included made the architect • a Virtual Research EYEGEN3 NTSC head- original plans by mounted display. Arno Kiesling in 1944 and also old black and white pho- tographs. The final model used by the rendering module contained about 160,000 polygons for the inside of the 3.3 Software System church and about 130,000 polygons for the outside of the church, made use of 11 textures in 20 locations,2 and The software system was comprised of the follow- was lit five sources. The observer was con- ing: by light strained to the inside of the church to improve perfor- • a custom rendering module, mance. The outside of the church was not used in this • a VRDECK Polhemus Fastrak control module, application. However, the full model was used for a set • a VRDECK Logitech 3D mouse control module, of animations (Collins, 1993). and • the VRDECK "vbuild" operator panel. 3.5 Presentation and Navigation Our objective was to as realistic a display stereoscopic Figure 4 shows an approximate diagram of the of as image the Frauenkirche possible while providing floor layout and operation configuration for the exhibit. interactive-rate of the scene. To achieve fast rendering The VRDECK operator panel, vbuild, was started each with this scene a rendering complexity, two-graphic pipe morning. From vbuild, a project file holding the system Silicon was used. All Graphics Onyx® Reality Engine configuration information was loaded. From that file, other tasks were run on an IBM RISC 6000®. System the system process (VRDECK Modules) were remotely Position was handled a tracker module run- tracking by started on the application machine, and on the I/O ma- on IBM ning the workstation. The tracker module com- chine. An event sent to the application module triggered municated with a Polhemus 3Space Fastrak over an the start of the application. RS-232 serial line at 19,200 baud. A joystick based on Once the application was started, the user, holding the the Logitech 3D mouse was used to supply the state of joystick in his hand and wearing the head-mounted dis- four-button inputs for navigation. The module handling play, could begin to "walk" around the model. Button this code was also run on the IBM workstation. presses were mapped to motion forward and backward The walkthrough was divided into three processes: and The mod- application, tracker, joystick. application 2. The textures ranged in size from 80 KB to over 3 MB. Most were ule consisted of Performer libraries, VRDECK code for around 350 KB. Some textures, like the ones for the organ and the alter, were based on old black and white photos that were scanned in, stereo VRDECK rules for rendering, application-specific adjusted for protection, and colorized by hand using a "paint box" tool navigation, and a Performer-readable model of the while trying to remain true to the originals. Other textures, such as the stone walls for the outside of the church, were drawn hand. Frauenkirche. The module is a VRDECK mod- entirely by tracking There are more locations than textures because some textures were ule for controlling and event-based interfacing with the used in multiple locations (e.g., floor and window).

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 Jalili et al. 91

ruleset main { rule startup when (void start) produce(void Loop) {

resetHeadPositionO ; From Workstation 2 Projector produce(Loop); } rule loop when (void Loop) produce(void Loop) Viewers stand here {

Viewers stand here calculateNewPositioriQ ;

Figure 4. Floor diagram for the Frauenkirche walkthrough exhibit. The produce(Loop); user is tethered by the Polhemus six-degree-of-freedom tracking sensor drawFrameO ; cable, the held in his or her hand, and the head-mounted joystick > display cable. Viewers watch the walkthrough in stereo wearing passive polarized glasses. } Figure 5. Rules to define rendering behavior. The start event is produced through vbuild, the operator panel. along the line of sight and also to motion up and down. An event sent from the vbuild panel to the application module could adjust the speed of motion. Events could cally or in stereo, with a large-screen projector, a head- mounted or a stereo monitor, all selected at just as easily have been mapped to utterances recognized display, via environment variables. by a speech recognition system, to another mouse-but- start-up The renderer was written calls to the Performer ton, or to a driving simulation (Koved et al., 1993). using API with new code for a Two pushbuttons were used to move the observer library, together producing stereo and a head-mounted forward or backward along the direction of gaze. The image driving display. same two buttons were used to move the observer up or was to of down. A third button used toggle the behavior 3.7 Ruleset s the two pushbuttons.3 A set of interface functions was created to give higher-level access to the renderer. These functions, in 3.6 Renderer Code turn, were called from within rules that were written to define the behavior of the The interface The rendering code developed for this project en- application. functions are called in reaction to tracker data events and abled loading the model of the Frauenkirche, setting up button events. lights, binding textures, and dynamically selecting a joystick viewpoint. We could also display the model monoscopi- The ruleset shown in Figure 5 defines the rendering behavior for the application. The ruleset shown in Fig- ure 6 defines the interpretation and use of the tracker 3. A malfunctioning fourth button could not be used. This necessi- data. The rulesets have been simplified to serve as ex- tated a in control at the last minute. Because quick change navigation of the kind of code that makes the behavior of the modularity of the code and the flexibility afforded bv the rule amples up language described in Section 2, the code change was trivial. specification for an application. Typically, rules exist to

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 92 PRESENCE: VOLUME 5, NUMBER I

ruleset trackerRules { rule newTrackerData when (T_TrackerPosition TrackerPosition) produce (void NavigatorAskForHeadPosition) Í /* set the head position */ produce(NavigatorAskForHeadPosition) ; processHeadPosition(HeadPosition); }

rule getMoreHeadData when (void NavigatorAskForHeadPosition) produce(T_TrackerName TrackerPositionQuery) {

/* ask for data from the sensor on the head */ produce (TrackerPositionQuery); 7. view > Figure Wireframe of the main section of the Frauenkirche the balustrade, , and } containing , organ.

Figure 6. Ruleset to define tracker interpretation. Other than for heating and ventilation problems at detect the presence of a tracker, the connection to it, a night, for which the machines were turned off, the sys- lost connection, coordinate transformations, and the tems were up and running without operator interven- like. tion the entire day, each day. From a technical point of view, the performance of the was not as The was to 3.8 Testing system hoped. goal generate an interactive stereoscopic image at 10 to 15 frames per Testing the application was done with the Xtracker second. We fell short of this goal in complex areas of the module, which is a GUI module that generates tracker church, while exceeding it comfortably in simpler areas. data. This capability also demonstrated the modularity, To preserve its artistic quality, the model was only flexibility, and on-the-fly reconfiguration possible with slightly simplified by 10%. Figure 7 illustrates the com- the VRDECK architecture. Another generally useful plexity of the model. In addition, the high-performance debugging feature is that vbuild, the GUI operator's graphics hardware and software configurations were panel, provides capability for generating events to send available only at the exhibit hall and not during the de- to modules (for instance, to generate the event that nor- velopment time before. Therefore, full use of this ad- mally would originate from a button press on a Logitech vanced configuration was not possible because of lack of 3D mouse). development time on the target system. For the most complex scenes, those with a full view of the altar and balustrade (see Fig. 8), the system delivered 4 Results a stereo image at the rate of 3-5 frames per second. At frame rates this low, latency due to tracking hardware The exhibit was very successful in achieving its and communication overhead are not an issue, since the goal of exposing the public to this new technology. The length of the rendering cycle overshadowed their contri- CeBIT fair attracted over 600,000 people. The exhibit, butions. in the center of the IBM hall number one booth, at- Better frame rates could be achieved in most other tracted approximately 14,000 visitors. Throughout the parts of the church. Specifically, a rate of 6-9 frames per fair, the system remained functioning and failure-free. second was measured in scenes with complexities, as in

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 Jalili et al. 93

Figure 8. A very complex scene showing the altar and balconies is Figure 10. A moderately complex scene showing the balconies and rendered at 3-5 frames per second. floor is rendered at 6-9 frames per second.

ware was custom-built with stereo wide-angle rear-pro- jection video. This application was made using the VAX 780 and an IKONAS raster graphics adapter, which were the fastest machines available at the time. The sys- tem was able to display a model with 8000 polygons. Modern systems display more polygons at a comparable rate. Another system that moves an observer around a large interior space is Larry Hodges' Phobia project (Hodges et al., 1993). That project seeks to help acraphobic indi- viduals overcome their fear of elevators by seeming to lift participants in a big room. Overall, the reaction from the visitors was quite favor- able. We believe this is attributable to the visitors' appre- ciation of the complexity of the scene, the beauty of the Figure 9. A moderately complex scene showing a view of the is church itself, and the format of presentation. One does rendered at 6-9 frames per second. get a sense of the size of the structure, and can appreciate the details. Even for those not wearing the head- mounted display, the stereo effect on the large projec- Figures 9 and 10. Trivial scenes with few or no objects tion screen, together with the dark theater, produced a were rendered at 30 frames per second. One should re- sense ofpresence. member, though, that the model contained 160K poly- gons, was textured, had several light sources, and was being rendered twice for the stereo effect. Acknowledgments In comparison, one of the earliest raster-graphics walkthroughs (Brooks, 1986) tried to get the feel of a We would like to thank the members of the Virtual computer science building under construction. The soft- Worlds Group at IBM Research and the Center for

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021 94 PRESENCE: VOLUME 5, NUMBER I

Digital Systems Engineering at RTL Special thanks goes Frauenkirche. IEEE Computer Graphics and Applications, to Robert Haak of IBM Germany for organizing the 13(6), 13-15. A tool- exhibit. The development team is indebted to Silicon Codella, C, Jalili, R., Koved, L., & Lewis, B. (1993). Graphics Computer Systems, both in the United States kit for developing multi-user, distributed virtual environ- ments. Proceedings of VRAIS 1993, 401-407. and in Germany, for their support of this project. In par- Codella, C, Jalili, R., Koved, L., Lewis, B., Ling, D., Lip- ticular, our thanks go to Michael Jones of the Iris Per- scomb, J., Rabenhorst, D., Wang, C. P., Sweeney, A. N., & former Group, David Orton of the High Performance Turk, G. (1992). Interactive simulation in a and Knoblack of SGI Ger- multi-person Systems Division, Ruediger virtual world. ACM CHI'92 Conference Chi- for their individual Proceedings, many cooperation. cago, IL. Hodges, L. F., Bolter, J., Mynatt, E., Ribarsky, W., & Teylin- gen, R. (1993). Virtual environments research at the Geor- References gia Tech GVU Center. Presence: Teleoperations and Virtual Environments, 2(3), 234—243. Brooks, F. P., Jr. (1986). Walkthrough—a dynamic graphics Thalman, M. N., & Thalman, D. (1992). Six hundred indexed system for simulating virtual buildings. Proceedings ofthe references on computer animation. Journal of Visualization 1986 Workshop on Interactive 3D Graphics, Chapel Hill, NC, and Computer Animation, 3(3), 147-174. 9-12. Whitney, C. (1994). From the ruins shall rise a church, and a Collins, B. (1993). From ruins to reality—The Dresden symbol. The New York Times.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres.1996.5.1.87 by guest on 23 September 2021