Interacting with Scientic Visualizations UserInterface Tools within Spatially Immersive Displays 1 1;5 2;3;4 John Bresnahan Joseph Insley and Michael E Papka 1 Distributed Systems Lab oratory Mathematics and Computer Science Division Argonne National Lab oratory Argonne IL 2 Futures Lab oratory Mathematics and Computer Science Division Argonne National Lab oratory Argonne IL 3 Computation Institute University of Chicago Chicago IL 4 Department of Computer Science University of Chicago IL 5 Department of Electrical Engineering and Computer Science University of Illinois at Chicago Chicago IL Abstract User interfaces represent a do orknob to computer applicatio ns and di rectly relate to the usability of a given application The desktop WIMP interface has b een developed and rened over the past years and works well for the desktop environment The question is What do es one do in the case of to days threedimensional immersive environments Do es one translate the desktop to ols to the D space and how do es one interact within these spaces In buildin g nu merous immersive scientic visualizati on application s we have developed a variety of dierent user interface to ols for use within these environments These interfaces include metho ds of controlling and driving the applicatio ns to ols for navigating datasets and underlying libraries for the development of collab orative versions of these to ols With the development of these to ols we have learned many lessons as well as gained new insight into future user interface work Introduction The b enets of scientic visualization to the analysis and understanding pro cess is nothing new Scientists have embraced its use in order to gain mean ingful insight into the data that they collect generate and compute As the compute resources scientists use continue to improve the complexity of the data that they are trying to interpret increases This situation has sparked a new wave of data analysis that moves scientic visualization o the desk top and into immersive virtual environments What scientists need now is an intuitive way to interact with their data within these virtual environments Dep ending on the data b eing visualized and the type of visualization b eing p erformed there are myriad dierent asp ects of the visualization that one would like to b e able to control These range from changing the p osition and orientation of a volume of data that is b eing explored to adjusting the parameters of an ongoing simulation whose output is b eing rendered in real time One would also like to b e able to control all of these asp ects from within Bresnahan Insley and Papka the virtual environment without having to return to the desktop to type in some command on the keyboard One solution is to b orrow the familiar D menuing scenario from the desktop But D menus dont always transition easily into D Because the resolution of many virtual reality systems is somewhat limited textbased menus are often dicult to read In addition it is often necessary to navi gate through several levels of menus to change the mo dality of the task that is b eing p erformed This pro cess can b e tedious and can make the system dicult to use The problem can b e alleviated somewhat by having multiple menus op en at once but that can lead to clutter where the menus start to obscure ones view of the data that is b eing visualized Furthermore it is simply unnatural to use D menus to interact with a D world whether that world is real or simulated As with any user interface it is imp ortant to provide visual cues and user feedback on the state of the system Such cues may involve highlighting an ob ject to let users know that it has b een selected or changing its color to indicate that its state has changed These cues let users know how the system has interpreted their actions provide a b etter understanding of how those actions are aecting the visualization and enable users to make informed decisions ab out how to pro ceed Our exp eriences have b een with developing scientic visualization applica tions primarily for the CAVE and ImmersaDesk Both are pro jectionbased virtual reality VR devices that use a pair of tracked LCD shutter glasses to provide a stereo usercentered p ersp ective Users interact with the envi ronment using a tracked D mouselike input device called a wand that has three buttons and a joystick We describ e here our exp eriences with some of the to ols that we have developed for interacting with scientic visualizations in these virtual environments In particular we discuss a control panel inter face and data selectionnavigation to ol a Dbased menuing system and an infrastructure for connecting these to ols in a collab orative session We briey describ e the each to ol discuss some lessons learned and present ideas on where we see the to ol heading We conclude with an overall discussion of the future of D user interfaces ControlPanel In order to control dierent asp ects of a visualization from within a virtual environment it is often desirable to use valuators such as sliders for changing the value of some parameter or a variety of buttons and toggles to change mo dalities or to trigger certain events As more control over a greater number of asp ects of the visualization is added however managing the placement of all of these controls often b ecomes problematic If individual comp onents are left oating freely in the space the space quickly b ecomes cluttered and unmanageable To help alleviate this problem we developed the ControlPanel Interacting with Scientic Visualization s Fig Image of user interacting with the CP on a Idesk CP which is a central lo cation where all of these controls can b e kept Originally designed for the ImmersaDesk Idesk which is a singlescreen drafting tablesized virtual reality device the CP sits at the b ottom edge of the screen Much like on the dashboard of a car all of the controls are now easily accessible yet out of the way so as to not obscure the users view of the rest of the environment See Figure Similar to windows in a desktop environment the CP can also b e minimized to hide the controls and provide an even b etter view of the scene The ControlPanel library is implemented by using Op enInventor The CP provides a number of generic widgets Interaction with these widgets is managed by the CP which uses raycasting from a ray at the end of the wand to determine whether an ob ject has b een intersected A p ointer which coincides with the ray is drawn at the end of the wand to make the selection of ob jects easier When an ob ject is intersected it changes color to indicate that it can now b e selected The ob ject can then b e selected by pressing one of the buttons on the wand which causes the ob ject to change color again letting the user know that it is now active All of these widgets are user programmable That is the application developer can sp ecify value ranges indicate how the b ehavior of some widgets is aected by the value of others and so on Also the widgets remain indep endent of the other ob jects in the Bresnahan Insley and Papka scene Once the ControlPanel has b een up dated the developer can get the values of the various widgets from the CP and use them to determine how or whether the rest of the scene should b e altered This approach keeps the CP somewhat generic while giving the programmer the desired control Since providing users with feedback ab out what they are communicating to the system is so imp ortant the CP also has a joystick ob ject that represents the joystick on the CAVE wand When the user pushes forward back left or right on the wands joystick the joystick on the CP moves in the same way to reect the user input If the joysticks values are not b eing used to control anything then the joystick on the CP remains in the default p osition In addition to the widgets provided by the ControlPanel library devel op ers can add their own custom ob jects to the CP Typically one simply provides the CP with metho ds to get the ob jects ro ot no de Op enInventor so that it can b e added to the CPs scene graph for up dating it and for getting and setting its values Widgets Among the widgets that the ControlPanel provides are sliders buttons text ob jects and scrolling lists Figure shows a view of the CP Fig Snapshot of the CP taken from the CAVE simulator Sliders The sliders can b e a single slider or a group of them Their minimum and maximum values are set by the programmer as well as whether their value should b e an integer or a oatingp oint number Interacting with Scientic Visualization s Buttons Buttons can also b e added to the CP individually or a group and they have variety of mo dalities Within a group of buttons each button may b e turned on and o indep endently of the others Alternatively it is p ossible to allow at most one button to b e active at a time so that when a button is activated if another button was already active it b ecomes deactivated A third mo dality is much like that of a radial button where exactly one button is active at all times in this case the currently active button can b e deactivated only by selecting one of the other buttons Text Ob jects The text ob ject is just a wrapp er around Op enInventors D text ob ject that makes it easier to add text to the ControlPanel As mentioned earlier text is often dicult to read in many virtual reality systems and the Idesk is no exception but it is also often necessary One needs to b e careful when adding text to ensure that it is large enough to read but that it is kept