Tangible + Virtual = a Flexible 3D Interface for Spacial
Total Page：16
File Type：pdf, Size：1020Kb
Tangible + Virtual = A Flexible 3D Interface for Spatial Construction Applied to DNA Steven Schkolne Hiroshi Ishii Peter Schröder Caltech MIT Media Lab Caltech ABSTRACT Ideas from tangible interfaces and VR ease a difficult spa- tial design task: construct a DNA molecule with desired characteristics. Our hybrid interface has both the physical intimacy of tangible media and the versatility of 3D digital display. Two new physical affordances: a raygun and a grip tool, enable kinesthetic control of the addition and removal of structure. We introduce 3D local menus which select multiple functions for each tool. New interactions for sensed tongs enable the sophisticated multi-object ar- rangement that the delicate, intricate DNA construction task demands. These flexible tools allow UI designers to create multiple interfaces upon the same physical substrate. In a user study, practicing research scientists expressed a strong preference for Silkworm, our 3D interface, when compared to mouse/monitor UI. We show that 3D tangible interfaces, heretofore only applied to freeform artistic crea- Figure 1: Tangible creation: Our experimental interface tion, also facilitate intuition in the highly structured task supports drawing, cutting, and bonding DNA in 3D that is our focus. space. Here the user draws part of a DNA cube directly in a head-tracked stereo display. This tangible tool can Keywords be repurposed to cut bonds between molecules, or draw Spatial construction, tangible user interface, virtual reality, single DNA strands. 3D interface, tongs, lightsaber, DNA design, molecular We use composite images like this one throughout the text biology. as they are more accurate representations of user experi- ence than direct photographs. For footage of the system INTRODUCTION in use, please see the accompanying video. Scientists researching molecular biology encounter a diffi- cult spatial design task. Their objects of study are so small that they can’t physically engage them. They are so intri- reality (VR) to view data [5] but never (to our knowledge) cate that diagramming them with 2D paper and pencil is to construct it. Perhaps this is because constructing models insufficient. Traditional interfaces support viewing, but are requires intricate spatial manipulations, and even moving a not rich enough to aid in creation. These problems are a single object through virtual space is still an area of active special case of the spatial construction problem encoun- research [3, 16]. Our problem of DNA construction in- tered by engineers, artists, and scientists alike. In this paper volves difficult tasks, such as making three double-helices we focus on the task of creating structures out of DNA, intersect so that their six strands smoothly merge (pairwise) currently explored by Seeman [6], Winfree [21], and many into three (see user study). This smoothness is crucial to the others. This specific problem is of increasing importance as success of a DNA molecule when realized in the lab [18]. we begin to design increasingly complex molecular ma- How can we make such manipulation feasible? We draw chines. inspiration from Tangible User Interfaces (TUIs) [10] The scientific visualization community has used virtual which use physical objects to add affordances to data. There are many success stories in the literature which show tactile cues enhance spatial understanding [9]. Current work either occurs in 2D space (using projectors to overlay Caltech CS TR 2002.011 data onto the physical interface [20]), or a 3D space that is not a display [2,8]. Neither of these approaches is sufficient because our problem is three-dimensional and our data is more dynamic than static physical models allow. Caltech CS TR 2002.011 Figure 2: Manipulation with sensed tongs: Holding the Figure 3: Drawing a bond: The user points at a molecule, molecule with the left hand, the user sweeps in with the squeezes the raygun’s trigger, and drags to create a bond. right-hand tong to adjust the position of a single DNA base Pointing at the second molecule and releasing the trigger (one of the blue cones). completes the operation. Our solution is a hybrid: we use the tangible affordances of Immersive systems have been applied to artistic creation TUIs to control a flexible virtual data representation, all of [12, 17]. These systems succeed in VR because of their which occurs in an integrated 3D space (the semi- ability to capture gesture. The DNA design problem raises immersive Responsive Workbench [13]). We present new new issues because gesture and emotional content are not interactions for moving objects and specifying topological important, but rather the demands on spatial properties are changes. More importantly we do this with a lightweight, more severe (successful design depends on interlocking variable toolset that provides the appropriate affordances components). For this reason the 3D intuition gained by for several tasks without cluttering the workspace. In a user merging VR and tangible interface has an important role in study, practicing research scientists compared our interface nanotechnology. (which we call Silkworm1) to a traditional 2D mouse + FLEXIBLE TANGIBLE TOOLS monitor interface. Our survey shows they all preferred the Silkworm supports several operations. An early implemen- experimental interface – most enthusiastically so. Several tation had six physical tools, one per task, on the table sur- of them suggested that Silkworm is superior to pencil and face. Users were frequently confused, spending significant paper for thinking creatively about the possibilities of time managing physical devices and their associated wires. DNA. Overloading physical tools with local menus (similar to pie THE TASK [4] and marking [14] menus) solved this difficulty. A menu Figure 3, which shows a user completing a hairpin mole- button illuminates a halo of functionality options (see Fig- cule, illustrates the basic components of the modeling task. ure 6). A more prominent action button is used to activate The cones represent bases – groups of molecules that, for each tool. It is instructive to think of affordance and action the purposes of this study, can be thought of as atomic. separately: The tangible handle enhances spatial manipula- Between the bases are two types of bonds – Phosphate tion while the virtual presence supports tool variability. bonds (the thin lines along the curved region in the left of Multipurpose tangible handles present a practical future the photo) and Hydrogen bonds (the thick lines seen span- where a single setup can support many applications with ning the helix in the bottom right of the image). This the intimacy and spatial understanding that is the strength placement of atoms and bonds constitutes a design, whose of these interaction metaphors. fulfillment of project-specific goals (such as: can these We considered a number of other options for overloading molecules interlock to tile space?) is dependent on the physical tools. Selecting from a traditional menu (either physical plausibility of the distances and orientations of the floating in air or on the table surface) directs attention away molecules envisioned. To aid in this process, our software from the area of action. A selection device, such as a dial or contains a physical simulation which moves the bases into toggle button, forces users to remember which tool is cur- configurations based on the topological structure of the rently active to make quick selections (this information is bonds. quickly forgotten when a tool is put down). RAYGUN: PICKING POINTS IN SPACE We built a raygun tool by dismantling a toy gun, rewiring the trigger as the action button, inserting a motion tracker, 1 because delicate things are being woven Caltech CS TR 2002.011 Figure 5: DNA’s spatial complexity: The face of a DNA cube, drawn by a subject in the user study. Note that this structure consists of four double helices (dashed boxes on Figure 4: Removing bonds: The lightsaber (in right hand) left) that are precisely rotated so that they form a smooth cuts a bond between molecules while tongs, held in the left chain without kinks (highlighted on right). The existence hand, move the structure in space. and structural integrity of such chains is crucial to the sta- bility of DNA. and adding a menu button where the hammer would be (see Figure 8). The physical shape and cultural connotation of The raygun allows precise specification of the beginning this tool supports the metaphor of pointing at objects. A and endpoints of glue over a large volume with a small virtual beam emanates from the gun’s tip, adding precision motion. The glue is started by pointing the ray at the start- to this choice. This collocation of physical and virtual ob- ing point and clicking. Dragging the ray to the endpoint ject breaks down the boundary between bodily and digital and releasing the trigger completes the bond. space. Picking, as the ubiquitous mouse pointer attests, is a highly Historically, many VR interfaces do object selection by versatile operation. In future work, we hope to explore asking users to move a stylus so that it is collocated with an other applications such as: a flamethrower, which activates object in space [13]. Since tracking is often imprecise, and a local physical simulation; an ice gun, which freezes a more importantly the physical tool blocks the rendered ob- region to turn off simulation; and spraypaint which changes ject, picking in this manner breaks the illusion of immer- a region’s color. sion. Research on distant object selection [3] typically fo- GRIP: 3D LOCATION CONTROL cuses on objects beyond the arm’s reach. DNA bases are The grip (see Figure 8) enables the accurate positioning of not distant as much as they are small and numerous. By 3D objects in space. In Silkworm, this prop controls dou- providing the appropriate affordance, the gun allows for ble-helix drawing, single-strand drawing, and the lightsaber precise selection within a dense volume.