A Scientist's 3D Interface for Molecular Modeling and Animation

Waldon et al.

RESEARCH SketchBio: A Scientist’s 3D Interface for Molecular Modeling and Animation

Shawn M Waldon*, Peter M Thompson, Patrick J Hahn and Russell M Taylor II

that can still convey complex structures” and the second was to “Develop easy-to-use interfaces that permit facile control of models”. We’re developing SketchBio to help scientists think about 3D molecular structures Abstract and interactions and to communicate them to others. Background: Because of the difficulties involved We found ourselves repeatedly using 2D hand- in learning and using 3D modeling and rendering drawings of complex 3D structures and their inter- software, many scientists hire programmers or actions in discussions with our close collaborators in animators to create models and animations. This cell biology, pathology, and chemistry, despite the fact both slows the discovery process and provides that the 3D crystal structures of the proteins mak- opportunities for miscommunication. Working with ing up these structures were known. Hanging David multiple collaborators, we developed a set of design Goodsell’s excellent “Molecular Machinery” poster [1] goals for a tool that would enable them to directly showing renderings of many of these molecules and construct models and animations. their interactions helped to grow our shared under- Results: We present SketchBio, a tool that standing of the structures in the mitotic spindle ( a incorporates state-of-the-art bimanual interaction structure that separates chromosomes during cell di- and drop shadows to enable rapid construction of vision), but real progress was made when we hired an molecular structures and animations. It includes artist to draw 3D scale models of the structures each three novel features: crystal by example, week and then develop 3D computer models [2]. pose-mode physics, and spring-based layout that Our group is not alone. Discussions among collabo- accelerate operations common in the formation of rators are often done using 2D whiteboard sketches. molecular models. We present design decisions and Presentations often consist of pasted images and 2D their consequences, including cases where iterative Powerpoint animations. design was required to produce effective Because of the difficulties involved in learning and approaches. using 3D modeling and rendering software, many Conclusions: The design decisions, novel features, scientists hire professional computer programmers and inclusion of state-of-the-art techniques enabled and/or animators to work with them to create mod- SketchBio to meet all of its design goals. These els and animations rather than use these programs features and decisions can be incorporated into themselves. This indirection both slows the discovery existing and new tools to improve their process and provides opportunities for miscommuni- effectiveness. cation. As toolsmiths, our aim is to provide scientists with a tool that is so rapid to learn and powerful to Keywords: Molecular Modelling; Animation; use that they can create these models and animations Collision Detection themselves. We aim to produce a general tool that is widely useful. Many researchers studying cell structure and phys- Background iology seek to construct and evaluate dynamic models In his introduction to the “Cellular and Molecular that incorporate random thermal motion as well as Data” session at BioVis 2013, Tom Ferrin identified conformational changes induced through intermolec- two open challenges for molecular biology software. ular interactions. Discovering, testing, and communi- The first was to “Provide easy-to-use learning tools cating hypotheses about these interactions requires the development of complex animated 3D molecular struc- *Correspondence: [email protected] University of North Carolina at Chapel Hill, 27599 Chapel Hill, NC, US tures. Modeling, simulation, and rendering of these hy- Full list of author information is available at the end of the article pothetical scenarios involves using a number of tools Waldon et al. Page 2 of 14

and databases (PDB, Pymol, Blender, NAMD, etc.) The third driving problem comes from Kerry Bloom, and then converting files to pass geometry and anima- who is constructing models of the mitotic spindle, a tions between tools. It also involves manual placement structure that separates chromosomes during cell di- and orientation of 3D objects, which is currently done vision. As in the Lord case, each step of model gener- using clunky 2D input devices and by-hand detection ation has required support from an artist, animator, and avoidance of collisions. As a result, it often takes and/or programmer to convert Bloom’s concepts and a team months to produce an acceptable model or an- those of his students into geometry for rendering and imation. We aim to produce a tool that reduces this simulation. to a single person working for hours or days. Our final driving problem also comes from the Bloom This paper describes that tool, SketchBio. collaboration. Many proteins beyond cohesin and con- densin contribute to mitosis, and Kerry Bloom is in- Driving Problems terested in them all. His lab is able to fluorescently la- Fred Brooks points out that the best way to construct bel both these proteins and chromosome locations and a tool that is generally usable is to focus on several determine relative distances and orientations between very different specific problems and build a tool that pairs of proteins. With accurate localization and track- solves them [3]. We followed this approach. ing for 3D images, these techniques provide partial in- The first driving problem for this project is collab- formation on the 3D layout of proteins and chromo- orator Susan Lord’s desire to construct a protofibril somes in wild-type and mutant mitotic spindles. Build- model based on geometric constraints among a set of ing models to match this information requires the de- individual fibrinogen molecules. The protein fibrino- velopment of semi-automatic layout of proteins. This gen is the main component of blood clots, where it will provide a partial set of constraints for the scientist is converted into fibrin and links together with other to construct protein-protein and protein-chromosome fibrin molecules to form strands. Two of these strands complexes that match experimental data. With these join together to form a protofibril, which form thick enhancements, SketchBio will be widely useful to other fibers that make up a large portion of the blood clot. researchers for the generation of hypothetical protein- Based on the crystallized structures of fibrin monomers complex structures from partial data. from different species and on only two sets of known interactions [4], Lord sought to construct 3D protofib- Design Goals ril structures matching those seen in her data, which The application-specific needs from our several col- suggested a structure in which two fibrin strands twist laborators can be summarized as a set of domain- around each other, and wanted to create a model that independent design goals for SketchBio: shows this interaction at the molecular level. Over • Easy to learn and to use. Scientists must be able several months, she and her students worked with to rapidly construct models and animations on resource staff scientist Joe Hsiao to use the power- their own using interfaces that enable them to ful UCSF Chimera tool to construct such a model concentrate their mental efforts on the design [5]. Building this model required repeated iteration of challenge rather than decyphering the interface. hand-placement of two molecules (using multiple 2D • Support molecular operations. It must be easy mouse interactions), followed by using replication tools to load molecules, extract the relevant substruc- to develop candidate models, which were then evalu- tures, describe conformational changes, group ated against the data. Lord’s desired use of Sketch- molecules, and color according to standard data. Bio was to construct this protofibril rapidly and semi- • Appropriately constrain layout. Some molecu- automatically by specifying which location on each fib- lar structures should not overlap, others (drug vs. rin should be in close contact with other molecules protein) overlap as part of their function, others and by specifying that the molecules do not overlap. (fibrin, actin) assemble into repeated structures. This same capability will enable generation of other In some cases, the distances between individual self-symmetric structures such as actin filaments and elements is known but their 3D layout is not. Sup- microtubules. porting all of these cases will enable a biologist to Our second driving problem comes from Peter most rapidly explore the space of possible confor- Thompson in Sharon Campbell’s lab. Peter wants to mations to produce consistent models. construct 3D models and animations of the interac- • Support rapidly iterated, in-context design. tion between actin filaments and vinculin. Actin fila- Whether working alone or in a group, understand- ments are one of the three main components of a cell’s ing the interactions between dozens of molecules cytoskeleton, and the protein vinculin binds to actin requires repeated adjustment of proposed loca- filaments, connecting them to other actin filaments or tions and motions. The reasonableness of interac- different proteins. tions depends on nearby molecules, which change Waldon et al. Page 3 of 14

over time. We have found that generating consis- [15]. These each require the scientist to learn the un- tent models requires trying and optimizing many derlying complex rendering tool plus additional plug- potential solutions before the final model is found. in interfaces, making them less easy to learn and use. • Support high-quality rendering. Once a pro- None of these tools currently support constrained lay- posed model has been completed, static and ani- out along with rapidly-iterated, in-context design. mated images that use the most-effective lighting Molecular Flipbook [16] aims at similar goals to and surface rendering techniques are critical to SketchBio, providing an easy to use molecule-focused conveying the model and its behavior to others. real-time interaction environment coupled to offline rendering using Blender and FFMPEG. It does not Prior Work currently support constraints on layout or bimanual in- Molecular modeling: There are many excellent molec- teraction for rapid 6-degree-of-freedom placement. We ular modeling applications that have been extended to also considered this approach, but providing full capa- include some aspects of high-quality rendering and an- bility would require re-implementing many existing ca- imation. UCSF Chimera [6], Pymol [7], Graphite Life pabilities already available in molecular modeling tools Explorer[8], and Visual Molecular Dynamics (VMD) and tracking new features as they are developed. [9] are the most relevant. Other software such as Pro- tein Explorer [10] and EZ-Viz [11] (an interface for Interactive Animation: The Molecular Control Tool- Pymol) attempt to offer easy-to-use interfaces for ex- kit [17] is also aimed at molecular modeling, providing ploring molecular structures. gesture- and speech-based user interface primitives to control motions of molecules with a Kinect or Leap VMD includes direct force-feedback-based placement Motion device [17]; it provides an API that can be used and manipulation of molecules in the context of driv- to connect their controls to existing molecular modeling molecular dynamics. SketchBio provides bimanual ing applications. These do not by themselves meet the control of much larger sets of molecules by reducing needs of our collaborators, but could be used within the physics to only what is necessary to avoid im- SketchBio as a separate front-end interaction inter- proper collisions and provide appropriate spacing, en- face. SketchBio uses similar two-handed 6-degree-of- abling large-scale geometric modeling and animation. freedom input devices (the Razer Hydra or two Wi- GraphiteLifeExplorer includes the ability to position iMote controllers), adding collision detection and sev- and twist segments of DNA and interpolate the sec- eral custom capabilities, and tying the resulting system tions between them (its DNA modeling tools go be- into existing powerful molecular modeling and render- yond what is available in SketchBio), but does not ing tools to produce a complete system for thinking, yet perform collision detection between molecules, the modeling, and rendering. ability to support animation, or the ability to main- Another tool aimed at simplifying the creation of tain specified distances between objects needed by our molecular animations, PresentaBALL[18], uses an in- collaborators. teractive web interface to an existing molecular mod- We considered the approach of extending the inter- eling tool [18]. This allows widespread use by non- action and rendering capabilities of one of these tools, experts to develop presentation materials for training. but this would require re-implementing existing ren- SketchBio provides a custom interface for experts to dering techniques and continual updating as new ren- use as a thinking aid that is tied to a powerful render- dering advances are made. We instead decided to har- ing engine to produce animations. ness the power of the existing tools through their built- SketchBio’s bimanual 6-degree-of-freedom manipula- in scripting languages (SketchBio has used both Py- tion sets it apart from all of the applications described mol and Chimera to load, surface, select, and label above because it lets the user move molecules and craft molecules by partial charge and other inputs). animations more rapidly and with less mental effort than tools that use a mouse and keyboard to manipu- Rendering: There are also excellent general-purpose late objects. Its support of appropriately-constrained rendering programs (such as the commercial Maya and layout using several features (configurable collision de- open-source Blender applications) and microscope- tection, spring-based layout, and crystal by example) simulation rendering tools (such as UNC’s Microscope meet needs expressed by our collaborators that are un- Simulator [12]). Several groups are building molecule- met by any published tool. specific loaders that plug into these programs, such as Autofill/Autopack/Autocell [13], and Molecular Maya Interactive Rendering: A common bottleneck in in- [14]. The Bioblender package also leverages Blender for teractive modeling and animation applications is the molecular modeling and supports collision detection speed of rendering a complex scene. Sketchbio requires Waldon et al. Page 4 of 14

real-time rendering due to the nature of its input – ob- Mash-ups: Individual capabilities of web-based ap- jects on the screen must be move with the user’s hand plications such as Google maps and real-estate listing as if the user were actually holding them. databasese have been effectively combined to provide One approach to improving rendering speed is to re- combination tools that include the best parts of each. duce the complexity of the objects that are drawn. A system for seamless integration of applications for This is done by replacing objects with imposters which visualization was done by Rungta et. al by adding a have simpler geometry. One type of imposter is a sim- layer above all of the applications of interest to pass plified version of the geometry that is textured to look events back and forth [29]. SketchBio takes a similar like the more complex version [19, 20, 21]. Another approach, using a novel core component that provides common imposter is a square that has a pre-rendered interactivity and custom features but using scripting image of the more complex object as its texture. As interfaces to harness the significant modeling and ren- long as the viewpoint stays near the same position, dering capabilities of existing tools into a seamless discrepancies between the imposter and the actual ge- workflow. ometry are be small [22, 23]. The level of simplification of an object can also be Methods dynamically determined according to the amount of SketchBio is a system for understanding subcellular the allotted rendering time that is needed to draw each biology through the building of complex 3D macro- level of detail. molecular structures and animating the structures over Another approach to enabling interactive rendering time. of complex design spaces is to precompute an ensem- The modeling and rendering of these hypothetical ble of possible solutions and then interactively explore structures currently involves using a number of tools the design space by directly manipulating portions of and databases and converting files and data to pass it and morphing between existing solutions [24]. The between tools. space of potential molecular interactions for dozens of SketchBio harnesses state-of-the-art tools and li- molecules is so large, and the ease of testing and ren- braries into a seamless workflow. It brings best- dering each configuration so small, that it was more practice interaction and display techniques to bear efficient for SketchBio to directly model and render. on molecular modeling, including bimanual real-time SketchBio uses Chimera and Blender to simplify ge- direct interaction and shadow-plane depth cues. It ometry and the Visualization ToolKit (VTK) library adds three novel features that accelerate this workflow: to adjust rendered level of detail [25]. crystal-by-example, pose-mode physics, and spring- based connectors. Its design decisions (a direct-manip- Collision Detection: In many models and animations, ulation, real-time interface; harnessing tools rather molecules should not overlap one another. If there are than re-implementing techniques; and making a sys- n molecules in the scene, then each pair of molecules tem usable in the scientists’ labs) led to a system that must be tested for collision. This has a complexity of met all of the design goals. O(n2) in the number of molecules. However, there are typically far fewer collisions than potential collisions System Overview and so optimizations can reduce the expected com- Figure 1 shows a screenshot of the SketchBio user in- plexity. The best expected complexity uses sweep and terface with a group of three actin molecules (left) and prune methods and assumes the primitives are sorted the tail region of a vinculin molecule (right). SketchBio along one dimension. This is O(n + c) where c is the uses imposters with simplified surface geometry while number of colliding pairs [26]. developing the animation, but uses full resolution mod- Another approach uses space partitioning to rule out els for final rendering. The small white spheres follow unnecessary tests. The PQP library from the UNC the two tracked hand-held controllers. Status informa- GAMMA group uses a bounding volume hierarchy tion is shown in the lower left. The current animation [27]. An alternate is to divide space into bins. Only time point is shown in the lower right. primitives in nearby bins need to be tested. This type Molecules in SketchBio are represented as rigid sur- of algorithm is especially effective on GPUs where faces approximating the Connolly Solvent-Excluded many local groups may be run in parallel [28]. Surface of the molecule. These were chosen because SketchBio directly links to PQP and uses it for basic our collaborators used surface models in their current collision detection. It extends these techniques in ways work for all four driving problems. The surfaces may that are specific to the kinds of molecular models be- use solid colors, be colored by surface charge, or be col- ing formed to gain an additional order of magnitude ored by their nearness along the protein backbone to reduction in collision tests for some objects. the N-terminus or C-terminus of the protein. Chimera Waldon et al. Page 5 of 14

is used to calculate and export the datasets for the latter two coloring schemes using PDB data. Object selection is indicated by drawing the outline of the oriented bounding box of the selected molecule molecules. Color of this outline indicates whether a group or single object is selected. (An earlier design showed the selected object in wireframe, but this was found to disrupt perception of the orientation of the molecule.) A set of “gift ribbons” drawn on the oriented bounding box indicates that an object has a keyframe at the current time. This was chosen to minimally obscure the molecule and selection indicators.

Figure 2 The left hand sets the base molecule while the right hand positions the copies in this two-handed construction of an actin ﬁber.

SketchBio uses a pair of Razer Hydra controllers to provide two 6-DOF trackers, each of which also has several buttons, a hi-hat controller, and an analog input. This enables a very expressive set of verbs (buttons), nouns (selection via 3-DOF positioning), and adjectives (magnitude via analog inputs, viewpoint via hi-hat, and pose via a combined 12-DOF tracking). This avoids the need for the system to recognize a large set of ambiguous gestures, as is the case for video-based user input. Use of this device enables the interface for moving objects to mirror a task users are already familiar with, namely reaching out, grabbing an object Figure 1 A screen shot from SketchBio showing three actin and moving it to a new position and orientation. monomers on the left colored yellow and the tail region of the Using one of the buttons to switch between modes vinculin protein on the right colored by surface charge. provides a sufficiently-large space of commands that almost all operations can be performed without putting down the controllers. The keyboard and mouse are used to name proteins and files on initial loading, and State-of-the-art Capabilities to set precise values as needed for one or two opera- Bimanual Interaction: Bill Buxton and others have tions. described the benefits of two-handed (bimanual) interaction. He and others observed that bimanual ma- Shadow Plane: Because selection in SketchBio re- nipulation brings “two types of advantages to human- quires placing the tracker within the bounding box computer interaction: manual and cognitive. Manual of the object, determining the relative depth be- benefits come from increased time-motion efficiency, tween tracker and object is an important and often- due to the twice as many degrees of freedom simulta- performed task. Initial testing of the application re- neously available to the user. Cognitive benefits arise vealed that determining the relative depth between an as a result of reducing the load of mentally composing object and the tracker or between two objects was and visualizing the task at an unnaturally low level the most difficult part of using SketchBio. Because imposed by traditional unimanual techniques” [30]. widespread adoption would be limited by requiring As seen in Figure 2, SketchBio brings bimanual in- stereo displays and head tracking, we sought another teraction to the construction of macromolecular struc- solution. tures. The entire interface is built around a set of Hendrix and Barfield found the most effective tech- world and root-object manipulation controls in the niques for aiding in depth estimation are a textured non-dominant hand and a set of individual-element plane and lines dropped from the center of an object manipulation controls using the dominant hand. to the textured plane [31]. To provide additional depth Waldon et al. Page 6 of 14

Figure 3 A screenshot from SketchBio showing colored molecules and a diﬀerent camera angle to emphasize the Figure 4 A screenshot showing the complexity of Blender’s shadow plane’s eﬀect. user interface.

cues, SketchBio displays a ground plane that is always splines to produce smooth motion and changes. The rendered below the viewpoint no matter the direction effects of this interpolation can be easily seen by the or position of the viewpoint and projects the shadows user by moving along the timeline or using the built-in of objects onto this plane. The trackers also cast shad- animation preview. The scene is exported to Blender ows onto this plane (which are darker and larger to with a set of predefined global settings for effects and highlight them). SketchBio assumes a light infinitely position of light sources to produce a full-quality ren- far away in the default camera’s up direction which dering. gives the same absolute position against the textured surface as the drop-lines while also giving information Grouping: Grouping of molecules eases construction about how close the boundaries of two objects are to of larger order structures and provides smooth anima- each other. The user can also rotate the camera while tion of objects that should moving together without leaving the light and shadow plane fixed to get a better the small variations that even the most careful hand understanding of the scene through motion parallax placement causes. Copy and paste is also implemented [See Figure 3]. (both single objects and groups can be copied and pasted) even between sessions. Additionally, a group Animations: For scientists creating animations of of molecules constituting a structure that a user wants molecules, SketchBio provides a basic interface to a to use multiple times in different projects can be saved much more complex system. Blender is a production and then imported, eliminating the need to rebuild level animation and rendering tool that has an ex- large structures. Molecules can be added to groups or tremely complex user interface with dozens of hotkeys, removed from them at keyframes. menus and buttons (see Figure 4). Blender also has a Python scripting interface that provides access to all of its functionality. SketchBio uses this scripting in- Importing Molecules: SketchBio generates molecu- terface to create its animations and render them in lar surfaces using UCSF Chimera via Python script- a high quality rendering engine, but provides a much ing. A custom plugin (ExportVTK) was written for simpler user interface. SketchBio provides a set sim- Chimera’s Python interface to export additional data ple operations that is sufficient to meet the animation from Chimera in the VTK file format. This plugin was needs of our driving problems: moving along the video contributed back to the Chimera developers and is now timeline, setting keyframes on objects and viewing a part of the standard source distribution. This data in- low resolution animation preview. cludes residue and chain identifier that map to a spe- Keyframes can modify color and grouping infor- cific location on the surface and electrostatic potential mation as well as object position and orientation. on the surface. SketchBio can use these data sets to These values are interpolated between keyframes using color the objects (see figure 1). Waldon et al. Page 7 of 14

Novel Capabilities To meet the needs of our collaborators, SketchBio supports novel operations beyond those available in the programs and libraries that it harnesses. These include “pose-mode physics” that enables rapid docking of one protein with others, a “crystal by example” mode that enables rapid formation of polymer molecular chains, and spring-like connectors to maintain expected distances between molecules. Each of these is described, along with how they enable optimization of collision detection.

Pose Mode Physics: Object motion in SketchBio is accomplished by applying forces and torques to pull towards the tracker location and orientation. This can result in the object lagging behind but also smoothes motion, especially rotation. Standard rigid-body dynamics was used as the origi- Figure 5 Crystal-by-example illustrating how a helix might be formed. nal collision response in SketchBio. Because the manipulated object pushed other objects around, this caused difficulty in assembling molecular groupings. This was solved by introducing “pose-mode physics”, generate chains of molecules. Each replication of the where the only objects that move are those directly base molecule follows the example set by the first two being manipulated. other objects do not move when molecules, with the third molecule’s placement rela- collision response forces are applied. This also greatly tive to the second molecule being the same as that reduces the time taken to compute collision detection of the second molecule to the first, and so on. In this (as described later). way, a repeated structure is formed by manipulating The first implementation of pose-mode physics onle only one pair of molecules rather than tediously mov- moved the object if its new location after being pulled ing each individual piece to its proper place, speeding by the tracker-attracting forces would be collision free. up the process of building structures. This caused objects to become stuck together and dif- This feature uses two copies of a molecule (A and ficult to pull apart because tracker rotation usually in- B) to define an entire repeated structure. Given TA troduced collisions even as they forces pulled objects and TB, the transformation matrices that define the apart. This also prevented sliding objects along each positions of A and B relative to the world origin, the other, which scientists often wanted to be able to do. transformation from A’s coordinate system to B’s co- In the final implementation, where collision response − ordinate system, TAB = T 1 ∗ TB, can be computed. forces act on the object being manipulated, objects can A B’s position can be rewritten TB = TA ∗ TAB. The be slid along one another but not collide. next repeated molecule, C, has position TC = TB ∗ T = T ∗ T 2 . This can be extended to generate a Crystal-by-example: Repeated structures that formed AB A AB by replicating a single protein are common in biology chain including an arbitrary number of molecules. (actin, microtubules, fibrin, etc.), so the “crystal-by- Many biological structures including actin fibers and example” feature was added to support their construc- microtubules (major components of a cell’s cytoskele- tion. Our collaborators wanted to construct variants ton) form in structures that can be defined this way. of such structures to study the changes caused by mu- Figure 6 shows an actin fiber generated this way in tant proteins and to understand their native packing SketchBio. By providing live updates of the entire for comparison to electron microscopy images. structure as the initial two objects are manipulated, A similar problem is addressed in [8] for DNA SketchBio lets the scientist explore potential structures molecules by letting users edit placement and twist of in real time. selected base pairs and interpolating in between these. The extent to which the user can control fine-grain That system forces the resulting structure to follow a maniuplations of the molecules depends on the input specified path. Crystal-by-example inverts this to show device in use, because resolution varies by device. Be- the structure resulting from a specified packing geom- cause some structures have a known transformation etry: the user places two molecules relative to one an- from one molecule to the next, SketchBio (like other other in six degrees of freedom and SketchBio repeat- programs) lets the user input the transformation di- edly applies same transformation for other copies to rectly. Waldon et al. Page 8 of 14

Collision Detection in Pose Mode Physics and Crystal- by-example: In pose mode, collision tests between objects that the user is not interacting with can be skipped because these objects do not move. This means that only collisions involving the objects that the user is moving need to be checked. This reduces the number of collision tests to m ∗ n where m is the number of objects that the user is currently moving. The typical number of objects that the user moves at a time is 1 or a small constant (in the case of moving a group), which reduces the number of collision tests needed to O(n) in this expected case. There are two ways that the user can interact with a crystal-by-example structure: moving the entire structure as a unit, or adjusting the internal transformation to change the shape of the structure. In the first case, only collision tests between the structure and the other Figure 6 Actin filament created with the crystal-by-example objects in the scene need to be done, and the above function using the transformation matrix from the PDB data from one monomer to the next. bound applies to the number of tests. In the second case, the internal structure does change and both internal and external collisions must be tested. External collisions must test every object in When acting as springs, connectors can have non- the structure with every external object as above. zero rest length. When editing a set of proteins some of In internal case we can leverage the known relation- whose separations are known experimentally (through ship between the objects to perform fewer tests. Let Xi two-color fluorescence labeleing, FRET, or other tech- be the ith object in the crystal by example structure niques as in our final driving problem), this can be with X1 and X2 being the two base object in the struc- used to specify soft constraints on the 3D layout of ture. Let Ti,j be the transformation matrix from Xi to the proteins, guiding the scientist away from impossi- Xj. The definition of the crystal-by-example structure ble structures. This greatly reduces the conformation is that Ti,i+1 is the same for all i and the geometries of space that must be searched to determine molecular all the Xis are the same. Because the geometries and arrangements. transformations are the same, if there is a collision between the ith and (i + 1)th objects anywhere in the Architecture structure, then there is also a collision between the 1st The architecture of SketchBio is shown in Figure 7. and 2nd objects. Thus testing only this one pair per- SketchBio harnesses external programs when possi- forms the work of n-1 tests where n is the number of ble (PyMOL, Chimera, Blender) and uses existing li- objects in the structure. This same argument holds for braries for other core functions (VTK, PQP, VRPN). any i and i + k, the 1st and (k + 1)th objects have the It maps from dozens of controls in Chimera and hun- same relative positions and the same collisions. Thus dreds of controls in Blender down to 4 input options only the 1st object in the structure needs to be tested and about 20 modeling and animation controls to against the others which allows O(n) tests to suffice streamline the tasks needed for creating structures and for all internal collisions in a repetitive structure of n animations. elements. Exporting data to Blender is done by writing a script to be run on Blender’s Python interface to produce Connectors: SketchBio also has connectors that can the animation. When exporting to MicroscopeSimula- be added between objects. These can act like springs tor, SketchBio writes out a Microscope Simulator XML and apply forces to keep objects positioned relative project file and loads the project into MicroscopeSim- to each other or they can simply indicate that two ulator. objects are connected. Many proteins have regions for Objects can be loaded into SketchBio as .obj files which the structure is unknown and these regions can from any program that writes this format or directly be represented with these connectors. Responding to a through the GUI (via harnessing UCSF Chimera from scientist’s request, the connector end can be snapped the PDB or a local .pdb file). Because VTK is used in to the N-terminus or C-terminus of a protein, removing SketchBio, any file format that VTK can read could the difficulty of precise hand placement. be imported with relatively minor changes. Waldon et al. Page 9 of 14

Figure 7 Architecture. SketchBio harnesses existing libraries and programs (shown in pink) to avoid replicating existing state-of-the-art algorithms. It also makes use of standard file formats, devices, and services (shown in purple) to provide maximum interoperability with existing modeling, rendering, and analysis workflows. Some techniques are internal, some are harnessed to appear to the user as internal (double arrows) and some are accessed via standard formats. SketchBio currently includes three types of output: real-time rendering for model and structure comprehension, high-quality offline rendering for animation (through Blender), and simulated confocal microscopy stacks for analysis and comparison to experiment (through UNC’s Microscope Simulator). It includes custom code only for the real-time interaction, animation, and modeling portions and for its novel features.

Design Decisions and controls on the two hand-held controllers. Finally, We list here some design decisions that helped Sketch- saving the file and rendering are again performed with Bio achieve its goals. the keyboard and mouse. The use of rich input devices enables the bulk of the action to take place from within Bimanual, 6-DOF interface: SketchBio’s two-handed the 3D environment, accelerating the most-challenging interface differs from that of most existing modeling parts of model and animation development. and rendering tools. This has the deficit of taking the user’s hands away from the keyboard, which requires Harness, don’t re-implement: The design of Sketch- them to put down the interaction devices to enter text Bio avoids reimplementing existing features where pos- and specific numerical data. Our collaborators report sible, instead using Python scripting to control subpro- that this small negative is greatly outweighed by the cesses to perform these operations. When reading in ability to rapidly perform the more-common and more- PDB files, instead of writing a PDB file reader, Sketch- challenging tasks of specifying positions, viewpoints, Bio calls UCSF Chimera as a subprocess to read in and animations in full 6 degrees of freedom. The abil- the protein and create a displayable surface from it. ity to move both the world/viewpoint and an animated Instead of writing a new rendering library, SketchBio molecule enables rapid planning of scenes and the abil- uses the Python scripting interface of Blender to create ity to simultaneously manipulate both of the molecules a Blender project that will produce the desired anima- that are coming together in an interaction are two ex- tion. SketchBio uses the open source Qt and VTK[25] amples of what is enabled. libraries for its user interface and internal rendering The workflow tends to stratify: initial loading of the and the open source Proximity Query Package (PQP) kinds of molecules to be used in an animation happens for collision detection [27]. The VRPN library is used first (with keyboard and mouse). Then positions, view- to communicate with the Razer Hydra input device points, and animation are described using the buttons [32]. Waldon et al. Page 10 of 14

One significant risk encountered when harnessing ex- to our laboratories to use them received great benefit, isting programs is that future versions of the programs but we wanted SketchBio to be more broadly avail- will not support required features, or will require mod- able. To maximize its impact, SketchBio is designed ifications to the harness. This can make maintainence to run on a laptop or desktop system such as a scien- challenging. To address this, each SketchBio release tist would have at home or in their laboratory and to includes a list of specific versions of the wrapped pro- use inexpensive commercial input devices. grams which with it is known to be compatible and have selected programs to harness that continue to Results and Discussion make old versions available (Chimera still releases in- SketchBio has been used by a several scientists and stallers from 2002 and Blender from 2003). It also in- has demonstrated success in meeting its design goals. cludes copies of custom plug-ins and scripts that are Easy to learn and use: To measure the ability of sci- not yet part of the harnessed packages’ released ver- entists to learn and use the system, we showed Sketch- sions. Bio to a visiting graduate student from NIH. She is Another risk is that the packages used will not be interested in using the system to study the proteins in- obtainable in the future, or for an operating system of volved in cell focal adhesions. After a 30-minute train- interest. SketchBio has been able to mitigate this risk ing session where she saw us using the system, she was by selecting open-source programs to harness. able to use SketchBio to load, replicate, and place the To measure the re-use of functionality, we com- molecules into relevant configurations. pare (1) the number of state-of-the-art operations har- After similar initial training, and with access to the nessed from existing tools: Chimera (connecting to the manual, biochemistry graduate student Peter Thom- protein data bank, parsing PDB file, selecting sub- son used the system to generate both static and ani- units, generating surfaces, generating data sets on the mated multi-protein models. surfaces, simplifying surfaces), Blender (surface ren- Peter created a model to compare the importance of dering, directional illumination, transparency, ambi- electrostatics between two different models for vinculin ent occlusion, parallel rendering, frame storage), and tail interaction with actin [40][41]. Microscope Simulator (point-spread-function 3D blur- He also created an animation of vinculin binding to ring, TIFF stack generation) and (2) the number of an actin fiber for use in a talk, based on the model internally-used existing libraries: VRPN (reading from presented in [42]. This video used crystal-by-example general peripheral devices), PQP (multi-object colli- to generate the actin and used traslucent connectors sion detection), VTK (geometric operations, real-time to indicate the connection between the head and tail rendering, level-of-detail rendering, object positioning, domains of vinculin – a region for which there is no spline interpolation) to (3) the number of custom op- crystal structure. The model in SketchBio is shown erations (crystal by example, pose-mode physics, drop in Figure 8 and a frame from the resulting video at shadows, bimanual interaction modes, spring connec- approximately the same time is shown in Figure 9. tors, grouping and animation). We find that most of Peter produced both a SketchBio animation and a the operations are supported by existing tools. We Microsoft Powerpoint animation of molecules (using compare this against other tools we have built to images of molecules pre-rendered from a single view- support biomedical applications [33]. SketchBio has point), as shown in figure 10, to test their relative a much better re-use ratio than tools which simi- speed and effectiveness. The Powerpoint animation larly span different domains (nanoManipulator, Cam- took 50 minutes of concentrated effort to produce, era Calibration, Chromatin Cutter, Template-Based while the SketchBio animation took 100. He reports that the Powerpoint animation failed to accurately Matching) and is on par with tools that are basically show rotation of the vinculin tail domain, to show the wrappers for calls to a single library (ImageTracker, linker region that scales as the domains move apart, Microscope Simulator). It bests several of our single- to show a change in actin movement rate, and to ac- domain tools (Video Spot Tracker, Video Optimizer, curately portray relative size and orientation of the and WebSlinger). Furthermore, the scripting interfaces molecules. He reports that the increase in correct pre- enable rapid inclusion of additional features from ex- sentation of the science was was well worth the internal programs without re-implementation. creased time.

Usable in-house: We have in the past built high- Support rapidly-iterated, in-context design: To mea- performance molecular graphics applications for sci- sure the speed of complex model construction, we re- entists that used head-tracked stereo, wide-area track- peated a task using SketchBio that had been done being systems, and force-feedback displays [34, 35, 36, forehand. Constructing the protoﬁbril models for Su- 37, 38, 39]. The scientists who were willing to travel san Lord took resource computer scientist Joe Hsiao Waldon et al. Page 11 of 14

Figure 10 A frame from a 2D animation created by Peter Thompson for the same case as ﬁgure 9. This video was Figure 8 A scene from a video created by Peter Thompson in created in Microsoft Powerpoint using separately-rendered SketchBio. Approximately the same timestep is shown single views of the proteins. rendered at its full resolution in Figure 9

create an animation of actin and vinculin (see supple- mental materials). We were able to load the molecules, replicate them, place them, plan camera and motion paths, and start rendering in half an hour. The first- person design view and available pre-animation were crucial to this process, enabling design intent to be rapidly translated into action and evaluation, resulting in uninterrupted planning and design iteration. These cases indicate that a series of brief training videos plus the online manual should suffice to get new users started, that scientists are able to use Sketch- Figure 9 A frame from the video created by Peter Thompson. Bio on their own, and that SketchBio compares favor- This shows the tail domains of vinculin binding to an actin ably to existing methods of producing animations and filament and slowing its motion. This video was created in SketchBio as seen in Figure 8 and rendered via the export to structural models. Blender feature. Support molecular operations: The video in supple- mental materials shows that a user who is familiar with 3-3.5 hours by hand-editing transformations within both tools is able to load, select subsets, and attach Chimera (a task challenging to teach to biologists). two molecules six times as fast using SketchBio as us- Using an early prototype of SketchBio, he constructed ing the combination of Chimera and Blender. As part the protofibril seen in Figure 11 in 1.5 hours (a task of development, we created a Chimera plug-in that we’d expect a collaborator to do just as rapidly). The exports the standard molecular labelings (main-chain lack of depth cues became apparent as Joe spent most index, partial charge, etc.) in a VTK data structure, of the time trying to figure out the relative depth be- enabling them to be used to color the molecules. Sci- tween the tracker and the molecules, prompting us to entist are able to use familiar PDB file and substruc- add the shadow plane. With this addition and other ture names to load and extract subsets of molecules. features, Joe reconstructed the model in 35 minutes. The animation and object-grouping features have been In all cases, the desired model was known a-priori; all used by our collaborators to produce models and ani- cases measure time on task and do not count the time mations meeting their needs. spent learning how to use the tool. In this case, Sketch- Bio enabled model creation in about one-fifth of the Appropriately constrain layout: Pose-mode physics, time for a case of interest to our collaborators. with the option to turn it off, supports both preventing To further measure the effectiveness of SketchBio for and allowing overlap between molecules, as appropri- the rapid construction of animations, we used it to ate to the task. The crystal-by-example feature has Waldon et al. Page 12 of 14

complex scenes will also require the ability to label important molecules. The motion of objects could be changed to directly map the user’s hand motion instead of moving toward it via force and torque being applied. This would provide a direct mapping of hand location to object location and possibly a better interface. This could be combined with the collision detection type where objects are only allowed to move to a location if the result is collision free. The disadvantage of this approach is that the smoothing by the forces and torques will not occur; transmitting any jitter in the device input directly to object motion. SketchBio currently supports only rigid structures for modeling and collision detection. For the collabo- rations we have worked with deformable models were Figure 11 A view of the model Joe Hsiao created with not needed and we felt our time was better spent pro- SketchBio to compare usibility with UCSF Chimera viding them with features they had requested. Molecular dynamics simulation is something Sketch- Bio does not do directly. This decision was motivated been used to produce both protofibrils and actin fila- by the cost of performing the molecular dynamics and ments from their monomers. Fixed-length springs pro- the requirement to provide real-time user interaction. vide the ability to rearrange sets of moluecules while SketchBio may eventually harness an external molec- maintaining the specified pairwise separations among ular dynamics simulator, but SketchBio will only be them. used to specify input configurations for the simulation or easily create videos from its output. While Sketch- Bio will not support molecular dynamics directly, a Support high-quality rendering: The image frames in molecular docking capability involving two individual the paper and video in the associated online materials molecules could be added. demonstrate full-resolution rendering with intra- and To avoid dependence on a particular hardware ven- inter-object shadowing displaying both complex local dor, SketchBio is being actively ported to use a pair of shape and 3D relationships between objects. Nintendo WiiMote controllers instead of the Razer Hy- dra controller. Its use of the VRPN library supports Limitations and future work switching devices by renaming the device and input After successfully using SketchBio for his initial needs, for each function; a general-purpose mapping layer has Peter has requested new features. He is particularly been added that reads from a configuration file to en- interested in using SketchBio as a thinking tool to able the user to customize this remapping. This en- determine how mutations in vinculin turn normally- ables new SketchBio users to continue to use the tool straight actin filament bundles into helices. Forming a until the next-generation Razer Hydra is released. model to fit experimental data can is challenging. We One consequence of the choice to provide a uni- are working on a module to optimize the placement of form environment that wraps functions from other pro- molecules based on a set of constraints. The resulting grams is that not all features of the wrapped programs optimization algorithms will enable other scientists to are available from within SketchBio. This limitation is semi-automatically construct multi-protein structures mitigated by enabling the user to export Blender files that match negative stain electron microscopy images. for later offline rendering and to import arbitrary ge- Our collaborators’ projects have so far involved ometry, but then the user has to learn the complexities hand-placed molecules of density sufficiently small to of the other tools to use these features. If it is the case be understood when all of them are visible. Thus, that most of these features are needed the interface to SketchBio does not yet support automatically-placed SketchBio will eventually become as complex as the molecules to fill the space, nor does it require com- sum of the tools it wraps. Because our collaborators plex occlusion-handling procedures. As the user base have been able to develop models and animations with- grows, we expect to need to harness importance-based out using most of the tools, and because most of the rendering techniques and autofill algorithms to handle rendering settings are not needed for molecular anima- a large number of background molecules. These more tions, we do not anticipate this happening in practice. Waldon et al. Page 13 of 14

Conclusions can be found at http://sketchbio.org. SketchBio is be- SketchBio is a new tool that enables scientists to ing released as open-source software and will be avail- rapidly construct and validate hypothetical macro- able for download at this location. molecular structures, to animate these structures, and to produce high-quality rendered animations. It has Acknowledgements been tested and shown to meet its design goals: This work was supported by NIH 5-P41-EB002025. • Easy to learn and to use. Scientists were able to Molecular graphics and analyses were performed rapidly construct models and animations on their with the UCSF Chimera package. Chimera is devel- own. oped by the Resource for Biocomputing, Visualization, • Support molecular operations. By harnessing and Informatics at the University of California, San Pymol and Chimera. Francisco (supported by NIGMS P41-GM103311). • Appropriately constrain layout. Configurable col- 3D rendering was performed using Blender (blender.org). lision detection, fixed-length springs, and crystal- Blender is an open source project supported by the by-example support all listed cases. Blender Foundation and the online community. • Support rapidly iterated, in-context design. Early versions of SketchBio used PyMOL (py- Real-time 6-degree-of-freedom interaction, live mol.org) to import data from the PDB. PyMOL is animation preview, and viewpoint control enable an open source project maintained and distributed by embedded design. Schrödinger. • Support high-quality rendering. By harnessing We thank Ping-Lin (”Joe”) Hsiao for evaluating the effectiveness of SketchBio on the construction of Blender. protofibrils. We also thank our collaborators Susan SketchBio includes state-of-the art bimanual inter- Lord, Sharon Campbell, and Kerry Bloom and the rest action, drop shadows to improve depth perception, of the UNC CISMM group. and other standard modeling and animation behaviors (grouping, spline interpolation, level-of-detail rendering, rapid collision detection, real-time preview). List of abbreviations used PQP: Proximity Query Package, VRPN: Virtual Reality Peripheral Network, SketchBio also includes novel interaction and compu- PDB: protein data bank. tational techniques that directly support the construc- Competing interests tion of macromolecular structures. Crystal by example The authors declare that they have no competing interests. and pose-mode physics both provide improved model- References ing capabilities and both enable more-rapid collision 1. Goodsell, D.S., Bamber, G.W.: Molecular Machinery: A Tour of the detection. Spring connectors show unspecified inter- Protein Data Bank. http://www.rcsb.org/pdb/static.do?p=general information/news pub actions and support semi-automatic structure forma- lications/newsletters/2004q1/poster available.html . Last accessed: tion. These novel capabilities can be added to existing 2014-3-15 (2004) molecular modeling tools and included in future tools, 2. II, R.M.T., Bloom, K.: In: Frankel, F., DePace, A.H. (eds.) Intersecting Art and Computer Graphics, pp. 102–107 (2012) providing the same acceleration of model building and 3. Brooks, F.P.: The computer scientist as toolsmith-studies in interactive evaluation. computer graphics. In: Gilchrist, B. (ed.) Information Processing 77, Both crystal-by-example and pose-mode physics en- Proceedings of IFIP Congress 77, pp. 625–634. North-Holland, Toronto (1977) able real-time collision-detection to scale to much 4. Lord, S.T.: Fibrinogen and fibrin: scaffold proteins in hemostasis. larger collections of molecules than are possible us- Current opinion in hematology 14(3), 236–241 (2007) ing existing techniques that must check for collisions 5. Huang, L., Ping-Lin, H., Taylor, R.M.I., Lord, S.T.: ”snapshots” and modeling of the early stages in fibrin polymerization. (submitted) among all objects. The ability to load arbitrary ge- 6. Pettersen, E.F., Goddard, T.D., Huang, C.C., Couch, G.S., Greenblatt, ometry files enables the tool to scale beyond molecule D.M., Meng, E.C., Ferrin, T.E.: Ucsf chimera - a visualization system types that can be found in the protein data bank. for exploratory research and analysis. Journal of computational chemistry 25(13), 1605–1612 (2004) The design decisions (a direct-manipulation, real- 7. LLC, S.: The PyMOL Molecular Graphics System. Computer Program, time interface; harnessing tools rather than re-imple- PyMOL version 1.5.0.4 (2013) menting techniques; and making a system usable in 8. Hornus, S., Lévy, B., Larivière,D., Fourmentin, E.: Easy dna modeling and more with graphitelifeexplorer. PloS one 8(1), 53609 (2013) the scientists’ labs) led to a system that met all of 9. Humphrey, W., Dalke, A., Schulten, K.: Vmd - visual molecular the design goals and is being used in our collabora- dynamics. Journal of Molecular Graphics 14, 33–38 (1996) tors’ own labs. The relative benefits of these decisions 10. Martz, E.: Protein explorer: easy yet powerful macromolecular visualization. Trends in Biochemical Sciences 27(2), 107–109 (2002) outweighed there potential pitfalls, making them likely 11. Grell, L., Parkin, C., Slatest, L., Craig, P.A.: Ez-viz, a tool for choices for other designers. simplifying molecular viewing in pymol. Biochemistry and Molecular SketchBio is built using portable libraries and has Biology Education 34(6), 402–407 (2006) 12. Quammen, C.W., Richardson, A., Haase, J., Harrison, B., Taylor II, been compiled and used on Windows, Mac OS X, and R.M., Bloom, K.S.: FluoroSim: A Visual Problem Solving Environment Ubuntu Linux. Descriptions and videos of SketchBio for Fluorescence Microscopy (2008) Waldon et al. Page 14 of 14

13. Johnson, G.: Autofill Web Page. http://www.grahamj.com/ (2013) 37. Taylor II, R.M.: Scientific applications of force feedback: Molecular 14. Maya, M.: Molecular Maya Web Page. simulation and microscope control. In: SIGGRAPH ’99. Course Notes http://www.molecularmovies.com/toolkit/ (2013) for ”Haptics: From Basic Principles to Advanced Applications” 15. Zini, M.F., Porozov, Y., Andrei, R.M., Loni, T., Caudai, C., Zoppè, 38. Taylor II, R.M., Chen, J., Okimoto, S., Llopis-Artime, N., Chi, V.L., M.: Bioblender: Fast and efficient all atom morphing of proteins using Frederick P. Brooks, J., Falvo, M., Paulson, S., Thiansathaporn, P., blender game engine. arXiv preprint arXiv:1009.4801 (2010) Glick, D., Washburn, S., Superfine, R.: Pearls found on the way to the 16. Flipbook, M.: Molecular Flipbook. http://molecularviewbook.org/ ideal interface for scanned-probe microscopes. In: Visualization ’97, pp. (2013) 467–470. IEEE Computer Society Press 17. Sabir, K., Stolte, C., Tabor, B., O’Donoghue, S.I.: The molecular 39. Taylor II, R.M., Robinett, W., Chi, V.L., Frederic P. Brooks, J., control toolkit: Controlling 3d molecular graphics via gesture and voice. Wright, W.V., Williams, R.S., Snyder, E.J.: The nanomanipulator: A Visualization and Computer Graphics, IEEE Transactions on (2013) virtual-reality interface for a scanning tunneling microscope. In: 18. Nickels, S., Stöckel, D., Mueller, S.C., Lenhof, H.-P., Hildebrandt, A., SIGGRAPH 93, pp. 127–134. ACM SIGGRAPH Dehof, A.K.: Presentaball–a powerful package for presentations and 40. Janssen, M.E., Kim, E., Liu, H., Fujimoto, L.M., Bobkov, A., lessons in structural biology. Visualization and Computer Graphics, Volkmann, N., Hanein, D.: Three-dimensional structure of vinculin IEEE Transactions on (2013) bound to actin filaments. Molecular cell 21(2), 271–281 (2006) 19. Décoret, X., Durand, F., Sillion, F.X., Dorsey, J.: Billboard clouds for 41. Thompson, P.M., Tolbert, C.E., Shen, K., Kota, P., Palmer, S.M., extreme model simplification. In: ACM Transactions on Graphics Plevock, K.M., Orlova, A., Galkin, V.E., Burridge, K., Egelman, E.H., (TOG), vol. 22, pp. 689–696 (2003). ACM et al.: Identification of an actin binding surface on vinculin that 20. Erikson, C., Manocha, D.: Simplification culling of static and dynamic mediates mechanical cell and focal adhesion properties. Structure scene graphs. UNC-Chapel Hill Computer Science TR98-009 (1998) (2014) 21. Cohen, J., Olano, M., Manocha, D.: Appearance-preserving 42. Thievessen, I., Thompson, P.M., Berlemont, S., Plevock, K.M., simplification. In: Proceedings of the 25th Annual Conference on Plotnikov, S.V., Zemljic-Harpf, A., Ross, R.S., Davidson, M.W., Computer Graphics and Interactive Techniques, pp. 115–122 (1998). Danuser, G., Campbell, S.L., et al.: Vinculin–actin interaction couples ACM actin retrograde flow to focal adhesions, but is dispensable for focal 22. Aliaga, D.G.: Visualization of complex models using dynamic adhesion growth. The Journal of cell biology 202(1), 163–177 (2013) texture-based simplification. In: Proceedings of the 7th Conference on Visualization’96, p. 101 (1996). IEEE Computer Society Press 23. Maciel, P.W., Shirley, P.: Visual navigation of large environments using textured clusters. In: Proceedings of the 1995 Symposium on Interactive 3D Graphics, p. 95 (1995). ACM 24. Coffey, D.M., Lin, C.-L., Erdman, A.G., Keefe, D.F.: Design by dragging: An interface for creative forward and inverse design with simulation ensembles. IEEE Trans. Vis. Comput. Graph. 19(12), 2783–2791 (2013) 25. Schroeder, W.: The Visualization Toolkit, 3rd edn. Kitware, Inc, ??? (2003) 26. Tracy, D.J., Buss, S.R., Woods, B.M.: Efficient large-scale sweep and prune methods with aabb insertion and removal. In: Virtual Reality Conference, 2009. VR 2009. IEEE, pp. 191–198 (2009). IEEE 27. Larsen, E., Gottschalk, S., Lin, M.C., Manocha, D.: Fast proximity queries with swept sphere volumes. Technical report, Technical Report TR99-018, Department of Computer Science, University of North Carolina (1999) 28. Oat, C., Barczak, J., Shopf, J.: Efficient spatial binning on the GPU. SIGGRAPH Asia (2008) 29. Rungta, A., Summa, B., Demir, D., Bremer, P.-T., Pascucci, V.: Manyvis: Multiple applications in an integrated visualization environment. Visualization and Computer Graphics, IEEE Transactions on 19(12), 2878–2885 (2013) 30. Leganchuk, A., Zhai, S., Buxton, W.: Manual and cognitive benefits of two-handed input: an experimental study. ACM Transactions on Computer-Human Interaction (TOCHI) 5(4), 326–359 (1998) 31. Hendrix, C., Barfield, W.: Relationship between monocular and binocular depth cues for judgements of spatial information and spatial instrument design. Displays 16(3), 103–113 (1995) 32. Taylor II, R.M., Hudson, T.C., Seeger, A., Weber, H., Juliano, J., Helser, A.T.: Vrpn: a device-independent, network-transparent vr peripheral system. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp. 55–61 (2001). ACM 33. http://cismm.org/downloads 34. Arthur, K., Preston, T., Taylor, R., Brooks, F., Whitton, M., Wright, W.: Designing and building the pit: a head-tracked stereo workspace for two users. In: Cruz-Neira, C. (ed.) 2nd International Immersive Projection Technology Workshop, pp. 11–12 35. Grant, B., Helser, A., II, R.M.T.: Adding force display to a stereoscopic head-tracked projection display. In: Proceedings of VRAIS ’98, pp. 81–88 36. Marshburn, D., Weigle, C., Wilde, B.G., Desai, K., Fisher, J.K., Cribb, J., O’Brien, E.T., Superfine, R., II, R.M.T.: The software interface to the 3d-force microscope. Proceedings of IEEE Visualization, 455–462 (2005) Figure 1

Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Additional files provided with this submission:

Additional file 1: sketchbio_v2_smaller.mp4, 19683K http://www.biomedcentral.com/imedia/8189175621333842/supp1.mp4 Additional file 2: bmc_article_sketchbio.tex, 68K http://www.biomedcentral.com/imedia/7428266221333846/supp2.tex Additional file 3: bmc_article_sketchbio.bbl, 18K http://www.biomedcentral.com/imedia/7254593421333848/supp3.bbl Additional file 4: sources.bib, 15K http://www.biomedcentral.com/imedia/1666243651133384/supp4.bib