Masaryk University Faculty of Informatics Video Recording of Molecular Structures in Virtual Reality Master’s Thesis Vojtěch Brůža Brno, Spring 2019 Masaryk University Faculty of Informatics Video Recording of Molecular Structures in Virtual Reality Master’s Thesis Vojtěch Brůža Brno, Spring 2019 This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Vojtěch Brůža Advisor: doc. RNDr. Barbora Kozlíková, Ph.D. i Acknowledgements Foremost, I would like to thank my thesis advisor doc. RNDr. Barbora Kozlíková, Ph.D. for her support and insight. I really appreciate her trust, invaluable guidance and time. I am very grateful for her belief in my capabilities and success. I would not be able to accomplish the results without her. My thanks belongs also to my colleagues from VisIt Lab and HCI Laboratories. They were always willing to offer help and advice. I especially want to thank Jan Mičan, a biochemist from Loschmidt Laboratories, for his ideas, suggestions, collaboration and enthusiasm. He was always ready for cooperation and willing to assist. Finally, I also thank my wife for her comfort and encouragement. I am extremely thankful for the support she gave me during this challenging time. I would also like to thank my family and closest friends for their support, patience and for motivating me. iii Abstract The goal of this work was to implement a virtual environment for interactive video recording of a scene containing molecular visual representation. We have designed a solution for intuitive manipula- tion with molecular representation and recording system. Using Unity engine, we have developed a prototype application for HTC Vive and Oculus Rift devices. The application can load a molecule from the online PDB database and display it using a standard molecular repre- sentation technique. We have integrated elements essential for video recording. Using the provided solution, structural biology experts are able to adjust the camera trajectory and generate a resulting video, which can be stored in the MP4 format and used for communication and presentation purposes. iv Keywords computer graphics, Unity, virtual reality, visualization, 3D data, story- telling, animation v Contents Introduction 1 1 Background and Related Work 5 1.1 Molecular Data ........................5 1.2 History of Molecular Visualization ..............6 1.3 Molecular Animations ....................7 1.3.1 Molecular Dynamics . .8 1.3.2 User-driven Animation . .9 1.3.3 Automating the Creation of Animations . 11 1.4 Molecular Visualization in Virtual Reality ......... 13 1.4.1 Virtual Reality Tools . 14 2 Design & Requirements 17 2.1 Requirements ......................... 17 2.2 Protein Download ....................... 19 2.2.1 Input Data . 19 2.3 Protein Visualization ..................... 20 2.3.1 Color Modes . 21 2.4 Interaction with Protein Representation ........... 22 2.5 Real-Time Video Recording .................. 23 2.6 Camera Manipulation ..................... 24 2.7 Camera Trajectory Planning ................. 25 2.8 Video Format ......................... 27 2.9 Scene Design ......................... 28 3 Implementation 31 3.1 Used Technology ....................... 31 3.1.1 Unity . 31 3.1.2 Hardware Interfaces . 33 3.2 The Main Scene ........................ 35 3.3 Protein Visualization ..................... 35 3.3.1 Molecule Rendering . 37 3.4 Interactions and Control ................... 37 3.4.1 Interaction Tools . 38 3.4.2 Interactable Objects . 39 3.4.3 Virtual GUI . 44 vii 3.4.4 Voice Control . 44 3.4.5 Movement . 45 3.5 Real-time Capture ....................... 46 3.5.1 Unity Recorder . 47 3.5.2 RockVR . 47 3.6 Camera Trajectory Planning ................. 48 4 Results & Conclusion 51 4.1 Results ............................ 51 4.2 Conclusion & Future Work .................. 53 Bibliography 57 viii List of Figures 1.1 The illustration depicts the levels of organization of protein structures [21]. 6 1.2 The first system for the interactive display of molecular structures devised at MIT in the mid-1960s [25]. Detail of the CRT screen, with the globe that control the direction and speed of the wireframe image rotation. 7 2.1 The structure of deoxy human haemoglobin [75], represented by the space-filling model (created by QuteMol [76]). 20 2.2 Molecule displayed in three different color modes. 22 2.3 The user avatar pointing at a molecule while watching the video preview on both screens simultaneously. The small preview screen is displayed above the camera. The large screen is on a fixed position in the scene. 24 2.4 An example of the animation curve, used to enhance the camera movement interpolation between two consecutive animation keyframes. It corresponds to the smooth animation of the speed between, with speeding up and slowing down at the beginning and end of the trajectory, respectively. 27 2.5 Representation of the user avatar which consists of a headset visualization and two hands that represent the controllers. 30 3.1 Both supported head mounted display devices. 33 3.2 HTC Vive controller layout of buttons [82] 34 3.3 Illustration of the whole scene of the prototype application. The scene contains two loaded molecules, the camera, the large preview screen, UI keyboard, the shadow cameras (described in Section 3.4.2), the user avatar (also visible through camera preview on the large screen) and the surrounding room. 36 3.4 Avatar of the user in our virtual environment, operating with the protein (PDB ID 1AON [85]). The molecule is colored according to individual chains. 38 ix 3.5 Virtual representation of the hand tools with pointers attached to the index fingers of both hands. 39 3.6 User pointing at the camera which is capturing a selected part of the protein. Both the camera and the selected part are highlighted. The boxes floating around the camera are used to control the camera recording system. The red and blue buttons are related to the recording system and the other two buttons to the trajectory planning. 42 3.7 The camera and its control elements floating next to the camera are also attached to the hand. The recording state indicators are displayed when the recording is active. 43 3.8 The shadow camera with one the control button used for spawning new waypoints. 44 3.9 The shadow camera representing a single keyframe is grabbed. The display shows the preview of the video in this keyframe. 45 3.10 The keyboard being used for importing new pdb file to the application. 46 3.11 The camera following the trajectory of three waypoints. 49 x Introduction Computer science plays an important role in biology and chemistry already for decades. Computational analysis and visualization tech- niques enable the biologists and biochemists to save time and re- sources. Instead of performing thousands of expensive laboratory experiments, in-silico simulations and analyses can significantly re- duce the amount of experiments by filtering out infeasible options. Visualization is one of the key fields for biology, since visualization helps to explore data sets and communicate hypotheses and findings to others [1]. In general, visualization can aid in three functions that are essential components of the scientific enterprise: synthesis (pro- cess of creating a model), analysis (the examination and exploration of either data or models), and communication (sharing and presenting information) [2]. This thesis is focused mainly on the third function, the communication of information through visualization. One of the main foci of structural biology is visualization at nano- scale. The nanoscale visualization represents an interesting domain, because the objects of study (for example molecules, which are smaller than the wavelength of light) are invisible to the naked eye. Behavior of molecules is governed by physical forces and interactions signifi- cantly different from those forces and interactions that we encounter during our day-to-day experience. In this domain, effective models and visualizations are vital to provide the insight required to make research progress [3]. Using techniques, such as X-ray crystallography, nuclear magnetic resonance (NMR) spectroscopy, and cryo-electron microscopy [4], bio- structural data from experimental approaches are being generated at accelerating rates. Today, there are over 150,000 macromolecules stored in the Protein Data Bank [5]. Significant progress in displaying their structure, properties, and dynamic behavior has been made in the last decades [6], utilizing progress in fields of computer graphics, visualization, and human-computer interaction. Nowadays, we carry high definition color graphics displays in our pockets. Computational power and graphics processing units (GPUs) now enable real-time interactive display of billions of atoms, millions of proteins. Utilizing 1 this growing arsenal of biophysical and visualization techniques, the number of visualization programs and their functionality is growing. As already mentioned, molecules are too small to be analyzed and observed by a naked eye. This problem was initially solved by picturing the molecules as sets of spheres, using manual drawing. Using pictures to communicate ideas is nothing new.