Visualizing 3D molecular structures using an augmented reality app Kristina Eriksen, Bjarne E. Nielsen, Michael Pittelkow 5 Department of Chemistry, University of Copenhagen, Universitetsparken 5, DK-2100 Copenhagen, Denmark. E-mail: [email protected] ABSTRACT 10 We present a simple procedure to make an augmented reality app to visualize any 3D chemical model. The molecular structure may be based on data from crystallographic data or from computer modelling. This guide is made in such a way, that no programming skills are needed and the procedure uses free software and is a way to visualize 3D structures that are normally difficult to comprehend in the 2D 15 space of paper. The process can be applied to make 3D representation of any 2D object, and we envisage the app to be useful when visualizing simple stereochemical problems, when presenting a complex 3D structure on a poster presentation or even in audio-visual presentations. The method works for all molecules including small molecules, supramolecular structures, MOFs and biomacromolecules. GRAPHICAL ABSTRACT 20 KEYWORDS Augmented reality, Unity, Vuforia, Application, 3D models. 25 Journal 5/18/21 Page 1 of 14 INTRODUCTION Conveying information about three-dimensional (3D) structures in two-dimensional (2D) space, such as on paper or a screen can be difficult. Augmented reality (AR) provides an opportunity to visualize 2D 30 structures in 3D. Software to make simple AR apps is becoming common and ranges of free software now exist to make customized apps. AR has transformed visualization in computer games and films, but the technique is distinctly under-used in (chemical) science.1 In chemical science the challenge of visualizing in 3D exists at several levels ranging from teaching of stereo chemistry problems at freshman university level to visualizing complex molecular structures at 35 the forefront of chemical research. Visualization can be especially challenging since molecules are getting larger and more complex and span 3D. An elegant way to visualize molecules in 3D is to 3D print the desired structure, and protocols of how to do this starting from molecular structures have recently been described.2 To describe the geometry and symmetry of complex molecules chemists are often forced to draw molecules in simplified or schematic ways and thus neglect information. One example of a highly 40 complex molecule that is difficult to display in 2D is the Molecular Borromean Rings prepared by Stoddart and co-workers (Figure 1a). In their structural representation some atoms and labelling of atoms are omitted to simplifying the structure.3 In the simplified 2D image bonds and atoms are overlapping and thus still make it difficult to visualize the geometry and symmetry. Many chemists even draw the same molecule twice in different formats in the same paper to better explain the connections 45 of the different elements and its geometry. This is illustrated with the porphyrin nano-ball by Anderson (Figure 1b) and the supramolecular complex between biotin[6]uril and the iodide anion (Figure 1d).4-5 It can be challenging to come up with a new synthetic route for complicated molecules such as Paclitaxel (Figure 1c) because it is hard to visualize how sterically congested regions effects each other.6 For these types of problems described above, a simple way to visualize molecules in 3D would be beneficial. 50 Journal 5/18/21 Page 2 of 14 Figure 1. Examples of complex molecules that are difficult to comprehend in 2D. Download the AR app on your device (phone or tablet) 55 to view these structures in AR. The app can be found via the QR code or the link in Figure 2. a) The Borromean rings are shown in two different ways to emphasize the geometry and symmetry of it. b) The porphyrin nanoball is presented in two different ways to highlight the geometry and the connection of the different elements in the nanoball. c) Paclitaxel is complex molecule with many stereo centers. A 3D view of it helps to appreciate the complexity and may aid retrosynthetic analysis. d) A topview and sideview of the crystal structure of the biotin[6]uril. Download the app by following the link below (Figure 2) and see how the AR works. 60 In this contribution we describe how to make a simple AR app for a mobile device (e.g. phones and tablets) to visualize molecules in 3D. It is important to emphasize that this guide is made in such a way, that no programming skills are needed, and that only free software is used. When the app is made it is free to transfer the app via an USB cable from the computer to the mobile devices. However, if you want 65 to publish the app in Google Play it requires a one-time payment to Google Play of 25 $. This guide must be followed tightly, because a series of programs are needed. When the AR app is made and transferred to a mobile device, a camera opens and it recognises an image. The image may be on a poster, in a book or on a screen. The recognition leads to a 3D model (of one or more molecules of your Journal 5/18/21 Page 3 of 14 own choice) is opening as a part of the real world through your mobile device. When all the software is 70 installed correctly, the app is simple to make and can be used at a poster session or in classroom. Download an example of the AR app (follow the link or QR code below) on your android mobile device and see how it works and what it looks like (in the real augmented world). Once the app is downloaded and opened, then point the camera at Figure 1 and 2 and a 3D model of the molecules will appear in AR. 75 Link to the AR app: https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR P Figure 2. A 2D image of Biotin[6]uril. Download the app from the link below or the QR code and point the mobile device to see the structure in 3D through the camera. https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR 80 85 Journal 5/18/21 Page 4 of 14 THE HOW TO GUIDE: STEP 1: MAKING A 3D MODEL OF THE MOLECULE If you don’t have a specific molecule or just want to make the 90 app to visualize a common molecule, then skip this step and go to “Step 2: Working with Jmol”. There are many programs that allow you to draw molecules in 3D. The most realistic 3D models for the given molecule are either from a crystal structure or a high 95 level optimization calculation by using software such as “Gaussian”. It is also possible to draw the molecule in chemical drawing software, such as “Chem3D” in the “ChemDraw” software package. When the geometry of your molecule is satisfying, then save it at the desktop as a “mol”-file. Only the geometry of the molecule is needed to be correct at this state. The format of your molecule is added in the next step. 100 STEP 2: WORKING WITH JMOL The format of your molecule is changed by using the software “Jmol”. If you don’t have Jmol then download it here: http://jmol.sourceforge.net/download/ and unfold the “Jmol- 105 14.29.46-binary.zip” in a known folder. Click at “jmol.bat” and Jmol will open. If you have any problems by running Jmol, then make sure that you are running the latest Java version from Sun. You find and install it here: www.java.com. 110 If you do not have a mol-file from Step 1 then click File→Create MOL and write e.g. “porphyrin”. Then a porphyrin molecule will appear on your screen in Jmol. If you saved a mol-file on the desktop from Step 1 then open it in Jmol. When it is opened click at File→Console. The console is a terminal where you write commands and thereby change the design of 115 your molecule. Here is a list of useful commands to change your design of your molecule: Journal 5/18/21 Page 5 of 14 http://cbm.msoe.edu/teachingResources/images/jmolTraining/changingColorsAndDisplayFormats.p df If you want the spacefill format of your 3D molecule, then write “spacefill on” in the console window and press Enter. The “Enter” activates any code in the console window. If you want the ball-and-stick format 120 then write “spacefill 20%,wireframe 0.15”and press Enter. If you want all the nitrogen atoms to be blue, then write “color nitrogen blue” or if you want the bonds to be black then write “color bonds black” etc. When you are happy with the design of your molecule then write “write molecule.obj” in the console, 125 press Enter and wait ca. 60 seconds until the console reports that the data is saved properly. Check that your molecule is saved as an MTL- and an OBJ-file in the unfolded Jmol folder. As a check you can open the “molecule.obj”-file and determine whether the format of your molecule looks satisfying. STEP 3: WORKING WITH UNITY 130 The software “Unity” is made by Unity Technologies and is a commercial multi-platform 3D game development tool and can be used at all levels of computer skills. It has a free version for people who are not using Unity at a professional 135 level. Unity runs on Windows and macOS but the app is only made for “Android” devices in this paper.