Visualizing 3D molecular structures using an augmented reality app
Kristina Eriksen, Bjarne E. Nielsen, Michael Pittelkow
5 Department of Chemistry, University of Copenhagen, Universitetsparken 5, DK-2100 Copenhagen, Denmark. E-mail: [email protected]
ABSTRACT 10 We present a simple procedure to make an augmented reality app to visualize any 3D chemical model.
The molecular structure may be based on data from crystallographic data or from computer modelling.
This guide is made in such a way, that no programming skills are needed and the procedure uses free
software and is a way to visualize 3D structures that are normally difficult to comprehend in the 2D
15 space of paper. The process can be applied to make 3D representation of any 2D object, and we envisage
the app to be useful when visualizing simple stereochemical problems, when presenting a complex 3D
structure on a poster presentation or even in audio-visual presentations. The method works for all
molecules including small molecules, supramolecular structures, MOFs and biomacromolecules.
GRAPHICAL ABSTRACT
20
KEYWORDS
Augmented reality, Unity, Vuforia, Application, 3D models.
25
Journal 5/18/21 Page 1 of 14
INTRODUCTION
Conveying information about three-dimensional (3D) structures in two-dimensional (2D) space, such as
on paper or a screen can be difficult. Augmented reality (AR) provides an opportunity to visualize 2D
30 structures in 3D. Software to make simple AR apps is becoming common and ranges of free software
now exist to make customized apps. AR has transformed visualization in computer games and films,
but the technique is distinctly under-used in (chemical) science.1
In chemical science the challenge of visualizing in 3D exists at several levels ranging from teaching of
stereo chemistry problems at freshman university level to visualizing complex molecular structures at
35 the forefront of chemical research. Visualization can be especially challenging since molecules are getting
larger and more complex and span 3D. An elegant way to visualize molecules in 3D is to 3D print the
desired structure, and protocols of how to do this starting from molecular structures have recently been
described.2 To describe the geometry and symmetry of complex molecules chemists are often forced to
draw molecules in simplified or schematic ways and thus neglect information. One example of a highly
40 complex molecule that is difficult to display in 2D is the Molecular Borromean Rings prepared by
Stoddart and co-workers (Figure 1a). In their structural representation some atoms and labelling of
atoms are omitted to simplifying the structure.3 In the simplified 2D image bonds and atoms are
overlapping and thus still make it difficult to visualize the geometry and symmetry. Many chemists even
draw the same molecule twice in different formats in the same paper to better explain the connections
45 of the different elements and its geometry. This is illustrated with the porphyrin nano-ball by Anderson
(Figure 1b) and the supramolecular complex between biotin[6]uril and the iodide anion (Figure 1d).4-5 It
can be challenging to come up with a new synthetic route for complicated molecules such as Paclitaxel
(Figure 1c) because it is hard to visualize how sterically congested regions effects each other.6 For these
types of problems described above, a simple way to visualize molecules in 3D would be beneficial.
50
Journal 5/18/21 Page 2 of 14
Figure 1. Examples of complex molecules that are difficult to comprehend in 2D. Download the AR app on your device (phone or tablet) 55 to view these structures in AR. The app can be found via the QR code or the link in Figure 2. a) The Borromean rings are shown in two different ways to emphasize the geometry and symmetry of it. b) The porphyrin nanoball is presented in two different ways to highlight the geometry and the connection of the different elements in the nanoball. c) Paclitaxel is complex molecule with many stereo centers. A 3D view of it helps to appreciate the complexity and may aid retrosynthetic analysis. d) A topview and sideview of the crystal structure of the biotin[6]uril. Download the app by following the link below (Figure 2) and see how the AR works. 60
In this contribution we describe how to make a simple AR app for a mobile device (e.g. phones and
tablets) to visualize molecules in 3D. It is important to emphasize that this guide is made in such a way,
that no programming skills are needed, and that only free software is used. When the app is made it is
free to transfer the app via an USB cable from the computer to the mobile devices. However, if you want
65 to publish the app in Google Play it requires a one-time payment to Google Play of 25 $.
This guide must be followed tightly, because a series of programs are needed. When the AR app is made
and transferred to a mobile device, a camera opens and it recognises an image. The image may be on a
poster, in a book or on a screen. The recognition leads to a 3D model (of one or more molecules of your
Journal 5/18/21 Page 3 of 14
own choice) is opening as a part of the real world through your mobile device. When all the software is
70 installed correctly, the app is simple to make and can be used at a poster session or in classroom.
Download an example of the AR app (follow the link or QR code below) on your android mobile device
and see how it works and what it looks like (in the real augmented world). Once the app is downloaded
and opened, then point the camera at Figure 1 and 2 and a 3D model of the molecules will appear in
AR.
75 Link to the AR app: https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR
P Figure 2. A 2D image of Biotin[6]uril. Download the app from the link below or the QR code and point the mobile device to see the structure in 3D through the camera. https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR 80
85
Journal 5/18/21 Page 4 of 14
THE HOW TO GUIDE:
STEP 1: MAKING A 3D MODEL OF THE MOLECULE
If you don’t have a specific molecule or just want to make the
90 app to visualize a common molecule, then skip this step and go
to “Step 2: Working with Jmol”.
There are many programs that allow you to draw molecules in
3D. The most realistic 3D models for the given molecule are either from a crystal structure or a high
95 level optimization calculation by using software such as “Gaussian”. It is also possible to draw the
molecule in chemical drawing software, such as “Chem3D” in the “ChemDraw” software package.
When the geometry of your molecule is satisfying, then save it at the desktop as a “mol”-file. Only the
geometry of the molecule is needed to be correct at this state. The format of your molecule is added in
the next step.
100 STEP 2: WORKING WITH JMOL
The format of your molecule is changed by using the software
“Jmol”. If you don’t have Jmol then download it here:
http://jmol.sourceforge.net/download/ and unfold the “Jmol-
105 14.29.46-binary.zip” in a known folder. Click at “jmol.bat” and
Jmol will open. If you have any problems by running Jmol,
then make sure that you are running the latest Java version from Sun. You find and install it here:
www.java.com.
110 If you do not have a mol-file from Step 1 then click File→Create MOL and write e.g. “porphyrin”. Then
a porphyrin molecule will appear on your screen in Jmol.
If you saved a mol-file on the desktop from Step 1 then open it in Jmol. When it is opened click at
File→Console. The console is a terminal where you write commands and thereby change the design of
115 your molecule. Here is a list of useful commands to change your design of your molecule:
Journal 5/18/21 Page 5 of 14
http://cbm.msoe.edu/teachingResources/images/jmolTraining/changingColorsAndDisplayFormats.p
df
If you want the spacefill format of your 3D molecule, then write “spacefill on” in the console window and
press Enter. The “Enter” activates any code in the console window. If you want the ball-and-stick format
120 then write “spacefill 20%,wireframe 0.15”and press Enter.
If you want all the nitrogen atoms to be blue, then write “color nitrogen blue” or if you want the bonds
to be black then write “color bonds black” etc.
When you are happy with the design of your molecule then write “write molecule.obj” in the console,
125 press Enter and wait ca. 60 seconds until the console reports that the data is saved properly. Check
that your molecule is saved as an MTL- and an OBJ-file in the unfolded Jmol folder. As a check you can
open the “molecule.obj”-file and determine whether the format of your molecule looks satisfying.
STEP 3: WORKING WITH UNITY 130 The software “Unity” is made by Unity Technologies and is
a commercial multi-platform 3D game development tool and
can be used at all levels of computer skills. It has a free
version for people who are not using Unity at a professional
135 level. Unity runs on Windows and macOS but the app is
only made for “Android” devices in this paper.
Download “Unity Hub”, create an account and activate the
license. Install the program “Unity 2019.2.4f1” and in “Add
modules to your installation” select “Android Build
140 Support” and the two subprograms “Android SDK” and
“Open JDK”! The subprograms becomes visible by clicking at the grey arrow next to “Android Build
Support”.
Journal 5/18/21 Page 6 of 14
When Unity is downloaded then create a new 3D project under “Projects” in Unity Hub. A new project is
now open. Click (bottom left) at Project→Create→Folder and name the new folder “molecule”.
145
Now, drag the molecule.MTL file into the new folder “molecule” in Unity and then the molecule.OBJ file.
The order is important! The MTL file stores the design and the OBJ file stores the geometry. A new small
box showing your molecule should appear in the folder “molecule” in Unity. Drag the box onto the “Scene
window” (in the middle of the screen).
150
STEP 4: WORKING WITH VUFORIA
“Vuforia” is an augmented reality software
development kit (AR-SDK) for mobile
155 devices. It recognizes planar images
through a camera of the mobile device and
the recognition leads to the chosen 3D
model is displayed as a part of the real-
world scene.
160 There are different ways to implement
Vuforia in Unity. In the newest version of
Unity (from 2019.2) is automatically
installing Vuforia but it needs to be
activated. Vuforia is activated in Unity by
165 clicking: Edit→Project settings→Player→XR Settings and check the” Vuforia AR supported” box.
For implementing an image into your app go to the website: https://developer.vuforia.com. Click on
“Register” and go through the Creating an Account process. At the same website click on “Develop” in
the Navigation Bar and click “Get Development Key”. Type any name you want in the box “License name”,
confirm the agreement and click at “confirm” in the end. In the new opened window click on the name
170 you entered before and copy the license key in the grey box. Now click at “Target Manager” in Navigation
Bar (next to “License Manager” in the Navigation Bar) and press “Add Database”. Give the database a
Journal 5/18/21 Page 7 of 14
name, select “Device” and click “Create”. A new Database is now created. Now find the image (choose an
image with high quality) you want your app to recognize and save it on your desktop. The image should
be the one you want on your poster and ideally just a picture of your chosen molecule. Return to Vuforia
175 and click at the new Database you just made and press “Add target”. Select the image you just saved on
the desktop and upload it. Add a name for the Database and click “Add”. Finally click at “Download
Database (All)”, select “Unity Editor” as a development platform and save the unity package onto the
Desktop and open it directly after downloading it. When it is opened, a new window in Unity is opening
too. In the new opned window in Unity click “Import”.
180 STEP 5: USING VUFORIA IN UNITY
Step 5 guides you through how to build the AR app
in Unity by using Vuforia.
In the Hierarchy panel (top left) right click at “Main
185 Camera” and press Delete. We'll only use the AR
Camera. Now click at GameObject→ Vuforia
Engine→AR camera and an AR camera shows up
in the Hierarchy panel.
Click at “AR camera” in the Hierarchy panel and a
190 new small window named “Inspector” will appear. In Inspector click at “Vuforia Behaviour (Script)” and
then “Open Vuforia Engine configuration”. In the new window, paste the License Key in the App License
Key Box. You copied the license key in Step 4.
Now, click at GameObject→Vuforia Engine→Image and your Image target shows up in the Scene
window. Back in the Hierarchy panel, click at ImageTarget. In the new Inspector Tab, under the “Image
195 Target Behaviour (Script)”, click at the Database dropdown and select “[name] Database” instead of
empty. Click at “molecule” and in the Hierarchy panel and find the Transform section where you can
modify the position, rotation, and scale of your 3D model relative to the Image target. Change the size
and position of the molecule so it is proportional to the image. Do the same for the AR camera, so the
Journal 5/18/21 Page 8 of 14
small box showing the AR camera’s view is acutally pointing at the molecule and image in the Scene
200 window,
Finally in the Hierarchy panel, click and drag the “molecule” object exactly onto the ImageTarget so it
becomes a child of it. The AR app is now one step from being done. You can choose between building
the AR app on your mobile device (go to Step 6A) or the more complicated way where it is shown how to
publish the app in Goggle Play (go to Step 6B). As mentioened the last step (Step 6B) requries an
205 development account in Google Play and costs 25 $ (2019).
Journal 5/18/21 Page 9 of 14
THE STEP 6A: TRANSFERRING THE DATA TO THE MOBILE DEVICES
Step 6 explains how you install your app
210 on your mobile device. First you have to
allow installing the app on your mobile
device via a USB cable connecting your
device and the computer. At your mobile
device go to “Settings” and find “Build
215 numbers”. Tap several times on “build
numbers” until a new window appears
saying “you are now a developer!”. Go
back to Settings and find “Developer
option” on your mobile device. Scroll
220 down and find and activate “USB
debugging”.
Connect your mobile device via an USB cable to your computer. A new window pops up on your device:
check the box to allow debugging again. In Unity at your computer click:
File→BuildSettings→Android→Switch Platform and then “Build and Run”. Save the project as an apk-
225 file on your computer. The app is now built on your mobile device. If you have any problems with Unity
recognizing the device then it is must likely because your computer needs a USB-adb-driver. This is not
an issue for windows 10, but later versions should install an universal USB-adb-driver. One example
on a free universal USB-adb-driver can be found here:
https://software.intel.com/en-us/xdk/docs/installing-android-debug-bridge-adb-usb-driver-on-
230 windows
When it is completed you can remove the connection to the computer and open your new app on your
mobile device. Point the camera on your image target and enjoy the AR view.
Journal 5/18/21 Page 10 of 14
235 STEP 6B: PUBLISHING THE APP IN GOOGLE PLAY
To publish an app to Google play, the app
must comply with the design rules set by
Google play. One of these rules is that the
240 app must be provided in a package
containing boths package for the 32 bit
and 64 bit architecture. To include 64 bit
package navigate in Unity to the Player
settings by clicking File→BuildSettings
245 →Android→Player Settings. In the Player
setting expand the “Other Options” section
and under configuration locate the
scripting backed and change this from
“mono” to “IL2CPP”. Now activate the
250 “Arm64” check box. When publishing an
app to Google play, the app cannot be
signed with an developer key. To change the signing of the package expand the “Publishing Settings”,
and click the “Keystore manager” button. If this is the first app you are creating then press “Create new
keystore” and fill in the required information. Close the Player Settings and return to Build Settings and
255 check the box “Build app bundle”. This generates the aab-file which is needed to upload to Google play.
To publish an app to Google play a developer account is needed, this requires a one-time payment of 25
$ (2019). Find the website by searching “Google play developer console”, pay for the account and log in.
A new app is created by clicking the “Create application” button and filling in the name and default
language, finish by clicking “Create” and the “Store listing” menu opens. In this menu fill in the following
260 information: 1. Short description, 2. Full description, 3. Upload an image to create the icon for the app,
this image must have a resolution of 512x512. 4. Upload at least two screen shots. 5. Upload a “feature
graphic” this image must have a resolution of 1024x500. 6. Chose the application type from the
Journal 5/18/21 Page 11 of 14
dropdown menu. 7. Choose the category from the dropdown menu. 8. Fill in the contact email for the
app author. 9. As this app have access to the camera on the device you are also required to provide a
265 link to a privacy statement describing how you handle personal informaiton. Finish the Store listing
section by clicking Save.
Open the “App releases” dialog via the left hand navigation menu, create a new app release by clicking
“manage” in the “Production track” section and press “Create release”. In this windows carry out the
following steps. 1. Choose to let Google manage and protect your app signing key. Please note that this
270 means you need the same keystore as used when signing creating the intial .AAB to sign any updates.
2. Browse the .aab file you created in unity in “Android App Bundles and APKs to add” section. 3. Update
the release notes for the app in the “What's new in this release?” section. Complete this step by clicking
Save.
The left hand navigation menu contains the two mandatory steps “Content rating” and “App content”
275 both of these are questionaires pertaining to the nature of the content, complete these by filling in the
required information.
Now navigate to the “Pricing & distribution” and perform the following steps. 1. Choose “FREE” app. 2.
Select the countries that you want the app to be avaliable in. 3. Verify that you app complies with content
guidelines and US export laws by checking the boxes in the “Consent” section. Finalise by pressing “save
280 draft”.
To publish the app navigate back to the App eleases section, press the “Edit release” button in the
“Production track” section. Finalise by clicking the review button. Google play will now review that all
the necesary information have been provide and the app complies with registration. This can take up to
seven days.
285 NOTE : PERFORMANCE IMPROVEMENTS
If very large structures or slower android devices are used performance of the app can be a little
sluggish. It is not withing the scope of this paper to describe the full set of rendering optimizations one
can do in Unity but the following two hints should get you a good portion of the way.
290
Journal 5/18/21 Page 12 of 14
Remove excess strutures:
When Jmol is used to generate the obj-file, an addition sphere is created for each atom, even though
these spheres is located at the same spot as the original atom sphere unity takes up processing power
calculating positions of these. To remove these extra sphere open the unity project again. In the
295 Hierarchy panel expand the hierachy until the structure is expanded, right click at the structure and
select “Unpack prefab”. Now scroll down to the children named “SprereXXX” where XXX is a integer,
select them all and press “delete” this should not affect the 3D structure of the molecule but
signigicantly increase the performance.
300 CONCLUSION
We have described how to make a simple augmented reality app to visualize any 3D chemical model
using free softwares. The method works for all molecules including small molecules, supramolecular
structures, MOFs and biomacromolecules.
305
AUTHOR INFORMATION Corresponding Author *E-mail: [email protected]
ACKNOWLEDGMENTS 310 We acknowledge support from the Danish Council for Independent Research (DFF 4181-00206) and
from the University of Copenhagen.
315
Journal 5/18/21 Page 13 of 14
REFERENCES 320
1. Maier, P.; Klinker, G. Augmented Chemical Reactions: 3D Interaction Methods for Chemistry.
iJOE. 2013, 9 (S8), 80−82. b) Yang, S.; Mei, B.; Yue, X. Mobile Augmented Reality Assisted
Chemical Education: Insights from Elements 4D. J. Chem. Educ. 2018, 1060-1062. c) Wolle,
P.; Müller, M. P.; Rauh, D. Augmented reality in Scientific Publications – Taking the
325 Visulization of 3D Structures to the Next Level. ACS Chem. Biol., 2018, 13, 496-499.
2. Chen, T.-H.; Lee, S.; Flood, A. H.; Miljanic, O. S., How to print a crystal structure model in
3D, CrystEngComm 2014, 16, 5488–5493. B) Scalfani, V. F.; Vaid, T. P. 3D Printed Molecules
and Extended Solid Models for Teaching Symmetry and Point Groups. J. Chem. Educ., 2014,
91, 1174-1180.
330 3. Chichak, K. S.; Cantrill, S. J.; Pease, A. R.; Chiu, S-H.; Cave, G. W. V.; Atwood, J. L.; Stoddart, J. F. Molecular Borromean Rings. Science. 2004, 304, 1308-1312. 4. Cremers, J.; Haver, R.; Rickhaus, M.; Gong, J. Q.; Favereau, L.; Peeks, M. D.; Claridge, T. D.
W.; Herz, L. M.; Anderson, H. L. Template-Directed Synthesis of a Conjugated Zinc Porphyrin
Nanoball. J. Am. Chem. Soc. 2018, 140, 5352-5355.
335 5. Lisbjerg, M.; Jessen, B. M.; Rasmussen, B.; Nielsen, B. E.; Madsen, A. Oe.; Pittelkow, M.
Discovery of a cyclic 6+6 hexamer of D-Biotin and formaldehyde. Chem. Sci. 2014, 5, 2647-
2650.
6. Holton, R. A.; Somoza, C.; Kim, H.-B.; Liang, F.; Biediger, R. J.; Boatman, P. D.; Shindo, M.;
Smith, C. C.; Kim, S.; Nadizadeh, H.; Suzuki, Y.; Tao, C.; Vu, P.; Tang, S.; Zhang, P.; Murthi, K.
340 K.; Gentile, L. N.; Liu, J. H. First Total Synthesis of Taxol. 1. Functionalization of the B Ring.
J.Am. Chem. Soc., 1994, 116, 1597-1598.
Journal 5/18/21 Page 14 of 14